Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that training with Ground-V, a 500K-pair dataset of automatically generated instruction-segmentation examples spanning five challenge dimensions, substantially improves pixel-level grounding of vision-language models…

desk verdict Large, well-curated grounding dataset, but the headline SOTA results rest on likely train/eval image overlap between Ground-V (COCO 2017) and RefCOCO/gRefCOCO (COCO 2014) — the paper needs deduplication and re-evaluation before the benchmark claims can be trusted. read the letter →

arxiv 2505.13788 v1 pith:JFRODULU submitted 2025-05-20 cs.CV

Yongshuo Zong , Qin Zhang , Dongsheng An , Zhihua Li , Xiang Xu , Linghan Xu , Zhuowen Tu , Yifan Xing
show 1 more author
Onkar Dabeer
This is my paper · ORCID
classification cs.CV
keywords visualgroundingreferringexpressionsegmentationinstructionfollowingdatasetgenerationvision-languagemodelshallucinationmitigationreasoningmulti-granularity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single large, automatically generated dataset can teach vision-language models to turn complex text instructions into accurate pixel-level segmentations. The dataset, Ground-V, contains roughly 500K instruction-segmentation pairs built by prompting a teacher model with few-shot examples that cover five real-world difficulties: multi-object references, multi-granular naming, hallucinated or absent objects, part-level queries, and reasoning. Training existing segmentation VLMs (LISA and PSALM) on this data improves their average generalized IoU by 4.4% and 7.9% across six benchmarks and yields state-of-the-art results on RefCOCO/+/g and gRefCOCO, including an 83.3% N-Acc on gRefCOCO that beats the prior state of the art by more than 20 points. If the result holds, it gives practitioners a low-annotation pathway to substantially better instruction-following segmentation.

What carries the argument

The machinery is the Ground-V data-generation pipeline. For each of five challenge dimensions, the authors hand-craft a three-shot prompt (image, instruction, answer) and query a teacher vision-language model to generate new instruction-response pairs that are then linked to existing pixel-level annotations, mainly from COCO 2017 and the PACO part annotations, preserving the original segmentation masks as supervision. Hallucination examples are deliberately assigned empty masks and text-only explanations so a correct model should abstain from producing a mask. The evaluation set is generated with the same workflow but then filtered through a judge model and two independent human annotators, retaining only pairs both annotators confirm. This pipeline is what carries the argument: it converts abundant existing masks into complex, nuanced instruction data at scale with minimal human cost.

What would settle it

Compute the image-ID intersection between the Ground-V training set (together with the COCO-based data it augments) and the test and validation splits of RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO; then retrain the same LISA and PSALM configurations on the deduplicated version of Ground-V and compare the gIoU and N-Acc numbers with the reported ones. If the gains shrink substantially after removing overlapping images, the headline improvements are at least partly due to training/test image leakage.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that scaling instruction-following data with a teacher-driven generation workflow elicits pixel-level grounding that existing training sets fail to provide. Concretely, the authors claim that augmenting the standard training mixtures of LISA and PSALM with Ground-V immediately improves grounding accuracy across six benchmarks, and that the resulting models surpass previous state of the art on RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO without requiring architectural changes. The most striking single number is a gRefCOCO N-Acc of 83.3%, exceeding the previous state of the art by more than 20 percentage points, which the paper attributes to the hallucination-mitigation portion of Ground-V.

Load-bearing premise

The evaluation results assume that the images used to build Ground-V (COCO 2017 train and val) are disjoint from the images used in the RefCOCO/+/g and gRefCOCO evaluation splits, and the paper does not report any deduplication between them; if those photographs overlap, the reported gains could partly reflect test images already seen during training.

Editorial extensions

If this is right

  • Training on Ground-V directly improves two differently architected segmentation VLMs, LISA and PSALM, with average gIoU gains of 4.4% and 7.9% across six benchmarks.
  • Ground-V-trained PSALM reaches an N-Acc of 83.3% on gRefCOCO, exceeding the previous state of the art by more than 20 points, indicating substantially better abstention when instructed objects are absent.
  • The hallucination subset, being text-only, can be dropped into general visual instruction tuning: the reported results on POPE, Vizwiz, MMBench, and SciQA show consistent improvements for a generalist VLM.
  • The model's performance is robust to rephrasing of the reasoning instructions, suggesting the gains reflect understanding rather than overfitting to one language style.
  • Data scaling within Ground-V shows monotonic improvement from 25% to 100% of the dataset, supporting the paper's data-centric scaling hypothesis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The caption-level overlap between COCO 2017 training images and COCO 2014 benchmark images is unresolved, so the headline gains should be treated as an upper bound until the authors report deduplication statistics.
  • Because the hallucination subset is text-only, it could be mixed into any visual instruction-tuning pipeline; the paper tests only one generalist VLM, but the abstention behavior may transfer to other models with similar tuning.
  • The ablation finding that hallucination training slightly reduces fine-grained recall suggests a tunable trade-off between abstention and sensitivity, so application-specific mixes of the five subsets could yield a family of models.
  • The zero-shot gain on MUSE hints that the multi-object and reasoning subsets encode transferable skills; testing PSALM-G5 on additional unseen referring benchmarks would clarify how far the dataset generalizes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Ground-V, a dataset of roughly 481K automatically generated instruction-segmentation pairs covering five challenges: multi-granularity, multi-object scenes, hallucinated references, part-level references, and reasoning-driven segmentation. The data is produced by prompting Claude 3 Sonnet to link natural language instructions to existing COCO annotations, with a smaller human-annotated test partition. The authors train two existing VLM segmentation models, LISA and PSALM, with Ground-V added to their standard training data, and report consistent gains on RefCOCO/+/g, gRefCOCO, D3, and the internal Ground-V test set, including new state-of-the-art results on gRefCOCO. The central claim is that this data-centric workflow substantially improves pixel-level grounding under complex instructions, with average gIoU gains of 4.4% for LISA and 7.9% for PSALM across six benchmarks.

Significance. If the empirical results are valid, the paper makes a useful data-centric contribution: the Ground-V dataset expands beyond simple referring expressions, and the evaluation across two model families with human-annotated test data is a strength. The authors also include a rephrasing robustness check and a general-VQA transfer experiment, which are thoughtful additions. However, the significance is contingent on resolving a serious train/eval image-overlap risk between the Ground-V training images (COCO 2017 train+val) and the public benchmark test images (built from COCO 2014), which the paper does not address. The internal test set also shares its generation pipeline with the training set, raising a circularity concern that is only partially mitigated. Because the headline SOTA and improvement claims depend on these evaluations, the current version does not fully support its central conclusions.

major comments (3)
  1. [Section 3.2 and Section 4.1] The paper states in Section 3.2 that Ground-V uses COCO 2017 train and validation splits as its image source, while the public benchmarks evaluated in Section 4.1 (RefCOCO/+/g and gRefCOCO) are built from COCO 2014 images. Since COCO 2017 train contains most COCO 2014 images, including much of the COCO 2014 validation set, the same photographs can appear in both Ground-V training and the benchmark test sets. The paper never reports any image-level deduplication against RefCOCO/+/g, gRefCOCO, or D3. With mask supervision on the exact test images, a model could exploit familiarity with object layouts and masks, inflating the cIoU/gIoU/N-Acc numbers in Tables 3, 4, and 6. Please provide a concrete overlap analysis (e.g., image hash or file-name comparison) and, if any overlap exists, retrain on a deduplicated Ground-V training set and re-report all benchmark results. This is necessary to support the headline claims of 4.4%/7.9% average improvements and state-of-the-art results.
  2. [Section 3.2 and Section 4.2, Table 7] The Ground-V test set is generated using the same Claude-based workflow as the training set, so large gains on that test set (Table 5) may partly reflect the model learning the teacher's instruction style and answer distribution rather than independent capability. The rephrasing experiment in Table 7, which is meant to address this, covers only the reasoning subset; the multi-granular, multi-object, hallucination, and part-reference subsets are not tested. Please extend the rephrasing robustness check to all five subsets, or provide a separate externally written evaluation set of comparable complexity, so the internal test results can be interpreted as evidence of general task ability rather than style matching.
  3. [Section 4.2, Figure 5] The scaling analysis in Figure 5 demonstrates that increasing the amount of Ground-V data improves performance on the Ground-V test set, but it does not isolate whether the gains are due to the content of the five addressed challenges or simply to adding a large volume of segmentation pairs. Please include a control experiment that trains on an equivalent volume of generic referring-expression data (for example, a randomly selected set of existing RefCOCO/RefCOCO+ training pairs of the same size) and report the resulting gains on the same benchmarks. Without such a control, the attribution of the improvements to the complex-instruction workflow is not fully supported.
minor comments (6)
  1. [Abstract and Tables 1-2] The abstract and introduction describe Ground-V as a 500K-pair dataset, while Table 1 reports 481K pairs and Table 2 totals 481,406 instruction pairs. Please reconcile these numbers by stating the exact count and explaining the rounding.
  2. [Table 4] The row for “PSALM (ft)” in Table 4 appears to have a formatting error: the first three entries read “-69.378.3 -”, which is ambiguous. Please clarify the intended values (or missing entries) for gIoU, cIoU, and N-Acc on the validation split.
  3. [Section 3.2 and Table 5] The paper states that the training set contains 50,000 images and the test set contains 5,000 images, but it does not explicitly describe how the test images are guaranteed to be disjoint from the training images. The Table 5 caption calls the test set “disjoint”; please specify the selection protocol that ensures no image-level overlap between the two partitions.
  4. [Section 4.2, footnote 4] Footnote 4 contains the typo “incorprated”; it should be “incorporated.”
  5. [Section 4.2, Table 7] The rephrasing experiment uses Qwen2-VL to paraphrase instructions, but no details are given about how the paraphrases were validated to preserve the original meaning and expected segmentation targets. Please describe the validation process for the rephrased instructions.
  6. [General] The paper does not state whether the Ground-V dataset and any associated code will be made publicly available. A data and code availability statement would be important for reproducibility and for the community to use the dataset.

Circularity Check

1 steps flagged · score 4.0 of 10

Internal Ground-V evaluation is self-referential because training and test prompts are generated by the same Claude workflow, but the external benchmarks keep the central claim partially independent.

  1. self definitional [Section 3.2, 'Detailed Data Generation Process'; Section 4.2, 'Generalization Beyond Language Priors']
    "This process is applied to generate both the training and evaluation sets. ... Since both the training and evaluation data in Ground-V are generated using the same workflow, we assess whether the performance improvements are due to overfitting to a consistent language style."

    The Ground-V training and test instruction-response pairs are both produced by the same teacher model (Claude 3 Sonnet) with the same manually crafted few-shot prompts. The internal test set is therefore drawn from the same distribution of instruction wording, answer format, object categories, and challenge taxonomy that the model is trained on. Performance on the Ground-V test set (Table 5) thus partly measures distribution matching to the generation pipeline, not independently established complex-instruction grounding. The paper's rephrasing check (Table 7) covers only the reasoning subset, so it does not remove this dependency for the other four challenges.

full rationale

The paper's core empirical claim is that adding Ground-V to training improves pixel-level grounding. This claim is substantially supported by external benchmarks that are not generated by the authors: RefCOCO/+/g, gRefCOCO, D3, ReasonSeg, and MUSE. For example, PSALM-G5 raises gRefCOCO N-Acc from 24.5 to 83.7, a large independent signal that the training data teaches abstention and generalized referring segmentation. These external results prevent the central claim from reducing to a self-consistent construction. The only genuinely self-referential element is the internal Ground-V test set, which is generated with the same Claude-based workflow as the training set. The paper itself acknowledges this and attempts a mitigation by rephrasing instructions on the reasoning subset; the results are stable, but the check covers only one of the five challenge dimensions and leaves the underlying task distribution unchanged. Therefore the internal evaluation scores are partly inflated by distribution matching, and the paper's use of those scores as evidence of general capability is partially circular. The COCO-2017/2014 image overlap concern raised in the review is a data-contamination and correctness issue, not a circularity of derivation; it is outside the scope of this circularity analysis. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled through citations. Overall circularity is moderate and localized to the internal benchmark, so the score is 4 rather than higher.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The Ground-V dataset and workflow rest on domain assumptions about the teacher model's reliability and about benchmark disjointness; there are no free mathematical parameters or newly posited physical entities in the paper.

assumptions (3)
  • domain assumption Claude 3 Sonnet, prompted with a few-shot demonstration, generates instruction-answer pairs that are correctly linked to the given segmentation masks.
    The entire training dataset depends on the teacher model's ability to map generated language to pixel annotations; the paper itself acknowledges noise in the training data in the Limitations section.
  • domain assumption COCO 2017 and PACO segmentation masks provide adequate ground-truth supervision for all five challenge types.
    The pipeline uses these existing masks as the supervision source without validating their correctness for the new instruction types (Section 3.2).
  • domain assumption The images used to build the Ground-V training set are disjoint from the images in the RefCOCO/+/g and gRefCOCO evaluation splits.
    The paper never states or demonstrates this disjointness; given the overlap between COCO 2017 train and COCO 2014 val, this assumption is likely false and is load-bearing for the claimed SOTA results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels." pith.science (2026). https://pith.science/paper/JFRODULU

@misc{pith2026250513788,
  author       = {Pith},
  title        = {Pith review of: Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFRODULU}},
  note         = {Machine review of arXiv:2505.13788}
}
read the original abstract

This work presents a simple yet effective workflow for automatically scaling instruction-following data to elicit pixel-level grounding capabilities of VLMs under complex instructions. In particular, we address five critical real-world challenges in text-instruction-based grounding: hallucinated references, multi-object scenarios, reasoning, multi-granularity, and part-level references. By leveraging knowledge distillation from a pre-trained teacher model, our approach generates high-quality instruction-response pairs linked to existing pixel-level annotations, minimizing the need for costly human annotation. The resulting dataset, Ground-V, captures rich object localization knowledge and nuanced pixel-level referring expressions. Experiment results show that models trained on Ground-V exhibit substantial improvements across diverse grounding tasks. Specifically, incorporating Ground-V during training directly achieves an average accuracy boost of 4.4% for LISA and a 7.9% for PSALM across six benchmarks on the gIoU metric. It also sets new state-of-the-art results on standard benchmarks such as RefCOCO/+/g. Notably, on gRefCOCO, we achieve an N-Acc of 83.3%, exceeding the previous state-of-the-art by more than 20%.

Figures

Figures reproduced from arXiv: 2505.13788 by the authors.

Figure 1
Figure 1. Performance comparison of LISA and PSALM models [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the diverse scenarios covered in our Ground-V dataset. These include reasoning-based segmentation (top left), [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Overview of the G5 data generation pipeline. We identify key real-world challenges, including hallucination, multi-object [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Our model demonstrates improved understanding of object attributes, enhanced reasoning capabilities, more precise segmentation [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Performance of models trained on different subsets. X [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Word Cloud visualizations for training (top) and evaluation (bottom) sets of hallucination mitigation subsets. We show word [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Word Cloud visualizations for training (top) and evaluation (bottom) sets of multi-granular subsets. We show word clouds for [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Word Cloud visualizations for training (top) and evaluation (bottom) sets of reasoning subsets. We show word clouds for [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Word Cloud visualizations for training (left) and evaluation (right) sets of part reference subsets. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Word Cloud visualizations for training (left) and evaluation (right) sets of multi-object subsets. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Human annotation interface for Ground-Vtest data that resemble reasoning segmentation. In this case, the question is asking [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Human annotation interface for Ground-Vtest data that resemble multi-granularity referring. In this example, the label “African [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Human annotation interface for Ground-Vtest data that resemble complex referrings that that could lead to hallucinations in [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 44 canonical work pages

  1. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023. 1, 2

    arXiv 2023

  2. [2]

    Language models are few-shot learners.Advances in Neural Information Process- ing Systems, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    work page 2020

  3. [3]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. InProceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1209–1218, 2018. 1

    work page 2018

  4. [4]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022. 2, 5

    work page 2022

  5. [5]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 5

    2023

  6. [6]

    Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    2024

  7. [7]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. InThirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 1

    work page 2023

  8. [8]

    Vision-language transformer and query generation for refer- ring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for refer- ring segmentation. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 16321–16330,

Show all 65 references
  1. [9]

    Segnext: Rethink- ing convolutional attention design for semantic segmenta- tion.Advances in Neural Information Processing Systems, 35:1140–1156, 2022

    Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethink- ing convolutional attention design for semantic segmenta- tion.Advances in Neural Information Processing Systems, 35:1140–1156, 2022. 2

    2022

  2. [10]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  3. [11]

    Partimagenet: A large, high- quality dataset of parts

    Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xi- aoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qi- hang Yu, and Alan Yuille. Partimagenet: A large, high- quality dataset of parts. InEuropean Conference on Com- puter Vision, pages 128–145. Springer, 2022. 1

    2022

  4. [12]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023...

    2023

  5. [13]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 1, 3, 4, 5, 7

    2024

  6. [14]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 2

    2023

  7. [15]

    Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023

    Yuanzhi Li, S ´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023. 5

    2023 arXiv

  8. [16]

    Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023. 8

    2023 arXiv

  9. [17]

    A real-time cross-modality correlation fil- tering method for referring expression comprehension

    Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross-modality correlation fil- tering method for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 10880–1088...

    2020

  10. [18]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, pages 740–755. Springer, 2014. 5, 1

    2014

  11. [19]

    Gres: Gen- eralized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Gen- eralized referring expression segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592–23601, 2023. 2, 3, 5, 7

    2023

  12. [20]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2023. 1, 2, 5, 8

    2023

  13. [21]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2

    2024

  14. [22]

    Poly- former: Referring image segmentation as sequential poly- gon generation

    Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Ku- mar Satzoda, Vijay Mahadevan, and R Manmatha. Poly- former: Referring image segmentation as sequential poly- gon generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1865...

    2023

  15. [23]

    Grounding dino: Marry- ing dino with grounded pre-training for open-set object de- tection.ECCV, 2024

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marry- ing dino with grounded pre-training for open-set object de- tection.ECCV, 2024. 1

    2024

  16. [24]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233. Springer, 2024. 8

    2024

  17. [25]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 5

    2021

  18. [26]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. InInternational Conference on Learning Representations, 2019. 1

    2019

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

  20. [28]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 1

    2016

  21. [29]

    Mod- eling context between objects for referring expression under- standing

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression under- standing. InComputer Vision–ECCV 2016: 14th European Conference, pages 792–807. Springer, 2016. 2, 5

    2016

  22. [30]

    Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Ground- ing multimodal large language models to the world. InThe Twelfth International Conference on Learning Representa- tions, 2024. 2

    2024

  23. [31]

    Perceptiongpt: Effectively fusing visual perception into llm

    Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into llm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27124– 27133, 2024. 3, 7

    2024

  24. [32]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  25. [33]

    Vision language models are blind.arXiv preprint arXiv:2407.06581, 2024

    Pooyan Rahmanzadehgervi, Logan Bolton, Moham- mad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind.arXiv preprint arXiv:2407.06581, 2024. 2

    2024 arXiv

  26. [34]

    Paco: Parts and attributes of common objects

    Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Mar- quez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2023

  27. [35]

    Learning to lo- calize objects improves spatial reasoning in visual-llms

    Kanchana Ranasinghe, Satya Narayan Shukla, Omid Pour- saeed, Michael S Ryoo, and Tsung-Yu Lin. Learning to lo- calize objects improves spatial reasoning in visual-llms. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 12977–12987, 2024. 2

    2024

  28. [36]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  29. [37]

    Sam 2: Segment anything in images and videos,

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

  30. [38]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    arXiv

  31. [39]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024. 1, 4, 7

    2024

  32. [40]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 2

    2024

  33. [41]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 7

    2024 arXiv

  34. [42]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,...

    2023

  35. [43]

    Cris: Clip- driven referring image segmentation

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip- driven referring image segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11686–11695, 2022. 2

    2022

  36. [44]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024. 1, 3, 7

    2024

  37. [45]

    Described object detection: Liberating ob- ject detection with flexible expressions.Advances in Neural Information Processing Systems, 36, 2023

    Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating ob- ject detection with flexible expressions.Advances in Neural Information Processing Systems, 36, 2023. 6

    2023

  38. [46]

    u-llava: Unifying multi- modal tasks via large language model.arXiv preprint arXiv:2311.05348, 2023

    Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Unifying multi- modal tasks via large language model.arXiv preprint arXiv:2311.05348, 2023. 3, 7

    2023 arXiv

  39. [47]

    Universal instance percep- tion as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Ze- huan Yuan, and Huchuan Lu. Universal instance percep- tion as object discovery and retrieval. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325–15336, 2023. 1, 2, 7

    2023

  40. [48]

    Lavt: Language-aware vision transformer for referring image segmentation

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Heng- shuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155–18165, 2022. 2, 7

    2022

  41. [49]

    Cross-modal self-attention network for referring image seg- mentation

    Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image seg- mentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10502– 10511, 2019. 2

    2019

  42. [50]

    Modeling context in referring ex- pressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring ex- pressions. InComputer Vision–ECCV 2016, pages 69–85. Springer, 2016. 2, 5, 1

    2016

  43. [51]

    Mattnet: Modular at- tention network for referring expression comprehension

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular at- tention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1307–1315, 2018. 2, 7

    2018

  44. [52]

    Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023. 2

    2023 arXiv

  45. [53]

    Psalm: Pixelwise segmentation with large multi-modal model.ECCV, 2024

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model.ECCV, 2024. 1, 2, 3, 5, 7

    2024

  46. [54]

    Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019. 1

    2019

  47. [55]

    Seqtr: A simple yet universal network for visual grounding

    Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. InEuropean Conference on Computer Vi- sion, pages 598–615, 2022. 2

    2022

  48. [56]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. InThe Twelfth International Conference on Learning Representa- tions, 2024. 1

    2024

  49. [57]

    Self-supervised multimodal learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2

    2024

  50. [58]

    African Bush Elephant

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once.Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 7 Ground-V: Teaching VLMs to Ground Complex Instruc...

    2024

  51. [59]

    Example: ”Corgi,” ”Macbook,” ”SUV”

    Fine-grained Level: Combines the specific instance and subcategory levels. Example: ”Corgi,” ”Macbook,” ”SUV”

  52. [60]

    Example: ”Dog,” ”Computer,” ”Car”

    General Category Level: Standard categories of objects. Example: ”Dog,” ”Computer,” ”Car”

  53. [61]

    Abstract Level: Broad, overarching categories. Example: ”Animal,” ”Electronic Device,” ”Transportation” Few-shot Examples: **Image 1 - Given objects: fish, dog - fine-grained labels: [”gold fish”, ”corgi”] **Image 2 - Present objects: laptop, container, box - ”fine-grained lab...

  54. [62]

    The answer must reference the given object class or its equivalent and should not imply other potential objects

  55. [63]

    The question should require some reasoning to answer and cannot be too broad

  56. [64]

    The question should describe a activity or incorporate world knowledge

  57. [65]

    (2) text answer: it should be one or a few coherent sentences connecting the objects in (1) that answer the question

    You will output two forms of answers: (1) objects: it contains the objects required to answer the question, each object should be in a form of object at [bounding box]. (2) text answer: it should be one or a few coherent sentences connecting the objects in (1) that answer the ...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.