Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InstructSeg unifies referring and reasoning segmentation in images and videos into one MLLM pipeline and reports state-of-the-art results on every benchmark it tests.

desk verdict The unified end-to-end system is real and the ablations hold together, but the headline SOTA margins are mostly unexplained because the base MLLM differs from every prior MLLM baseline. read the letter →

arxiv 2412.14006 v1 pith:WWSBROR3 submitted 2024-12-18 cs.CV

Cong Wei , Yujie Zhong , Haoxian Tan , Yingsen Zeng , Yong Liu , Zheng Zhao , Yujiu Yang This is my paper · ORCID
classification cs.CV
keywords instructedvisualsegmentationreferringexpressionreasoningvideoobjectmulti-modallargelanguagemodelsobject-awareperceivervision-guidedmulti-granularitytextfusionend-to-endtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that four text-guided segmentation tasks—referring and reasoning segmentation in both images and videos—can be handled by a single end-to-end model rather than by separate specialists or by bolting together a video-LLM, a tracker, and a segmentation module. It introduces InstructSeg, a 3-billion-parameter multi-modal large language model pipeline with frozen vision encoders and LoRA fine-tuning, and reports state-of-the-art numbers on RefCOCO/+/g, ReasonSeg, Ref-YouTube-VOS, Ref-DAVIS, and ReVOS. On the reasoning-heavy ReVOS benchmark it claims a +11.0 J&F gain over the previous best method while using fewer parameters. The point of the unification is that image and video segmentation share the same instruction-following core, so a single training run should transfer understanding across both domains. The practical payoff would be one model that a user can point at either a photo or a video, ask in plain language for an object or something requiring world-knowledge reasoning, and get a mask.

What carries the argument

The two modules that carry the argument are the Object-aware Video Perceiver (OVP) and the Vision-guided Multi-granularity Text Fusion (VMTF). OVP uses learnable queries in cross-attention layers to condense a short sequence of reference frames plus the language instruction into fixed-length tokens that the LLM consumes, giving video tasks temporal and object information without a separate tracker. VMTF takes the LLM's per-token text embeddings, pools a global text embedding, and refines both global and detailed embeddings through cross-attention with fine-grained visual features, producing multi-granularity text embeddings that act as the mask classifier. The surrounding design—Mask2Former-style mask decoding, LoRA fine-tuning of the LLM, and frozen CLIP and visual encoders—makes the pipeline end-to-end trainable with a single loss composed of text, classification, and mask terms.

What would settle it

Run InstructSeg's training recipe with its two new modules removed but the same Mipha-3B backbone, and run LISA or VISA with the same backbone under the same data. If the +7.2 cIoU on RefCOCO+ and +11.0 J&F on ReVOS reasoning largely persist with a shared backbone, the claim that the architecture is responsible would be falsified; if they shrink or vanish, the backbone is the driver.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that instructed visual segmentation (IVS)—referring expression segmentation, reasoning segmentation, referring video object segmentation, and reasoning video object segmentation—is a single task family, and that one end-to-end MLLM can master all of them. InstructSeg feeds the image or key frame, text tokens, and compressed reference-frame tokens from the object-aware video perceiver into one large language model, which emits mask tokens and detailed text embeddings; the vision-guided multi-granularity text fusion module combines global and per-token text information under visual guidance, and a Mask2Former-style decoder produces masks and scores. Trained jointly on RefCOCO/+/g, ReasonSeg, Ref-YouTube-VOS, ReVOS, and LLaVA-150k, the model reports higher scores than segmentation specialists and prior MLLM-based generalists on all reported benchmarks, including image-level referring, image-level reasoning, video referring, and video reasoning splits.

Load-bearing premise

The load-bearing premise is that the reported margins come from the proposed architecture: all comparisons use different base language models (LLaVA/Vicuna, Chat-UniVi, Phi-2), and no same-backbone control is reported, so part of the gain could be the stronger base model rather than the new modules.

Editorial extensions

If this is right

  • A single 3B model replaces the two-stage pipelines that combine a frame sampler, a video-LLM, and a tracker for video segmentation.
  • Training jointly on image and video tasks transfers across them: the paper reports that an image-only model shows zero-shot ability on video reasoning, and a video-only model already performs well on RefCOCO.
  • Reasoning-style training alone transfers to referring tasks, supporting the view that reasoning and referring share one instruction-following core.
  • The reported gains on RefCOCO/+/g and ReasonSeg indicate that an MLLM with frozen encoders and LoRA can beat dedicated segmentation specialists, not just other generalists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If multi-granularity text fusion is the right mechanism, similar fusion of global and token-level embeddings could improve other MLLM grounding outputs, such as referring comprehension or grounded captioning, not just masks.
  • The object-aware video perceiver's token compression points toward scaling to longer videos: the paper already reports gains when the reference frame count rises from 4 to 8, suggesting a longer-video benchmark could test where further gains saturate.
  • The IVS task union suggests a natural next step: a combined benchmark where a model receives a mix of image and video instructions and is scored on all four task types at once, making cross-domain transfer an explicit evaluation rather than a by-product.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper defines Instructed Visual Segmentation (IVS) as the union of referring and reasoning segmentation at both image and video levels, and proposes InstructSeg, an end-to-end segmentation pipeline built on a lightweight MLLM (Mipha-3B). The method combines an Object-aware Video Perceiver (OVP) for temporal and object information, a Vision-guided Multi-granularity Text Fusion (VMTF) module, and a Mask2Former-style decoder. The model is trained jointly on RefCOCO/+/g, ReasonSeg, Ref-YouTube-VOS, and ReVOS, and reports state-of-the-art results on these benchmarks, along with competitive VQA results. The paper also includes ablations of the proposed components, reference-frame count, text fusion strategy, and training-data recipes.

Significance. If the reported results are reliable, InstructSeg provides a strong simplified alternative to multi-specialist pipelines such as VISA: a single 3B-scale MLLM with frozen encoders, LoRA fine-tuning, and a single decoder handles four text-guided segmentation tasks with state-of-the-art performance. The release of code is a strength for reproducibility. However, the significance is tempered by two main issues: the SOTA claims are confounded by the choice of base MLLM (Mipha-3B/Phi-2 versus LLaVA/Vicuna/Chat-UniVi in the baselines), and the proposed modules contribute only modest gains in the ablations compared with the large headline margins. The absence of error bars further weakens the support for the claimed improvements. With additional same-backbone comparisons and uncertainty quantification, the contribution could become solid.

major comments (4)
  1. [Section 4.1, Tables 2-4, and Table 6] The central SOTA claim is confounded by the base MLLM choice. InstructSeg uses Mipha-3B (Phi-2, Section 3.1) while the strongest MLLM-based baselines (LISA, VISA, PSALM, TrackGPT) use LLaVA-7B/13B, Vicuna, or Chat-UniVi. The ablation in Table 6 shows that OVP+VMTF add only +2.5 cIoU on RefCOCO val and +1.6 J&F on ReVOS overall, whereas the headline margins over prior methods are much larger (e.g., +7.2 cIoU on RefCOCO+ val and +7.0 J&F on ReVOS overall). Without a same-backbone comparison (e.g., training the baselines with Mipha-3B, or removing OVP/VMTF from InstructSeg while keeping the same base), the gains cannot be attributed to the proposed architecture. Please provide such experiments or, if they are impossible, explicitly discuss the base-model contribution and temper the attribution claim.
  2. [Section 3.4 and evaluation protocol] The final mask is derived by applying a threshold to mask scores, but the threshold value and selection procedure are not reported. If the threshold is tuned per benchmark or on the test set, the comparison with baselines could be unfair. Please specify the threshold, state whether it is fixed across all datasets, and describe how it was chosen; ideally, show sensitivity to threshold variation.
  3. [Tables 2-6] No error bars or multiple-seed results are reported. Several improvements are modest (e.g., +0.7 J&F on Ref-DAVIS over VISA-13B in Table 4; +1.6 J&F overall in the Table 6 ablation). Without estimates of variance, it is unclear whether these differences are significant or within run-to-run noise. Please report means and standard deviations (or comparable uncertainty measures) for the main tables and ablations.
  4. [Section 4 (Datasets) and Table 3] The paper trains on ReVOS and evaluates on ReVOS, but does not state the train/test split. Please clarify whether the evaluation set is a held-out portion disjoint from training; if training and evaluation use the same ReVOS set, the ReVOS results would be circular. This is load-bearing because ReVOS is a cited SOTA benchmark.
minor comments (5)
  1. [Section 4 (Implementation details)] There is a typo: "Maks2Former" should be "Mask2Former" (appears also in Supplementary Material A).
  2. [Section 3.2, Eqs. (1)-(2)] The notation E is used both for input text tokens and for the output embeddings of the LLM in Eq. (2); please disambiguate these two uses.
  3. [Section 1 and Table 1] Table 1 shows that VISA already covers all four tasks of IVS. The novelty of this work should be positioned as the simplified end-to-end pipeline rather than the task union itself, otherwise the contribution statement overclaims.
  4. [Section 4.1] The sentence "surpasses all the previous works" is made per benchmark, but the set of baselines differs across Tables 2-4. Please qualify the SOTA claim accordingly (e.g., "state-of-the-art on each benchmark" rather than an unqualified global claim).
  5. [References] Several cited methods are arXiv preprints at the time of writing (e.g., [41], [42], [47]). Please add official versions or venues if they have been published, as this helps the reader assess maturity of the baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the results are benchmark evaluations with same-backbone ablations; self-citations are not load-bearing.

full rationale

The paper's central claims are empirical: InstructSeg is trained with a standard supervised loss (Eq. 4) on standard benchmark training splits and evaluated on held-out test sets (Tables 2-4). No reported number is produced by fitting a parameter to that number or by renaming an input as an output. The proposed modules are ablated in Table 6 under a fixed backbone (Mipha-3B/Swin-B), so the reported contributions of OVP and VMTF are not defined in terms of the headline SOTA margins. The only self-citations (LaSagnA [41], HyperSeg [42]) appear in related work and baseline tables; they are not invoked as a proof, uniqueness theorem, or justification of the architecture. The term 'IVS' is a task taxonomy, not a derived result that presupposes InstructSeg's performance. The strongest concern is that cross-backbone SOTA comparisons may be confounded by the choice of Mipha-3B versus LLaVA/Chat-UniVi in prior systems; that is a validity or attribution concern, not a circularity of derivation. Accordingly, no circular step is exhibited and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The model is an empirical system; the free parameters are architecture and hyperparameter choices. The central SOTA claim rests on benchmark comparability and on the base model not dominating the comparison.

free parameters (4)
  • Reference frames count Tr = 4
    Selected based on ablation (Tab. 7); performance slightly higher with Tr=8 but 4 chosen for efficiency.
  • Perceiver layers N1 = 3
    Set by default without ablation.
  • VMTF layers N2 = 3
    Set by default without ablation.
  • Loss weights lambda_cls, lambda_mask, lambda_b, lambda_d = 1.0 each
    Set 'experimentally' (Section 3.4), no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Benchmark datasets and metrics (cIoU, gIoU, J&F) reliably measure instructed segmentation quality and are computed identically across compared methods.
    All conclusions compare single numbers from Tables 2-4; inconsistent post-processing or evaluation code would invalidate the SOTA claim.
  • domain assumption Phi-2/Mipha-3B offers reasoning and segmentation capability comparable or superior to prior 7B/13B base LLMs, so the reported gains are not solely due to the base model.
    No same-backbone baseline is run; prior methods use LLaVA-7B/13B and Chat-UniVi-7B/13B.
  • domain assumption Frozen CLIP and Swin-B features provide sufficient visual information; the learnable modules can bridge them to the LLM and decoder.
    Architecture relies on frozen encoders, as stated in Section 3.1.
  • ad hoc to paper The mask score threshold used to derive the final mask (Section 3.4) is consistent with the evaluation settings of baseline methods.
    The threshold value is not reported, yet it could affect cIoU and J&F comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models." pith.science (2026). https://pith.science/paper/WWSBROR3

@misc{pith2026241214006,
  author       = {Pith},
  title        = {Pith review of: InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWSBROR3}},
  note         = {Machine review of arXiv:2412.14006}
}
read the original abstract

Boosted by Multi-modal Large Language Models (MLLMs), text-guided universal segmentation models for the image and video domains have made rapid progress recently. However, these methods are often developed separately for specific domains, overlooking the similarities in task settings and solutions across these two areas. In this paper, we define the union of referring segmentation and reasoning segmentation at both the image and video levels as Instructed Visual Segmentation (IVS). Correspondingly, we propose InstructSeg, an end-to-end segmentation pipeline equipped with MLLMs for IVS. Specifically, we employ an object-aware video perceiver to extract temporal and object information from reference frames, facilitating comprehensive video understanding. Additionally, we introduce vision-guided multi-granularity text fusion to better integrate global and detailed text information with fine-grained visual guidance. By leveraging multi-task and end-to-end training, InstructSeg demonstrates superior performance across diverse image and video segmentation tasks, surpassing both segmentation specialists and MLLM-based methods with a single model. Our code is available at https://github.com/congvvc/InstructSeg.

Figures

Figures reproduced from arXiv: 2412.14006 by the authors.

Figure 1
Figure 1. We define Instructed Visual Segmentation (IVS) as the union of four text-guided segmentation tasks across image and video [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of InstructSeg. InstructSeg tackles Instructed Visual Segmentation tasks in an end-to-end pipeline. For challenging [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. The structure of the Vision-guided Multi-granularity Text [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: The structure of the Segmentation Decoder module. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results of InstructSeg’s capability in referring expression segmentation. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results of InstructSeg in reasoning segmentation. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results of InstructSeg demonstrate its capability in the complex reasoning video object segmentation task and referring [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    HRSeg combines high-resolution image crops, region attention, and cross-attention mask enhancement, improving reasoning segmentation over LLM-Seg by up to 13.2 gIoU points on LLM-Seg40K.

  2. Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    OpenBench, a new benchmark with categories semantically far from the COCO training space, shows that fine-tuning CLIP hurts open-vocabulary segmentation, and the proposed OVSNet method achieves state-of-the-art on bot...

  3. On Domain-Adaptive Post-Training for Multimodal Large Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A generate-then-filter, open-source-only synthesis pipeline plus single-stage post-training consistently improves MLLM performance across biomedicine, food, and remote sensing.

  4. Reasoning Segmentation for Images and Videos: A Survey

    cs.CV 2025-05 conditional novelty 3.0 of 10

    The paper organizes the field of reasoning segmentation into image and video tracks, cataloging 26 methods, 12 metrics, and 29 datasets.

Reference graph

Works this paper leans on

57 extracted references · 29 canonical work pages · cited by 4 Pith papers

  1. [7]

    Masked-attention mask trans- former for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask trans- former for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022. 3, 5, 6, 1

    work page 2022

  2. [53]

    Psalm: Pixelwise segmentation with large multi-modal model

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. arXiv preprint arXiv:2403.14598, 2024. 2, 3, 5, 6, 1

    arXiv 2024

  3. [27]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 6, 1

    work page 2021

  4. [31]

    Spectrum-guided multi-granularity referring video object segmentation

    Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guided multi-granularity referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 920– 930, 2023. 6, 7

    work page 2023

  5. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716–23736,

  6. [2]

    Minigpt4-video: Advancing multimodal llms for video under- standing with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Es- sam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video under- standing with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024. 3

    arXiv 2024

  7. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 8

    arXiv 2023

  8. [4]

    A closer look at referring expressions for video object segmentation

    Miriam Bellver, Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i Nieto. A closer look at referring expressions for video object segmentation. Multimedia Tools and Applications, 82(3):4419–4438, 2023. 3

    work page 2023

Show all 57 references
  1. [5]

    End-to-end referring video object segmentation with multi- modal transformers

    Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multi- modal transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4985–4995, 2022. 3, 7

    2022

  2. [6]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195,

    arXiv

  3. [8]

    Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pages 640–658. Springer, 2022. 3

    2022

  4. [9]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 8

    2023 arXiv

  5. [10]

    Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023. 8

    2023

  6. [11]

    Vision-language transformer and query generation for refer- ring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for refer- ring segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 16321–16330,

  7. [12]

    Vlt: Vision-language transformer and query generation for re- ferring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vlt: Vision-language transformer and query generation for re- ferring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7900–7916, 2022. 7

    2022

  8. [13]

    Mevis: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2694–2703, 2023. 2, 7

    2023

  9. [14]

    Dataseg: Taming a universal multi-dataset multi-task segmentation model

    Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang, Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang-Chieh Chen, et al. Dataseg: Taming a universal multi-dataset multi-task segmentation model. Ad- vances in Neural Information Processing Systems, 36, 2024. 5

    2024

  10. [15]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jy- oti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1:3, 2023. 1 9

    2023

  11. [16]

    Mdetr-modulated detection for end-to-end multi-modal understanding

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Syn- naeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 1780–1790, 2021. 3

    2021

  12. [17]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 4015–4026, 2023. 3

    2023

  13. [18]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. ArXiv, abs/2308.00692, 2023. 2, 3, 5, 6, 7

    2023 arXiv

  14. [19]

    You only infer once: Cross-modal meta-transfer for referring video object segmentation

    Dezhuang Li, Ruoqi Li, Lijun Wang, Yifan Wang, Jinqing Qi, Lu Zhang, Ting Liu, Qingquan Xu, and Huchuan Lu. You only infer once: Cross-modal meta-transfer for referring video object segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1297–1305...

    2022

  15. [20]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 3, 8

    2023 arXiv

  16. [21]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 3

    2023 arXiv

  17. [22]

    Referring transformer: A one-step approach to multi-task visual grounding

    Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems, 34:19652–19664,

  18. [23]

    Llama-vid: An im- age is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An im- age is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer,

  19. [24]

    Gres: Gener- alized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Gener- alized referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23592–23601, 2023. 2, 6

    2023

  20. [25]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3, 6, 8

    2024

  21. [26]

    Polyformer: Referring image segmentation as sequential polygon genera- tion

    Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon genera- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 18653–18663,

  22. [28]

    Multi-task collaborative network for joint referring expression comprehension and segmentation

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , page...

    2020

  23. [29]

    Soc: Semantic- assisted object cluster for referring video object segmentation

    Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yitong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic- assisted object cluster for referring video object segmentation. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  24. [30]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 3

    2023 arXiv

  25. [32]

    Mod- eling context between objects for referring expression under- standing

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression under- standing. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 792–807. Springer,

    2016

  26. [33]

    Perceptiongpt: Effectively fusing visual perception into llm

    Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into llm. arXiv preprint arXiv:2311.06612, 2023. 3

    2023 arXiv

  27. [34]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356, 2023. 2, 6

    2023 arXiv

  28. [35]

    Pixellm: Pixel rea- soning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel rea- soning with large multimodal model. ArXiv, abs/2312.02228,

    arXiv

  29. [36]

    xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms

    Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms. arXiv preprint arXiv:2410.16267, 2024. 3

    arXiv

  30. [37]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16 , pages 208–223. Springer,

    2020

  31. [38]

    Trackgpt–a generative pre-trained trans- former for cross-domain entity trajectory forecasting

    Nicholas Stroh. Trackgpt–a generative pre-trained trans- former for cross-domain entity trajectory forecasting. arXiv preprint arXiv:2402.00066, 2024. 7

    2024 arXiv

  32. [39]

    Con- trastive grouping with transformer for referring image seg- mentation

    Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Con- trastive grouping with transformer for referring image seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23570– 23580, 2023. 3

    2023

  33. [40]

    Cris: Clip- driven referring image segmentation

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip- driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 11686–11695, 2022. 2, 3, 6 10

    2022

  34. [41]

    Lasagna: Language-based segmentation assistant for complex queries

    Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-based segmentation assistant for complex queries. arXiv preprint arXiv:2404.08506, 2024. 2, 3, 6

    2024 arXiv

  35. [42]

    Hyperseg: Towards universal visual segmentation with large language model

    Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: Towards universal visual segmentation with large language model. arXiv preprint arXiv:2411.17606, 2024. 3

    2024 arXiv

  36. [43]

    Onlinerefer: A simple online baseline for referring video object segmentation

    Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. Onlinerefer: A simple online baseline for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2761–2770, 2023. 2, 7

    2023

  37. [44]

    Language as queries for referring video object segmentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4984, 2022. 2, 3, 7

    2022

  38. [45]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. arXiv preprint arXiv:2312.10103, 2023. 2, 3, 6

    2023 arXiv

  39. [46]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 3

    2024 arXiv

  40. [47]

    Visa: Reasoning video object segmentation via large language mod- els

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language mod- els. arXiv preprint arXiv:2407.11325, 2024. 2, 3, 4, 5, 6, 7

    2024 arXiv

  41. [48]

    Lavt: Language-aware vision transformer for referring image segmentation

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Heng- shuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155–18165, 2022. 2, 6

    2022

  42. [49]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 6

    2016

  43. [50]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1

    2023

  44. [51]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406.19389, 2024. 2, 6

    2024 arXiv

  45. [52]

    Coupalign: Coupling word-pixel with sentence-mask alignments for referring image segmentation

    Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, and Wei Ke. Coupalign: Coupling word-pixel with sentence-mask alignments for referring image segmentation. Advances in Neural Information Processing Systems , 35:14729–14742,

  46. [54]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3

    2023 arXiv

  47. [55]

    A comprehensive overhaul of multimodal assistant with small language models

    Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, and Jian Tang. A comprehensive overhaul of multimodal assistant with small language models. arXiv preprint arXiv:2403.06199,

    arXiv

  48. [56]

    Generalized decoding for pixel, image, and lan- guage

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and lan- guage. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages ...

  49. [57]

    A baseball catcher with an open mitt

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024. 6 11 InstructSeg: Unifying Instructed Visual Segmentatio...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.