Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RSVP shows that overlaying numbered grid lines on an image lets an off-the-shelf multimodal LLM turn abstract queries into coarse localizations, which a refCOCO-trained segmenter then converts into state-of-the-art zero-shot masks.

desk verdict Solid two-stage zero-shot reasoning segmentation pipeline, but the 'outperforming fine-tuned LISA' claim contradicts its own Table 2 and the config was tuned on the test split. read the letter →

arxiv 2506.04277 v1 pith:VWG444UB submitted 2025-06-04 cs.CV cs.AI

Yi Lu , Jiawang Cao , Yongliang Wu , Bozheng Li , Licheng Tang , Yangguang Ji , Chong Wu , Jay Wu
show 1 more author
Wenbo Zhu
This is my paper
classification cs.CVcs.AI
keywords reasoningsegmentationmultimodalchain-of-thoughtvisualpromptingregionproposalreferringzero-shotlargelanguagemodelsSAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reasoning segmentation—producing a pixel mask for the object an abstract query implies—has usually meant fine-tuning a large model on segmentation data. This paper argues that the task can be solved without such training by splitting it into two stages connected by a simple spatial interface. A multimodal LLM is shown the image overlaid with numbered horizontal and vertical grid lines and is prompted through a chain of thought to name the object, say whether it appears, and return the IDs of every strip touching it. A second module, a referring segmenter trained only on refCOCOg, then crops that region and turns the object name into a precise mask. On the ReasonSeg benchmark the best variant reaches 60.3 generalized IoU and 60.0 cumulative IoU zero-shot, and 49.7 mean average precision on SegInW, numbers that land near or above fine-tuned baselines. The point is that explicit reasoning and visual grounding can be engineered through prompting instead of learned end-to-end.

What carries the argument

The load-bearing mechanism is the region-aware multimodal chain-of-thought visual prompt. The input image is divided into $N$ vertical and $N$ horizontal numbered strips; an MLLM is given two such annotated images and a step-by-step prompt that asks it to infer the object, decide whether it exists, list every vertical and horizontal region ID that contains part of it, and justify the answer. From those IDs the pipeline forms a padded bounding box, crops the image, and hands the crop plus the object description to the Vision-Language Segmentation Module, where BEiT-3 encodes image and text jointly and the SAM prompt encoder and decoder produce the mask. The grid density (the $9\times 9$ split is best), the padding (20% is best), and the prompt structure (manual multi-step beats plain CoT) are empirical knobs on this same mechanism.

What would settle it

Keep VLSM fixed on the ReasonSeg test set and replace the first-stage MLLM's ids_vertical and ids_horizontal with random IDs drawn from the same distribution while keeping the object name. If cIoU stays near the reported 60.0, the region proposal is not doing the work; if it collapses, the localization prompt is load-bearing.

Watch

Extended reading notes

Core claim

The paper's central discovery is that an off-the-shelf multimodal LLM can act as the reasoning-and-localization front end of a segmentation system if the query is structured as a chain of thought over a numbered grid. The image is split into $N$ horizontal and $N$ vertical strips, each labeled, and the model is asked to output the object's short description, the vertical and horizontal IDs that touch any part of it, and a rationale; an empty ID list is the prescribed answer when the object is absent. Those IDs define a padded bounding box, and the second stage—a vision-language segmenter built on the Segment Anything Model's prompt encoder and decoder with BEiT-3 as the joint text-image encoder—refines the crop into the final mask. RSVP reports state-of-the-art zero-shot results on ReasonSeg (gIoU 60.3, cIoU 60.0 with GPT-4o) and SegInW (mAP 49.7), and its ablations show that the region proposal, the two-modality distillation, the 9-by-9 grid density, and the structured chain-of-thought prompt each contribute several points of cIoU. The interpretation the authors offer is that reasoning segmentation decomposes into query comprehension plus localization, and both can be elicited rather than trained.

Load-bearing premise

The pipeline assumes that an off-the-shelf multimodal LLM, given only numbered grid lines and the prompt, will reliably follow the chain-of-thought, infer the correct object, and return the right vertical and horizontal region IDs; if it mislocalizes or returns empty IDs, the segmentation stage cannot recover.

Editorial extensions

If this is right

  • A new or stronger MLLM can be dropped into the reasoning stage without retraining the segmenter; RSVP-GPT outperforms RSVP-LLaVA and RSVP-Qwen with the same VLSM.
  • Fine-tuning the segmentation stage on reasoning data is not required for state-of-the-art zero-shot performance: VLSM trained only on refCOCOg suffices.
  • The 9-by-9 region density and 20% padding are non-trivial design choices; deviating from them (13-by-13 or 0% padding) measurably lowers cIoU.
  • If the first-stage model cannot reason or localize (e.g., the 2B Qwen model), performance falls to 45.8 gIoU, so the framework inherits the MLLM's ceiling.
  • Because no LLM weights are modified, the same prompt can be reused as stronger closed-source models appear; the paper explicitly notes GPT-4o outperforms Gemini-Flash and LLaVA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The numbered-grid interface could be translated to other output axes—for instance, replacing vertical and horizontal IDs with temporal clip IDs for video grounding—an extension the paper does not test.
  • If the bottleneck truly is the first-stage region proposal, then a consistency check between the final mask and the proposed bounding box could serve as a cheap confidence score for deployment, which the paper does not discuss.
  • Benchmark numbers tied to GPT-4o and Gemini-Flash will drift as those API models are updated; the paper's specific 60.3/60.0 figures are a snapshot rather than a stable property of the method, whereas the modular design claim would still hold.
  • A testable implication of the paper's existence-reasoning step is that negative queries (object absent) should return empty masks; measuring precision on such queries would extend the framework into a rejection-capable segmenter, which the paper leaves for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The manuscript proposes RSVP, a two-stage framework for reasoning segmentation under zero-shot settings. In the first stage, an off-the-shelf MLLM (e.g., GPT-4o) is prompted with a multi-modal chain-of-thought visual prompt: the image is overlaid with horizontal/vertical grid cells with numeric IDs, and the model is asked to reason about the query, output the target object, the set of grid-cell IDs containing the object, and a rationale. In the second stage, a Vision-Language Segmentation Module (VLSM), built from BEiT-3 and SAM and trained on refCOCO, crops the proposed region and produces a mask. The paper reports state-of-the-art zero-shot results on ReasonSeg (Test Overall 60.3 gIoU / 60.0 cIoU with GPT-4o) and on SegInW (49.7 mAP), with ablations on grid density, padding, CoT prompt design, MLLM choice, and temperature.

Significance. If the reported results are supported by a clean protocol, the paper makes a useful contribution: it shows that a modular, training-free (on reasoning-segmentation data) pipeline with an MLLM can match or exceed the zero-shot performance of end-to-end trained LISA variants, and the visual-prompting idea is simple and interpretable. The paper is honest about limitations (MLLM dependence, robustness), provides detailed ablation tables, and includes the full prompt in the appendix. However, two protocol issues prevent me from considering the headline SOTA claim reliable at this stage: the reported configuration was tuned on the ReasonSeg test split, and the prose explicitly claims superiority over fine-tuned models that is contradicted by the paper's own Table 2.

major comments (2)
  1. [§4.3, Table 2] The sentence 'RSVP achieves state-of-the-art gIoU and cIoU in zero-shot settings, outperforming fine-tuned models without additional training' is internally inconsistent with Table 2, where LISA-13B (ft) attains Test Overall 61.3 gIoU / 62.2 cIoU and Val Overall 65.0 / 72.9, above RSVP-GPT's 60.3 / 60.0 and 64.7 / 63.1. The claim should be restricted to zero-shot baselines or to the 7B fine-tuned variants; otherwise the central SOTA assertion is overstated.
  2. [§4.4 and Appendix B.1, Tables 7–9] The reported RSVP configuration was selected using the ReasonSeg-Test split: the 9×9 grid density (Table 7), the 20% padding (Table 9), the manual CoT prompt (Table 8), and the choice of GPT-4o among MLLMs are all determined by their performance on the same Test split used for the headline numbers. The sensitivity is material (for GPT-4o, 9×9 gives 63.1 cIoU versus 59.6 for 5×5; 20% padding gives 60.0 versus 57.1 for 0%), relative to the claimed +9.2 cIoU advantage over zero-shot LISA-13B. This selection effect means the reported 'state-of-the-art zero-shot' numbers are not supported by a clean evaluation protocol. Please either hold out a validation split for hyperparameter selection or report all tested configurations with the selection procedure.
minor comments (5)
  1. [§3.2 and §3.4] The notation in Eqs. (1)–(2) is confusing: idh and idv are defined, but the formulas swap logical roles of py and px, and the crop-size equations in §3.4 mix H/M and W/N with the horizontal/vertical indices. Please clarify the coordinate conventions.
  2. [§3.5 and §4.1] The test split is described as 'around 770 samples' in §4.1 but as 754 images in §3.5; please reconcile the numbers.
  3. [Table 2 caption] The caption says 'Columns with gray backgrounds indicate training-free methods, while grayed-out columns are LISA models fine-tuned on ReasonSeg Training split'; as printed it is unclear which rows or columns are being referenced, so please reformulate the caption.
  4. [Throughout] There are several typos, e.g., 'firsr-stage' in §3.5, 'summarativedescription' in Figure 9, 'Comprasion' in Table 5, and 'LlSA' in Figure 1; a careful proofreading pass is needed.
  5. [Table 1] The layout of Table 1 is garbled, with model sizes and timings interleaved in the same column; please reformat it into separate columns.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RSVP's benchmark results are computed against external ground-truth masks and do not reduce to its own inputs by construction.

full rationale

The paper's load-bearing claims are empirical: RSVP uses an off-the-shelf MLLM with a grid-based CoT prompt to produce region IDs, then crops the image and passes it to a VLSM trained on refCOCO to produce masks. The reported metrics (gIoU, cIoU on ReasonSeg; mAP on SegInW; cIoU on refCOCOg) are all evaluated against external, independently annotated ground-truth masks, not against RSVP's own outputs or fitted values. No equation in Sec. 3.2-3.4 defines the target metric in terms of the model's predictions, and no fitted parameter is renamed as a prediction. The ablations in Sec. 4.4 and Appendix B select hyperparameters such as grid density, padding ratio, and prompt style on the ReasonSeg-Test split; this is a benchmark-protocol concern about selection on the test set, not a circular reduction, because the final numbers still require the external ground-truth masks to be computed. Self-citations (e.g., Reframe Anything, Veu-Bench, Video Repurposing, Number It) appear in related-work context and are not load-bearing for the zero-shot SOTA claim. The limitations section explicitly acknowledges MLLM dependence and generalization risks, which further confirms that the results are presented as empirical findings rather than as consequences of a definition or a prior author-derived theorem. Therefore the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method is empirical and reuses existing models and benchmarks. The central claim rests on the MLLM's zero-shot region-ID localization, on VLSM's transfer from refCOCO, and on several hyperparameters chosen by ablation on the evaluation benchmark rather than on a held-out validation set.

free parameters (4)
  • Region division density n = 9x9
    Selected by ablation over 5x5, 9x9, and 13x13 grids (Table 7); 9x9 gave the best cIoU for both LLaVA and GPT-4o.
  • Crop padding ratio p = 20% of region height/width
    Selected by padding ablation on ReasonSeg-Test (Table 9); 0% and 40% both gave lower gIoU and cIoU.
  • CoT prompt template = Prompt A, the manual hierarchical prompt
    Chosen over a plain CoT prompt (Table 8) and over an adapted literature CoT prompt (Table 10) based on benchmark performance.
  • VLSM training schedule = 16,000 epochs (likely steps), batch 256, lr 1e-4
    Stated in Sec. 4.2; the value is internally implausible and exact reproduction requires a corrected training recipe.
assumptions (4)
  • domain assumption Off-the-shelf MLLMs follow the grid-overlay prompt and return accurate region IDs for objects named by common-sense reasoning.
    Invoked in Sec. 3.2-3.3 with the prompt in Fig. 9; the entire first stage depends on this behavior.
  • domain assumption A segmentation model trained on refCOCO (VLSM) transfers to ReasonSeg and SegInW categories without task-specific tuning.
    Used in Sec. 3.4 and Sec. 4.3; the zero-shot claims rest on this transfer.
  • domain assumption The MLLM's region IDs and object-name text are sufficient localization input for the second-stage segmentation model.
    The crop formula in Sec. 3.4 uses only the grid-derived box; if the object extends beyond that box, the final mask cannot recover.
  • standard math The standard benchmark metrics gIoU, cIoU, and mAP faithfully measure reasoning segmentation quality.
    Assumed by the evaluation setup in Sec. 4.1; note that the text mistakenly describes gIoU/cIoU with bounding boxes rather than masks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought." pith.science (2026). https://pith.science/paper/VWG444UB

@misc{pith2026250604277,
  author       = {Pith},
  title        = {Pith review of: RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWG444UB}},
  note         = {Machine review of arXiv:2506.04277}
}
read the original abstract

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable reasoning capability while lack explicit mechanisms for visual grounding and segmentation, creating a gap between cognitive reasoning and visual perception. To bridge this gap, we introduce Reasoning Segmentation via Visual Prompting (RSVP), a novel framework that unifies multi-step multimodal reasoning with grounded visual understanding. RSVP is a two-stage structuralized framework that integrates reasoning-driven localization with segmentation refinement. In the reasoning stage, RSVP employs multimodal chain-of-thought visual prompts to help MLLMs understand queries and infer targets, generating interpretable region proposals that enhance visual grounding. In segmentation stage, RSVP refines these proposals with a Vision-Language Segmentation Module (VLSM), seamlessly integrates textual and visual cues to produce precise segmentation masks. By explicitly modelling the interaction between multimodal reasoning and segmentation, RSVP introduces a new paradigm for interpretable reasoning segmentation. It exploits MLLMs' inherent localization capabilities, enabling the models to not only reason about objects but also generate structured visual representations. Our extensive experiments demonstrate that RSVP achieves state-of-the-art performance, surpasses state-of-the-art methods by up to +6.5 gIoU and +9.2 cIoU on ReasonSeg, and achieves 49.7 mAP on SegInW under zero-shot settings. These results validate RSVP as an effective and scalable framework for integrating cognitive reasoning with structured visual understanding.

Figures

Figures reproduced from arXiv: 2506.04277 by the authors.

Figure 1
Figure 1. (a) and (b) depict different aspects of our segmenta￾tion pipeline performance. More demo results are available in Sec. C. a key challenge remains unresolved: bridging the gap between cognitive reasoning and visual seg￾mentation. Reasoning Segmentation has emerged as a cru￾cial task in multi-modal grounding, requiring mod￾els to produce segmentation masks from complex, implicit textual queries (Lai et al., 2024). Un… view at source ↗
Figure 2
Figure 2. Overview of the proposed model. An input image is divided into horizontal and vertical regions to assist localization. In the reasoning stage, an MLLM receives a query about the object’s protective features and identifies ”the shell” as the protective object, generating region proposal using region IDs (idv and idh). Red boxes indicate the regions of interest determined by the MLLM, yellow box denotes the padding p … view at source ↗
Figure 3
Figure 3. Illustration of the CoT processing strategy in action for a query about a dragon boat race. prompt to explicitly structure spatial queries. As il￾lustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 6
Figure 6. Figure 6: Illustration of Low-quality Segmentation mask cases produced by LISA. C.2 Case: Low-Quality Segmentation Mask [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 4
Figure 4. Figure 4: Illustration of Incorrect Localization cases produced by LISA. Query: Looking at the image, it seems that the dog is engaged in a playful activity. What part of the dog's body is likely to be holding an object during this playful activity? RSVP Reasoning: The dog's mou…
Figure 5
Figure 5. Figure 5: Illustration of Low-quality Segmentation mask cases produced by LISA. C.1 Case: Incorrect Localization [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 7
Figure 7. Figure 7: Illustration of bad cases caused by false localization or prompt misinterpretation. As shown in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 11
Figure 11. Figure 11: Visual prompt example: the image is evenly di￾vided into vertical/horizontal regions. The dividing density is set to five for demonstrative purposes [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 9
Figure 9. Figure 9: The prompt utilized to query MLLMs in our implementation [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The prompt utilized to experiment on CoT framework ablation experiment [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 12
Figure 12. Figure 12: Visualization of produced visual prompts and region proposals for demonstrated cases [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.

  2. A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

    cs.CV 2025-12 reject novelty 6.0 of 10

    A 'tool bottleneck' framework—VLM tool selection plus learned spatial fusion—matches or beats black-box classifiers, especially on scarce data.

Reference graph

Works this paper leans on

57 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. https://api.semanticscholar.org/CorpusID:269449935 Hallucination of multimodal large language models: A survey . ArXiv, abs/2404.18930

    arXiv 2024

  2. [2]

    Jiawang Cao, Yongliang Wu, Weiheng Chi, Wenbo Zhu, Ziyue Su, and Jay Wu. 2024. https://api.semanticscholar.org/CorpusID:268357047 Reframe anything: Llm agent for open world video reframing . ArXiv, abs/2403.06070

    work page Pith review arXiv 2024

  3. [3]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. 2025. https://api.semanticscholar.org/CorpusID:276937570 Towards reasoning era: A survey of long chain-of-thought for reasoning large language models . ArXiv, abs/2503.09567

    arXiv 2025

  4. [4]

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024. https://api.semanticscholar.org/CorpusID:270062772 M3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought . In Annual Meeting of the Association for Computational Linguistics

    work page 2024

  5. [5]

    Anurag Das, Xinting Hu, Li Jiang, and Bernt Schiele. 2024. Mta-clip: Language-guided semantic segmentation with mask-text alignment. In European Conference on Computer Vision, pages 39--56. Springer

    work page 2024

  6. [6]

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. https://api.semanticscholar.org/CorpusID:236987044 Vision-language transformer and query generation for referring segmentation . 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 16301--16310

    work page 2021

  7. [7]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    arXiv 2021

  8. [8]

    Xinting Hu, Li Jiang, and Bernt Schiele. 2024. Training vision transformers for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4007--4017

    work page 2024

Show all 57 references
  1. [9]

    Berg, Wan-Yen Lo, Piotr Doll \'a r, and Ross B

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll \'a r, and Ross B. Girshick. 2023. https://api.semanticscholar.org/CorpusID:257952310 Segment anything . 2023 IEEE/...

    2023

  2. [10]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  3. [11]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579--9589

    2024

  4. [12]

    Bozheng Li, Yongliang Wu, Yi Lu, Jiashuo Yu, Licheng Tang, Jiawang Cao, Wenqing Zhu, Yuyang Sun, Jay Wu, and Wenbo Zhu. 2025. https://api.semanticscholar.org/CorpusID:278130017 Veu-bench: Towards comprehensive understanding of video editing . ArXiv, abs/2504.17828

    2025 arXiv

  5. [13]

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. 2024. https://api.semanticscholar.org/CorpusID:268793990 Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want . ArXiv, ...

    2024 arXiv

  6. [14]

    Chang Liu, Henghui Ding, and Xudong Jiang. 2023 a . https://api.semanticscholar.org/CorpusID:258999674 Gres: Generalized referring expression segmentation . 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23592--23601

    2023

  7. [15]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . https://doi.org/10.48550/arXiv.2304.08485 Visual Instruction Tuning . Preprint, arXiv:2304.08485

    2023 doi arXiv

  8. [16]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. 2023 c . https://api.semanticscholar.org/CorpusID:257427307 Grounding dino: Marrying dino with grounded pre-training for open-set object detec...

    2023 arXiv

  9. [17]

    Ilya Loshchilov and Frank Hutter. 2017. https://api.semanticscholar.org/CorpusID:53592270 Decoupled weight decay regularization . In International Conference on Learning Representations

    2017

  10. [18]

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. 2020. https://api.semanticscholar.org/CorpusID:213176225 Multi-task collaborative network for joint referring expression comprehension and segmentation . 2020 IEEE/CVF Conference on Compu...

    2020

  11. [19]

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana-Maria Camburu, Alan Loddon Yuille, and Kevin P. Murphy. 2015. https://api.semanticscholar.org/CorpusID:8745888 Generation and comprehension of unambiguous object descriptions . 2016 IEEE Conference on Computer Vision and Patte...

    2015

  12. [20]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, and Lama Ahmad. 2024. https://doi.org/10.48550/arXiv.2303.08774 GPT-4 Technical Report . Preprint, arXiv:2303.08774

    2024 doi arXiv

  13. [21]

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. 2025. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536

    2025 arXiv

  14. [22]

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, and Tat-Seng Chua. 2024. https://api.semanticscholar.org/CorpusID:270371381 Unified text-to-image generation and retrieval . ArXiv, abs/2406.05814

    2024 arXiv

  16. [24]

    Muhammad Faza Ilmanuddiin Rachmadi, Salman Shalahuddin, Heru Permana, Urip Teguh Setijohatmo, and Jonner Hutahaean. 2023. Xlm-roberta model for key information extraction on military document. In 2023 10th International Conference on ICT for Smart Society (ICISS), pages 1--6. IEEE

    2023

  17. [25]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  18. [26]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. https://api.semanticscholar.org/CorpusID:221191193 Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters . Proceedings of the 26th ACM SIGKDD International C...

    2020

  19. [27]

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. https://api.semanticscholar.org/CorpusID:267212047 Grounded sam: Asse...

    2024 arXiv

  20. [28]

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024. https://api.semanticscholar.org/CorpusID:271051212 Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reaso...

    2024

  21. [29]

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. https://doi.org/10.1109/ICCV51070.2023.01101 What does CLIP know about a red circle? Visual prompt engineering for VLMs . In 2023 IEEE / CVF International Conference on Computer Vision ( ICCV ) , pages 1195...

    2023

  22. [30]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, and Jiahui Yu. 2024. https://doi.org/10.48550/arXiv.2312.11805 Gemini: A Family of Highly Capable Multimodal Models . Preprint, arXiv:2312.11805

    2024 doi arXiv

  23. [31]

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. 2023. https://api.semanticscholar.org/CorpusID:264439297 Sam-clip: Merging vision foundation models towards ...

    2023

  24. [32]

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442

    2022 arXiv

  25. [33]

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yan Guo, Ming Gong, and Tongliang Liu. 2021. https://api.semanticscholar.org/CorpusID:244729320 Cris: Clip-driven referring image segmentation . 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11676--11685

    2021

  26. [34]

    Zhaoqing Wang, Xiaobo Xia, Ziye Chen, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. 2024. Open-vocabulary segmentation with unpaired mask-text supervision. arXiv preprint arXiv:2402.08960

    2024 arXiv

  27. [35]

    Xia, Quoc Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. https://api.semanticscholar.org/CorpusID:246411621 Chain of thought prompting elicits reasoning in large language models . ArXiv, abs/2201.11903

    2022 arXiv

  28. [36]

    Gee, and Yixin Nie

    Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oğuz, James C. Gee, and Yixin Nie. 2023. https://api.semanticscholar.org/CorpusID:265212798 The role of chain-of-thought in complex vision-language reasoning task . ArXiv, abs/2311.09193

    2023 arXiv

  29. [37]

    Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, and Tong He. 2024 a . https://arxiv.org/abs/2403.12488 DetToolChain : A New Prompting Paradigm to Unleash Detection Ability of MLLM . Preprint, arXiv:2403.12488

    2024 arXiv

  30. [38]

    Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. 2024 b . Number it: Temporal grounding videos like flipping manga. arXiv preprint arXiv:2411.10332

    2024 arXiv

  31. [39]

    Yongliang Wu, Wenbo Zhu, Jiawang Cao, Yi Lu, Bozheng Li, Weiheng Chi, Zihan Qiu, Lirian Su, Haolin Zheng, Jay Wu, and Xu Yang. 2024 c . https://api.semanticscholar.org/CorpusID:274656300 Video repurposing from user generated content: A large-scale dataset and benchmark . ArXiv...

    2024 arXiv

  32. [40]

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023. https://api.semanticscholar.org/CorpusID:257405338 Open-vocabulary panoptic segmentation with text-to-image diffusion models . 2023 IEEE/CVF Conference on Computer Vision and Pattern R...

    2023

  33. [41]

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. 2023. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325--15336

    2023

  34. [42]

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun yue Li, and Jianfeng Gao. 2023 a . https://api.semanticscholar.org/CorpusID:266149987 Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v . ArXiv, abs/2310.11441

    2023 arXiv

  35. [43]

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. 2023 b . An improved baseline for reasoning segmentation with large language model. arXiv e-prints, pages arXiv--2312

    2023

  36. [44]

    Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. 2023 c . Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36:40924--40943

    2023

  37. [45]

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip H. S. Torr. 2021. https://api.semanticscholar.org/CorpusID:244909191 Lavt: Language-aware vision transformer for referring image segmentation . 2022 IEEE/CVF Conference on Computer Vision and Pattern Re...

    2021

  38. [46]

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023 d . https://doi.org/10.48550/arXiv.2309.17421 The Dawn of LMMs : Preliminary Explorations with GPT-4V (ision) . Preprint, arXiv:2309.17421

    2023 doi arXiv

  39. [47]

    Congzhi Zhang, Linhai Zhang, Jialong Wu, Yulan He, and Deyu Zhou. 2025. Causal prompting: Debiasing large language model prompting based on front-door adjustment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25842--25850

    2025

  40. [48]

    Congzhi Zhang, Linhai Zhang, and Deyu Zhou. 2024 a . Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19533--19541

    2024

  41. [49]

    Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. 2024 b . Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14227--14238

    2024

  42. [50]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alexander J. Smola. 2023. https://api.semanticscholar.org/CorpusID:256504063 Multimodal chain-of-thought reasoning in language models . Trans. Mach. Learn. Res., 2024

    2023

  43. [51]

    Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. 2024. https://doi.org/10.48550/arXiv.2405.13872 Image-of- Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models . Preprint, arXiv:2405.13872

    2024 doi arXiv

  44. [52]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. https://doi.org/10.48550/arXiv.2304.10592 MiniGPT-4 : Enhancing Vision-Language Understanding with Advanced Large Language Models . Preprint, arXiv:2304.10592

    2023 doi arXiv

  45. [53]

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. 2023. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2023

  46. [54]

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Singh Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. 2022. https://api.semanticscholar.org/CorpusID:254926770 Generalized decoding for pixel, ima...

    2022

  47. [55]

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2024. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36

    2024

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.