Pith. sign in

REVIEW 5 major objections 5 minor 52 references

Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a vision-language model on captions with marked landmarks lifts zero-shot Visual Spatial Reasoning accuracy to 56.14%, a reported 3.15-point gain over prior methods.

desk verdict The SUN-Spot v2.0 dataset is a real resource, but the headline VSR gain is not established because the benchmark was rewritten and no SoM-matched baseline exists. read the letter →

arxiv 2505.12194 v1 pith:FUM3P6KK submitted 2025-05-18 cs.RO

Xuefei Sun , Doncey Albin , Cecilia Mauceri , Dusty Woods , Christoffer Heckman This is my paper · ORCID
classification cs.RO
keywords SpatialreferringexpressionsSet-of-MarkspromptingMultimodallargelanguagemodelsVisualreasoningSUN-Spotv2.0datasetRGB-Dgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to show that multimodal large language models can be taught spatial referring expressions by training on a dataset that annotates not just the target object but also the landmark objects used in descriptions. The authors build SUN-Spot v2.0, 90,000 image-caption pairs over 10,000 RGB-D indoor images, and augment every caption with Set-of-Marks tags that tie each mention to a marked region. Fine-tuning LLaVA v1.5 on 75,000 question-answer pairs generated from this data yields 56.14% accuracy on the zero-shot Visual Spatial Reasoning benchmark, which the paper reports as a 3.15% gain over previous methods. A sympathetic reading is that the gain comes from aligning language to marked landmarks, not from object semantics.

What carries the argument

The load-bearing mechanism is the Set-of-Marks (SoM) prompt: small alphanumeric markers are placed at the center of each object in an image, and the same markers are appended to object mentions in captions and questions. This creates an explicit pointer from language to visual region, which the paper argues weakens reliance on semantic cues such as object class and color. Around it sits the SUN-Spot v2.0 dataset, built by extending SUN RGB-D with spatial referring expressions and landmark annotations, and conversation data generated from those captions by a language model. The fine-tuning keeps the visual encoder frozen and updates only the projector and language model on single-turn conversations so answers do not leak between questions.

What would settle it

Run Spatial-LLaVA 13b on the original Visual Spatial Reasoning test protocol (caption-label pairs with True/False labels and no set-of-marks tags) and compare with the rewritten-protocol score; a large drop indicates the headline gain depends on the question reformatting rather than spatial understanding.

Watch

Extended reading notes

Core claim

The central claim is that object-level grounding through Set-of-Marks prompting, applied at training and evaluation, lets an MLLM learn spatial relations with less bias from object identity. On the paper's own benchmark, SUN-Spot v2.0 Expert, Spatial-LLaVA 13b reaches 76.13% accuracy versus 66.22% for LLaVA v1.5 13b, and on the Visual Spatial Reasoning test set it reaches 56.14% versus 52.37%. This is presented as evidence that landmark-level annotations in the training data, not larger backbones or additional semantic cues, drive the improvement.

Load-bearing premise

The assumption that rewriting Visual Spatial Reasoning captions into yes/no questions with set-of-marks tags leaves the task's difficulty unchanged.

Editorial extensions

If this is right

  • A dataset with landmark-level annotations is enough to improve zero-shot spatial reasoning in a general MLLM without retraining from scratch.
  • SoM prompting can be applied at test time in the same form used in training, making the improvement applicable to any image with detected or ground-truth regions.
  • Single-turn conversation fine-tuning reduces the model's ability to infer answers from previous questions, so reported gains are more likely to reflect per-question spatial judgment.
  • The 56.14% Visual Spatial Reasoning result suggests spatial language understanding can be improved by training data rather than by larger visual encoders or larger language backbones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Visual Spatial Reasoning evaluation rewrote the benchmark's 'True'/'False' captions into 'Yes'/'No' questions and inserted SoM tags; if that reformatting changes difficulty, the 3.15-point gain may partly measure format compatibility rather than spatial reasoning.
  • The reported improvement should be tested on the untouched Visual Spatial Reasoning protocol, with no question reconstruction and no SoM tags, before treating 56.14% as a true zero-shot spatial reasoning score.
  • Since the dataset is RGB-D, the same fine-tuning pipeline could be applied to depth-based tasks such as grasping or navigation instruction following, which the paper names as future work.
  • A cheaper test of the SoM hypothesis would be to train the same model on SUN-Spot v2.0 captions with markers removed and compare Visual Spatial Reasoning accuracy; if the gap disappears, the markers, not the captions, carry the benefit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces SUN-Spot v2.0, a dataset of RGB-D images with spatial referring expressions and landmark-object annotations, and proposes Spatial-LLaVA, an MLLM obtained by fine-tuning LLaVA on GPT-4o-generated conversational data that incorporates Set-of-Marks (SoM) prompting. The model is evaluated on a held-out split of SUN-Spot v2.0 Expert and on the Visual Spatial Reasoning (VSR) benchmark, reporting accuracy gains over several baselines, including a 3.15% improvement on VSR claimed in the abstract.

Significance. If the results hold, the dataset contribution is potentially valuable: SUN-Spot v2.0 provides landmark annotations for spatial referring expressions, and the SoM-based fine-tuning idea is plausible for reducing semantic bias. The paper includes a human annotation pipeline with second-round review, which is a strength. However, the evidence for the central claims is weakened by evaluation-protocol issues: the VSR benchmark is reformatted and augmented with SoM markers without matched baselines, the SUN-Spot Expert comparison is between a fine-tuned model and zero-shot baselines, and no ablation isolates the effect of SoM. The machine-generated component of the dataset is described but does not appear to be used in training or evaluation. These issues are addressable with additional experiments and clarifications, so the underlying contribution is defensible but not yet convincingly supported.

major comments (5)
  1. [Section IV-D2, Table IV] The VSR evaluation does not follow the original benchmark protocol: the authors reconstruct the caption-label True/False entailment pairs into yes/no questions and add SoM markers to the images. No baseline is evaluated under the same rewritten format, so the reported gain of 56.14% versus 52.37% for LLaVA-13b may reflect alignment between the modified test format and the model's single-turn yes/no fine-tuning data, plus the extra SoM grounding, rather than improved spatial reasoning. Please run all baselines on the same rewritten questions with the same SoM overlay, and also report results on the original True/False entailment format.
  2. [Section IV-D1, Table III] Table III compares Spatial-LLaVA, which is fine-tuned on 75k question-answer pairs derived from SUN-Spot v2.0 Expert and SUNRefer, against baselines that are evaluated zero-shot (no fine-tuning on the SUN-Spot training split). The claimed 7.5% (7b) and 9.91% (13b) improvements over LLaVA v1.5 are therefore not like-for-like comparisons. The authors should fine-tune at least one baseline on the same training conversations, or re-frame the claim as 'zero-shot baselines vs. supervised fine-tuning on the same distribution.'
  3. [Section III-A2, Section III-C, Section IV-B] The SUN-Spot v2.0 Machine-generated dataset is introduced as a contribution and described in detail, but the training pipeline in Section III-C and Section IV-B uses only SUN-Spot Expert and SUNRefer to generate 75k question-answer pairs. The machine-generated captions never appear in training or evaluation. Please clarify whether this component is used at all; if not, it should be repositioned as a dataset resource for future work rather than as part of the method's training data.
  4. [Abstract and Table IV] The abstract claims 'Spatial-LLaVA outperforms previous methods by 3.15% on the zero-shot Visual Spatial Reasoning benchmark dataset,' but Table IV reports 56.14% for Spatial-LLaVA-13b versus 52.37% for LLaVA-13b (a 3.77-point difference) and 53.60% versus 52.95% for the 7b models (a 0.65-point difference). No comparison in Table IV yields 3.15%. Please state precisely how the 3.15% figure is computed, or correct the abstract and any related claims.
  5. [Section III-B and Section IV-D2] The paper credits Set-of-Marks prompting with improving spatial understanding, but no ablation isolates its contribution. There is no experiment removing SoM from Spatial-LLaVA at inference, no experiment adding SoM to a baseline LLaVA model, and no comparison between training with and without SoM. Without such controls, the observed gains cannot be attributed to SoM rather than to the fine-tuning data or the question format.
minor comments (5)
  1. [Section IV-D2] Typo: 'prepossessing' should be 'preprocessing'.
  2. [Table I] The table header contains 'avgerage' (should be 'average'), and the number of expert descriptions is inconsistent between the text (7,987) and the table (7,990).
  3. [Table I] The machine-generated row reads '10, 333' with an extra space; also, the entry '90k image-caption pairs' in the abstract should be reconciled with the detailed counts in Table I and Table II.
  4. [Throughout] The model name is inconsistently rendered as 'Spatial-LLaV A' with a space before the final 'A' in several places; please standardize to 'Spatial-LLaVA'.
  5. [Section II-B] The sentence 'Thus, to perform well on these benchmarks, a trained model does not necessarily need to fully understand the spatial relationships' appears without context and seems to contradict the motivation of the paper; please clarify or remove it.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found; the VSR and SUN-Spot evaluations are empirical tests, though the VSR protocol rewrite and same-distribution held-out evaluation add minor self-referentiality.

full rationale

The paper's central claim is empirical: fine-tuning an MLLM on SoM-marked spatial conversation data yields higher accuracy on a held-out SUN-Spot split and on the external VSR test set. No equation or fitted parameter is defined in terms of the target metric, and no prediction is a renamed training statistic. The self-citation to SUN-Spot [35] is provenance for the dataset, not a load-bearing justification of the reported gains. The VSR evaluation does modify the original benchmark by reconstructing caption-label pairs into Yes/No questions and adding SoM prompting, and Table III evaluates on a held-out split of the same distribution used for training; these are external-validity and comparability concerns, not circular reductions. Under the strict standard of exhibiting a specific reduction of a claim to its inputs, no circular step is present. Score 2 reflects the minor self-referential protocol elements while confirming that the derivation itself is self-contained.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on dataset quality and benchmark comparability assumptions rather than on mathematical derivation; there are no invented physical entities. The main unexamined choices are the VSR rewriting protocol, the use of GPT-4o-generated conversations, and a held-out split of the training distribution used as a benchmark.

free parameters (2)
  • Fine-tuning learning rate and epochs = 2e-6, 3 epochs (plus 2e-3, 1 epoch for pretraining)
    Chosen by hand; no ablation shows how sensitive the VSR gain is to these values.
  • VSR question rewrite and SoM marker protocol = N/A (design choice, not numeric)
    The authors transformed VSR into binary yes/no questions and added SoM tags; this choice changes the task and is a hand-chosen protocol that affects the reported accuracy.
assumptions (4)
  • domain assumption Held-out SUN-Spot v2.0 Expert split measures spatial reasoning quality.
    Table III reports accuracy on a held-out split of the same dataset used for training; this assumes no leakage and that the split represents the task distribution.
  • domain assumption Rewriting VSR caption-label pairs into binary questions preserves benchmark validity.
    Section IV-D2; if the rewrite or added SoM markers change task difficulty, the zero-shot comparison is not apples-to-apples.
  • domain assumption GPT-4o generated conversations provide reliable supervision for spatial reasoning.
    Section III-A2 states quality declines with more complex prompts and depends on ground truth labels; the paper still uses GPT-4o output for 75k QA pairs.
  • domain assumption Set-of-Marks markers do not leak object identity in a way that inflates VSR performance.
    The markers are added to the input image and question text; the paper does not test whether this alone improves any model on VSR.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding." pith.science (2026). https://pith.science/paper/FUM3P6KK

@misc{pith2026250512194,
  author       = {Pith},
  title        = {Pith review of: Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FUM3P6KK}},
  note         = {Machine review of arXiv:2505.12194}
}
read the original abstract

Multimodal large language models (MLLMs) have demonstrated remarkable abilities in comprehending visual input alongside text input. Typically, these models are trained on extensive data sourced from the internet, which are sufficient for general tasks such as scene understanding and question answering. However, they often underperform on specialized tasks where online data is scarce, such as determining spatial relationships between objects or localizing unique target objects within a group of objects sharing similar features. In response to this challenge, we introduce the SUN-Spot v2.0 dataset1, now comprising a total of 90k image-caption pairs and additional annotations on the landmark objects. Each image-caption pair utilizes Set-of-Marks prompting as an additional indicator, mapping each landmark object in the image to the corresponding object mentioned in the caption. Furthermore, we present Spatial-LLaVA, an MLLM trained on conversational data generated by a state-of-the-art language model using the SUNSpot v2.0 dataset. Our approach ensures a robust alignment between the objects in the images and their corresponding object mentions in the captions, enabling our model to learn spatial referring expressions without bias from the semantic information of the objects. Spatial-LLaVA outperforms previous methods by 3.15% on the zero-shot Visual Spatial Reasoning benchmark dataset. Spatial-LLaVA is specifically designed to precisely understand spatial referring expressions, making it highly applicable for tasks in real-world scenarios such as autonomous navigation and interactive robotics, where precise object recognition is critical.

Figures

Figures reproduced from arXiv: 2505.12194 by the authors.

Figure 1
Figure 1. While LLaVA Agrees With Both Queries, Spatial-LLaVA Accu [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Top: SUN-Spot Expert, Middle: SUN-Spot Machine-generated, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. System Structure dataset # of annotations # of target objects # of objects per caption w/ image annotation Annotated on avgerage caption length ScanRefer[6] 51,583 11,046 1 704 scenes 20.3 Nr3D[1] 41,503 5,879 1 642 scenes 11.4 REVERIE[39] 21,702 4,140 1 90 scenes 18.0 SUNRefer[27] 38,495 7,699 1 7,699 RGBD images 16.3 SUN-Spot[35] 7,990 3,245 1 1,948 RGBD images 14.1 SUN-Spot v2.0 (expert) 7,990 3,245 2.91 1,948 RG… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Caption generation and annotation process. Top: SUN-Spot V2.0 Expert dataset, Bottom: SUN-Spot V2.0 Machine-generated dataset [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Caption Length Comparison model performance by allowing quick adaptation to task￾specific requirements without the need for extensive retrain￾ing from scratch. IV. EXPERIMENTS In this section, we will discuss the training setup and perform a comparative analysis of Spa…
Figure 6
Figure 6. Figure 6: Comparison of Spatial-LLaVA and LLaVA Across Datasets: SUN-Spot v2.0, SUNRefer, and VSR [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 33 canonical work pages

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16 , pages 422–440. Springer, 2020

    work page 2020

  2. [2]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    2023

  3. [3]

    Paligemma: A versatile 3b vlm for transfer, 2024

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias B...

    work page 2024

  4. [4]

    Meyer, Yuning Chai, and Yong Jae Lee

    Mu Cai, Haotian Liu, Dennis Park, Siva Karthik Mustikovela, Gre- gory P. Meyer, Yuning Chai, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts, 2024

    work page 2024

  5. [5]

    End-to-end object detec- tion with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detec- tion with transformers. arXiv:2005.12872 [cs] , 05 2020

    arXiv 2005

  6. [6]

    Chang, and Matthias Nießner

    Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language, 2020

    work page 2020

  7. [7]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Worts- man, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. 06 2023

    work page 2023

  8. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

Show all 52 references
  1. [9]

    Devlin, Jacob

    Aakanksha Chowdhery, Sharan Narang, and et al. Devlin, Jacob. Palm: Scaling language modeling with pathways. arXiv:2204.02311 [cs], 04 2022

    2022 arXiv

  2. [10]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, and et al.0 Longpre, Shayne. Scaling instruction-finetuned language models. arXiv:2210.11416 [cs] , 10 2022

    2022 arXiv

  3. [11]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  4. [12]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettle- moyer. Qlora: Efficient finetuning of quantized llms, 2023

    2023

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 10 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 10 2018

    2018

  6. [14]

    Eva: Exploring the limits of masked visual representation learning at scale, 12 2022

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale, 12 2022

    2022

  7. [15]

    Sebastian Borgeaud

    Gemini Team, Rohan Anil, and et al. Sebastian Borgeaud. Gemini: A family of highly capable multimodal models, 2024

    2024

  8. [16]

    Arzen-llm: Code-switched egyptian arabic-english translation and speech recognition using llms

    Ahmed Heakl, Youssef Zaghloul, Mennatullah Ali, Rania Hossam, and Walid Gomaa. Arzen-llm: Code-switched egyptian arabic-english translation and speech recognition using llms. In 6th International Conference on AI in Computational Linguistics , 2024

    2024

  9. [17]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei- Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016

    2016

  10. [18]

    What’s “up” with vision-language models? investigating their struggle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19785 , 2023

    2023 arXiv

  11. [19]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 787–798, 2014

    2014

  12. [20]

    Computational genera- tion of referring expressions: A survey

    Emiel Krahmer and Kees Van Deemter. Computational genera- tion of referring expressions: A survey. Computational Linguistics , 38(1):173–218, 2012

    2012

  13. [21]

    When to retrieve: Teaching llms to utilize information retrieval effectively, 05 2024

    Tiziano Labruna, Jon Ander Campos, and Gorka Azkune. When to retrieve: Teaching llms to utilize information retrieval effectively, 05 2024

    2024

  14. [22]

    Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models, 2023

    2023

  15. [23]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    2022

  16. [24]

    Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation, 2021

    2021

  17. [25]

    Lawrence Zitnick, and Piotr Doll´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll´ar. Microsoft coco: Common objects in context, 2015

    2015

  18. [26]

    Visual spatial reasoning, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023

    2023

  19. [27]

    Refer-it-in-rgbd: A bottom-up approach for 3d visual grounding in rgbd images

    Haolin Liu, Anran Lin, Xiaoguang Han, Lei Yang, Yizhou Yu, and Shuguang Cui. Refer-it-in-rgbd: A bottom-up approach for 3d visual grounding in rgbd images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6032– 6041, 2021

    2021

  20. [28]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  22. [30]

    Clevr- ref+: Diagnosing visual reasoning with referring expressions

    Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr- ref+: Diagnosing visual reasoning with referring expressions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4185–4194, 2019

    2019

  23. [31]

    Roberta: A robustly optimized bert pretraining approach, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, and Paul Allen. Roberta: A robustly optimized bert pretraining approach, 2019

    2019

  24. [32]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  25. [33]

    An embarrassingly simple approach for llm with strong asr capacity

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Yu Fan, Qian Chen, Siqi Zheng, Shiliang Zhang, and Xie Chen. An embarrassingly simple approach for llm with strong asr capacity

  26. [34]

    Generation and comprehension of unambiguous object descriptions, 2016

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions, 2016

    2016

  27. [35]

    Sun-spot: An rgb-d dataset with spatial referring expressions

    Cecilia Mauceri, Martha Palmer, and Christoffer Heckman. Sun-spot: An rgb-d dataset with spatial referring expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision Work- shops, pages 0–0, 2019

    2019

  28. [36]

    Domain terminology integration into machine translation: Leveraging large language models, 2023

    Yasmin Moslem, Gianfranco Romani, Mahdi Molaei, Rejwanul Haque, John Kelleher, and Andy Way. Domain terminology integration into machine translation: Leveraging large language models, 2023

    2023

  29. [37]

    Mmger: Multi-modal and multi- granularity generative error correction with llm for joint accent and speech recognition

    Bingshen Mu, Yangze Li, Qijie Shao, Kun Wei, Xucheng Wan, Naijun Zheng, Huan Zhou, and Lei Xie. Mmger: Multi-modal and multi- granularity generative error correction with llm for joint accent and speech recognition. arXiv preprint arXiv:2405.03152 , 2024

    2024 arXiv

  30. [38]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. https://arxiv.org/pdf/ 2303.08774v3, 2023. Accessed: 2024-07-29

    2023 arXiv

  31. [39]

    Reverie: Remote embodied visual referring expression in real indoor environments, 2020

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments, 2020

    2020

  32. [40]

    Learn- ing transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learn- ing transferable visual models from natural language supervision. arXiv:2103.00020 [...

    2021 arXiv

  33. [41]

    Sun rgb-d: A rgb-d scene understanding benchmark suite

    Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 567–576, 2015

    2015

  34. [42]

    Eva- clip: Improved training techniques for clip at scale, 03 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva- clip: Improved training techniques for clip at scale, 03 2023

    2023

  35. [43]

    Self-retrieval: Building an information retrieval system with one large language model

    Qiaoyu Tang, Jiawei Chen, Bowen Yu, Yaojie Lu, Cheng Fu, Haiyang Yu, Hongyu Lin, Fei Huang, Ben He, Xianpei Han, Le Sun, and Yongbin Li. Self-retrieval: Building an information retrieval system with one large language model

  36. [44]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lang...

    2023 arXiv

  37. [45]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Atten- tion is all you need, 2023

    2023

  38. [46]

    The generation of natural descrip- tions: corpus-based investigations of referring expressions in visual domains

    Henriette Anna Elisabeth Viethen. The generation of natural descrip- tions: corpus-based investigations of referring expressions in visual domains. PhD thesis, Macquarie University, 2022

    2022

  39. [47]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    2023

  40. [48]

    Xlnet: Generalized autoregressive pretraining for language understanding, 2019

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2019

    2019

  41. [49]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Com- puter Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69–

    2016

  42. [50]

    A joint speaker-listener-reinforcer model for referring expressions

    Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speaker-listener-reinforcer model for referring expressions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7282–7290, 2017

    2017

  43. [51]

    Prompting large language model for machine translation: A case study, 2023

    Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study, 2023

    2023

  44. [52]

    Toolqa: A dataset for llm question answering with external tools

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Sys- tems, volume 36, pag...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.