Pith. sign in

REVIEW 3 major objections 5 minor 58 references

LMM-Det: Make Large Multimodal Models Excel in Object Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A large multimodal model, with only a vision encoder, a projector, and a language model, can perform object detection at 47.5 AP on COCO without any specialist detection module.

desk verdict A solid empirical recipe for LMM-based detection that sets a new SOTA, but the per-category inference protocol changes the task and the 'no extra detection modules' claim is nested inside specialist-pretrained components. read the letter →

arxiv 2507.18300 v1 pith:IM64NQXJ submitted 2025-07-24 cs.CV

Jincheng Li , Chunyu Xie , Ji Ao , Dawei Leng , Yuhui Yin This is my paper · ORCID
classification cs.CV
keywords objectdetectionlargemultimodalmodelsrecallrateper-categorypromptingpseudo-labelinginstructiontuningCOCOzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to show that a large multimodal model, composed only of a vision encoder, a projector, and a large language model, can do vanilla object detection without any specialist detection module bolted on. The authors identify the main bottleneck as a low recall rate: the model produces far fewer usable boxes than specialist detectors. They attack that bottleneck with two moves: adjusting the training data distribution by merging pseudo-labels from a pretrained specialist detector with ground truth via non-maximum suppression, and an inference strategy that queries the model once per object category instead of asking for all boxes at once. On COCO val, the resulting LMM-Det reaches 47.5 AP, 66.5 AP50, and 63.6 AR@100 after fine-tuning, while preserving captioning and VQA ability in the four-stage variant. The practical stake is that a single generalist model could replace the detector-plus-LMM stacks currently used for grounded dialogue, robotics, and scene understanding.

What carries the argument

The mechanism is the combination of a high-resolution open-vocabulary visual encoder (OWLv2-ViT, 1008×1008 input, with visual tokens fed uncompressed), a linear projector, and a 7B language model; data distribution adjustment, in which a pretrained specialist detector generates pseudo-labels that are merged with COCO and Object365 ground truth by NMS, and the model is trained to emit each box's coordinates followed by a confidence score token; and inference optimization, where detection is decomposed into one autoregressive pass per category, each pass producing boxes for that single class, repeated over the 80 COCO categories. The reorganized multi-turn instruction data, with positive and negative category questions in a 1:1 ratio and per-turn and per-box randomization, keeps training consistent with this per-category inference. The load-bearing object is the per-category prompt: it raises recall by letting each decoding step focus on one class, which the authors show yields more and better proposals than a single all-objects query.

What would settle it

Run LMM-Det on COCO val with the identical training recipe but replace the per-category prompts with a single all-objects prompt; if AR@100 stays near 63.6 rather than falling toward the roughly 27.5 the authors attribute to a single-pass model, the recall mechanism claimed for inference optimization would be refuted. A second check is to evaluate without retraining on Object365's 365 categories using the same per-category prompts; if AR@100 collapses, the fixed-vocabulary assumption is confirmed as load-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that a large multimodal model possesses detection capability without any extra detection modules. Concretely, LMM-Det, trained from an OWLv2-ViT visual encoder, a linear projector, and a 7B language model, outputs object coordinates and class-specific confidence scores as plain text and reaches 47.5 AP, 66.5 AP50, and 63.6 AR@100 on COCO val after fine-tuning, compared with 38.7 AP for a re-trained LLaVA baseline and 55-57 AP for strong specialist detectors. The authors argue that the gap to specialists is caused mainly by insufficient recall, and they show that two targeted interventions, data distribution adjustment via pseudo-label merging and per-category inference prompting, raise AR@100 from 50.5 to 63.6 and AP from 38.7 to 47.5. They also show a variant trained on the standard 665K LLaVA instruction set keeps most of the detection gain (47.1 AP) while retaining image captioning and VQA performance.

Load-bearing premise

The load-bearing premise is that detection is evaluated against a fixed, known category vocabulary: inference runs one prompt per class, so if the label set is unknown, open-ended, or very large, the per-category prompting strategy cannot be applied as-is and the reported recall gains would not transfer.

Editorial extensions

If this is right

  • Specialist detection modules, such as region proposal networks, Grounding DINO, or other bolt-on detectors, are not required for an LMM to do vanilla object detection; LMM-Det's gains come from data and inference changes, not architecture additions.
  • Raising recall, rather than classification or localization quality, is the main lever for closing the LMM-specialist detection gap on fixed-vocabulary benchmarks.
  • A single model can serve detection, referring expression comprehension, captioning, and VQA: the Stage-IV variant keeps 47.1 AP on COCO while retaining captioning and VQA, and RefCOCO val improves from 81.4 to 85.7 when detection and REC are trained together.
  • Inference cost scales with category count: LMM-Det needs about 4.0 seconds per image with greedy decoding for 80 category passes on COCO, trading latency for accuracy compared with real-time specialist detectors.
  • Zero-shot detection on COCO improves to 24.5 AP after detection pretraining on Object365, suggesting the capability is learnable without any specialist component.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed-category assumption is the main boundary: at inference the model is asked one question per category, so on open-vocabulary benchmarks such as LVIS or on a very large label set the per-category strategy would multiply cost or require a different aggregation; the reported 47.5 AP should not be expected to transfer unchanged.
  • Because pseudo-labels come from a single specialist detector, the ceiling of the data distribution adjustment is partly set by that detector's own recall; a weaker pseudo-labeler would likely produce a smaller gain, which could be tested by swapping pseudo-label sources.
  • One testable extension is to cluster or batch categories per pass, for example asking for 'person, dog, handbag' in a single prompt, to cut the 80-fold inference cost while retaining most of the recall benefit, since the paper's argument only requires per-category focus, not literally one category.
  • The per-category design may interact with model scale and resolution; the authors use a 7B language model and 1008×1008 input, so it is an open question whether larger models or higher resolutions would narrow or widen the gap to specialist detectors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LMM-Det, a method that adapts a large multimodal model (OWLv2-ViT encoder, linear projector, Vicuna-7B) to perform COCO-style object detection without a dedicated detection head or region proposer. The authors identify low recall as the main bottleneck and address it with two techniques: data distribution adjustment (merging ground-truth labels with Salience-DETR pseudo-labels via NMS) and inference optimization (querying the model separately for each category). After Stage III fine-tuning, the method reports 47.5 AP on COCO val, and an optional Stage IV preserves some captioning and VQA ability. The paper claims that a large multimodal model can possess detection capability without extra detection modules.

Significance. If the central claim is accepted, the paper is a useful demonstration that an LMM can be turned into a competitive detector using only data and prompting changes, and the recall-oriented analysis is a valuable addition to the literature on LMMs and detection. The strengths include a clear three-stage training recipe, systematic ablations (Table 5), and released datasets, models, and code. However, the headline claim is materially qualified by the evaluation protocol and by the use of detection-pretrained components and specialist pseudo-labels, so the paper's significance depends on how these qualifications are resolved.

major comments (3)
  1. [Section 3.3 / Section 4.1 / Table 3] The inference protocol decomposes one 80-class detection problem into 80 per-category localization problems: each image is queried with 'Detect all the objects in the image that belong to the category set {cat}', and the model never performs class selection. The specialist detectors in Table 3 are evaluated in a single pass over all classes. The reported 47.5 AP is therefore not earned under the same task protocol and is not directly comparable to the specialist numbers in that table. The Abstract and Figure 1 should be reframed as category-conditioned localization, or the authors should provide a single-pass multi-class evaluation to support the 'vanilla object detection' claim.
  2. [Section 3.1 / Table 5] The claim of operating 'without any extra detection modules' is undercut by the choice of OWLv2-ViT as the visual encoder. OWLv2 is an open-vocabulary object detector, and its ViT backbone carries detection-oriented pretraining; Table 5 shows that swapping CLIP-ViT for OWLv2-ViT alone raises AP by 3.4 points (38.7 to 42.1). The phrase 'no extra detection modules' should be restricted to the absence of task-specific heads, proposers, or decoders, and the detection-pretrained encoder should be acknowledged as a form of specialist knowledge transfer.
  3. [Section 3.2 / Table 5] The data distribution adjustment trains the model to imitate pseudo-labels generated by Salience-DETR, a specialist detector, including the confidence scores assigned by that detector. The +2.1 AP attributed to DDA in Table 5 is therefore partly inherited from the specialist model that the paper compares against in Table 3. To support the claim that the LMM itself acquires detection capability, the paper should present a control with pseudo-labels from a different source or otherwise quantify how much of the gain is due to additional label coverage versus imitation of the specialist.
minor comments (5)
  1. [Appendix A] Appendix A states that 'we set a threshold for calculating AP and AR' but does not give the threshold value or explain whether this replaces the standard COCO AP sorting procedure. Please specify the exact evaluation protocol, since the reported AP values depend on it.
  2. [Section 2.1 / Figure 1 / Table 3] The evaluation protocol for LLaVA* is inconsistent: Section 2.1 and Appendix A say exploratory models output all bounding boxes simultaneously, while Figure 1 and its caption describe LLaVA* as being queried per class. Please clarify which protocol produced the 38.7 AP reported for LLaVA* in Table 3.
  3. [Table 2] The term 'zero-shot' is used for LMM-Det after Stage II training on Object365 and for other models evaluated with their released checkpoints; this is misleading. Please relabel the setting as 'zero-shot on COCO' and explicitly note the different training-data exposures of the compared models.
  4. [Section 4.4 / Table 4] The claim that LMM-Det 'maintains' captioning and VQA abilities is overly strong: after Stage IV, CIDEr drops from 108.9 to 99.0 and VQAv2 accuracy from 78.5 to 74.1. Please qualify the wording to reflect a measurable degradation.
  5. [Various] There are several typographical issues, including 'LLM-Det' in Section 3.2, inconsistent 'LLaV A' spacing, and 'OVLv2' in Table 5's header. A proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the test-time model is module-free and the COCO-val results are not forced by the Salience-DETR pseudo-labels or per-category prompting by construction.

full rationale

The paper's central claim is about test-time capability: LMM-Det uses only a visual encoder, projector, and LLM at inference, with no region proposer or specialist detector in the architecture (Figure 4c). Training does use Salience-DETR pseudo-labels and an OWLv2 detection-pretrained visual encoder, but these are data sources and backbone initializations, not a formal reduction of the output to those inputs; the model is trained with a standard next-token objective (Eq. 1) and must learn to emit coordinates, and Table 5 shows incremental contributions (OWLv2 +3.4 AP, DDA +2.1 AP, INO +3.3 AP) rather than an identity between teacher outputs and predictions. The per-category inference protocol is disclosed in Section 3.3 and is matched by class-specific instruction data during training, so it is a comparison-protocol concern about whether the reported AP is directly comparable to single-pass specialist detectors, not a self-referential derivation. Self-citations [37, 44, 45] appear only in the related-work survey and are not load-bearing for the detection claim. The results are evaluated on held-out COCO val against external baselines, so the derivation is self-contained and not circular by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. Its results rest on three unstated assumptions about distribution transfer, pseudo-label quality, and the neutrality of a detection-pretrained backbone. The main free parameters are unstated thresholds and hand-chosen instruction caps.

free parameters (3)
  • Pseudo-label confidence and NMS merge threshold = not specified
    Pseudo-labels from Salience-DETR are merged with ground truth via NMS; the thresholds and the detector's confidence scores become training targets. The paper does not report these values, so the central result depends on unstated choices.
  • Inference score threshold for AP/AR = not specified
    Appendix A states the authors set a threshold to compute AP and AR but does not give the value; this affects the reported AP and AR numbers.
  • Instruction round caps and positive/negative ratio = 80 (COCO), 365 (Object365), 1:1 ratio
    These hand-chosen caps and ratios shape the training distribution and recall behavior; the paper does not ablate them.
assumptions (3)
  • domain assumption A trained LMM tends to approximate the training data distribution, so adjusting that distribution transfers to detection behavior.
    Section 3.2 states this to justify pseudo-label augmentation; it is empirically motivated but not proven beyond the reported experiments.
  • domain assumption Pseudo-labels from the Salience-DETR specialist detector are sufficiently accurate to improve LMM detection without injecting harmful noise.
    The entire data distribution adjustment relies on this; only indirect evidence via AP gains on COCO is provided.
  • domain assumption The OWLv2-ViT detection-pretrained visual encoder can be treated as a standard visual encoder for the purposes of the no-specialist-modules claim.
    OWLv2 is an open-vocabulary object detection model; its ViT is trained for detection. The paper uses it as the visual backbone, which transfers detection-specific features, and does not count it as a specialist module.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LMM-Det: Make Large Multimodal Models Excel in Object Detection." pith.science (2026). https://pith.science/paper/IM64NQXJ

@misc{pith2026250718300,
  author       = {Pith},
  title        = {Pith review of: LMM-Det: Make Large Multimodal Models Excel in Object Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IM64NQXJ}},
  note         = {Machine review of arXiv:2507.18300}
}
read the original abstract

Large multimodal models (LMMs) have garnered wide-spread attention and interest within the artificial intelligence research and industrial communities, owing to their remarkable capability in multimodal understanding, reasoning, and in-context learning, among others. While LMMs have demonstrated promising results in tackling multimodal tasks like image captioning, visual question answering, and visual grounding, the object detection capabilities of LMMs exhibit a significant gap compared to specialist detectors. To bridge the gap, we depart from the conventional methods of integrating heavy detectors with LMMs and propose LMM-Det, a simple yet effective approach that leverages a Large Multimodal Model for vanilla object Detection without relying on specialized detection modules. Specifically, we conduct a comprehensive exploratory analysis when a large multimodal model meets with object detection, revealing that the recall rate degrades significantly compared with specialist detection models. To mitigate this, we propose to increase the recall rate by introducing data distribution adjustment and inference optimization tailored for object detection. We re-organize the instruction conversations to enhance the object detection capabilities of large multimodal models. We claim that a large multimodal model possesses detection capability without any extra detection modules. Extensive experiments support our claim and show the effectiveness of the versatile LMM-Det. The datasets, models, and codes are available at https://github.com/360CVGroup/LMM-Det.

Figures

Figures reproduced from arXiv: 2507.18300 by the authors.

Figure 1
Figure 1. Visualizations of a specialist detector and large multimodal models for object detection on the validation set of COCO. For clarity, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Bounding box (Bbox) distribution of ground-truth and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Overview of the proposed LMM-Det. Compared to the [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Visualizations of LMM-Det on COCO. Model COCO Image Captioning VQAv2 AP BLEU@4 METEOR CIDEr SPICE Accuracy LLaVA 0.2 29.4 29.3 108.9 23.6 78.5 LMM-Det† 47.1 26.7 28.1 99.0 22.4 74.1 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Chat example demonstrates that LMM-Det not only enables detection but also retains its intrinsic multimodal capabilities. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 32 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, pages 23716–23736, 2022. 2

    work page 2022

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 8

    arXiv 2023

  3. [3]

    Cascade r-cnn: Delving into high quality object detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018. 6

    work page 2018

  4. [4]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, pages 13817–13827, 2024. 4

    work page 2024

  5. [5]

    X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160, 2023. 8

    arXiv 2023

  6. [6]

    Allava: Harnessing gpt4v- synthesized data for a lite vision-language model, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for a lite vision-language model, 2024. 8

    work page 2024

  7. [7]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195,

    arXiv

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 2, 5, 6, 8

    arXiv 2024

Show all 58 references
  1. [9]

    Glm: General language model pretraining with autoregressive blank infilling

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In ACL, pages 320–335, 2022. 8

    2022

  2. [10]

    LVIS: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR,

  3. [11]

    Efficient multimodal learning from data-centric perspective

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530, 2024. 8

    2024 arXiv

  4. [12]

    Cogvlm2: Visual language models for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qing- song Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500,

    arXiv

  5. [13]

    Salience detr: Enhancing detection trans- former with hierarchical salience filtering refinement

    Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, and Badong Chen. Salience detr: Enhancing detection trans- former with hierarchical salience filtering refinement. In CVPR, pages 17574–17583, 2024. 1, 5, 6

    2024

  6. [14]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. EMNLP, 2024. 8

    2024

  7. [15]

    Detrs with hybrid matching

    Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. CVPR, 2023. 4, 6

    2023

  8. [16]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In EMNLP, pages 787–798, 2014. 2

    2014

  9. [17]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Ui- jlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Mu...

    2017

  10. [18]

    Building and better understanding vision-language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 2, 8

    2024 arXiv

  11. [19]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 2, 8

    2023

  12. [20]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024. 8

    2024

  13. [21]

    Generative region-language pretraining for open-ended object detection

    Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. In CVPR, pages 13958–13968, 2024. 2, 8

    2024

  14. [22]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 2

    2014

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 2, 3, 4, 5, 6, 7

    2023

  16. [24]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 8

    2024

  17. [25]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 2, 6, 8

    2023 arXiv

  18. [26]

    Ziya- visual: Bilingual large vision-language model via multi-task instruction tuning

    Junyu Lu, Dixiang Zhang, Xiaojun Wu, Xinyu Gao, Ruyi Gan, Jiaxing Zhang, Yan Song, and Pingjian Zhang. Ziya- visual: Bilingual large vision-language model via multi-task instruction tuning. arXiv preprint arXiv:2310.08166, 2023. 8

    arXiv

  19. [27]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. arXiv preprint arXiv:2404.13013, 2024. 2, 6, 8

    2024 arXiv

  20. [28]

    Scal- ing open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scal- ing open-vocabulary object detection. NeurIPS, 36, 2024. 4, 7

    2024

  21. [29]

    Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas...

    2025

  22. [30]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 2, 6, 8

    2023 arXiv

  23. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021. 7

    2021

  24. [32]

    Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. CVPR,

  25. [33]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. TPAMI, 39(6):1137–1149, 2016. 2, 4, 6

    2016

  26. [34]

    Grounding dino 1.5: Advance the "edge" of open-set object detection, 2024

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wen- long Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding dino 1.5: Advance the "edge" of open-set object detection, 2024. 2

    2024

  27. [35]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. InICCV, pages 8430–8439, 2019. 2, 5

    2019

  28. [36]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024. 8

    2024

  29. [37]

    Iaa: Inner-adaptor architecture empowers frozen large language model with multimodal capabilities

    Bin Wang, Chunyu Xie, Dawei Leng, and Yuhui Yin. Iaa: Inner-adaptor architecture empowers frozen large language model with multimodal capabilities. In AAAI, pages 21035– 21043, 2025. 8

    2025

  30. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    arXiv

  31. [39]

    Vary: Scaling up the vision vocabulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109,

    arXiv

  32. [40]

    General ocr theory: Towards ocr-2.0 via a unified end-to-end model

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. arXiv preprint arXiv:2409.01704, 2024

    2024 arXiv

  33. [41]

    Skywork: A more open bilingual foundation model

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanq...

    arXiv

  34. [42]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. NeurIPS, 2024. 2, 6, 8

    2024

  35. [43]

    Deepseek- vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024

  36. [44]

    Ccmb: A large-scale chinese cross- modal benchmark

    Chunyu Xie, Heng Cai, Jincheng Li, Fanjing Kong, Xiaoyu Wu, Jianfei Song, Henrique Morimitsu, Lin Yao, Dexin Wang, Xiangzheng Zhang, et al. Ccmb: A large-scale chinese cross- modal benchmark. In ACM MM, pages 4219–4227, 2023. 8

    2023

  37. [45]

    Fg-clip: Fine-grained visual and textual alignment

    Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-clip: Fine-grained visual and textual alignment. arXiv preprint arXiv:2505.05071, 2025

    2025 arXiv

  38. [46]

    Llava-cot: Let vision language models reason step-by-step, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024. 8

    2024

  39. [47]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 2, 8

    2024 arXiv

  40. [48]

    mplug-owl: Modularization empowers large language models with multimodality, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models w...

    2023 arXiv

  41. [49]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 2

    2023 arXiv

  42. [50]

    Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring, 2024

    Yufei Zhan, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring, 2024. 6, 8, 1

    2024

  43. [51]

    Griffon: Spelling out all object locations at any granularity with large language models

    Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. In ECCV, pages 405–422. Springer, 2025. 6, 8, 1

    2025

  44. [52]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chun- yuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.02949,

    arXiv

  45. [53]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2

    2023 arXiv

  46. [54]

    Ferret-v2: An im- proved baseline for referring and grounding with large lan- guage models

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An im- proved baseline for referring and grounding with large lan- guage models. arXiv preprint arXiv:2404.07973, 2024. 8

    2024 arXiv

  47. [55]

    Detrs beat yolos on real-time object detection, 2024

    Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection, 2024. CVPR. 3, 6

    2024

  48. [56]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2, 8

    2023 arXiv

  49. [57]

    Deformable detr: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021. 6 A. More Implementation Details. The statistical results of COCO are listed in Table A. train validation #images 118,2...

    2021

  50. [58]

    Multi-step

    For an image containing n visible categories (e.g., cat, dog), we generaten positive instructions where the model is asked to output bounding boxes. 2) We then sample n negative instructions by randomly selecting non-present cat- egories from the remaining label set (i.e., 80−...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.