Pith. sign in

REVIEW 4 major objections 4 minor 58 references

DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that Group Relative Query Optimization (GRQO), a reinforcement-style training objective, makes visual-prompt object detectors generalize better than supervised fine-tuning.

desk verdict A promising training recipe for visual-prompt detection, but the central reward term as written is degenerate and the load-bearing mask is left unspecified. read the letter →

arxiv 2505.24025 v2 pith:KBXCCZLA submitted 2025-05-29 cs.CV cs.AI

Chenbin Pan , Wenbin He , Zhengzhong Tu , Liu Ren This is my paper · ORCID
classification cs.CVcs.AI
keywords visualpromptingobjectdetectionreinforcementlearninggrouprelativequeryoptimizationDETRopen-vocabularyKLregularizationquery-levelreward
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to bring R1-style reinforcement training to vision foundation models, focusing on visual-prompting detection, where users specify targets by example instead of text. It argues that supervised fine-tuning gives visual-prompt detectors sparse and unstable supervision, and that its Group Relative Query Optimization (GRQO) provides dense, group-normalized query rewards plus objectness regularization. The payoff, if correct, is a training recipe: a detector trained on Objects365 beats supervised fine-tuning baselines on COCO, LVIS, and ODinW, with the largest variant gaining +8.8 and +4.4 mAP on ODinW13 and ODinW35 and +4.9 mAP on fine-tuned COCO. The authors frame this as incentivizing visual reasoning, but the concrete contribution is a practical optimization strategy for prompt-based detection.

What carries the argument

The central mechanism is Group Relative Query Optimization (GRQO), a training objective designed for query-based detectors. For each decoder query, the paper computes a matching cost against ground-truth instances, converts the minimum cost into a query reward, and normalizes rewards within the image so that queries are pushed to beat the dynamic group average rather than an absolute threshold. A KL-style regularization term, Equation 11, anchors the current objectness distribution to a frozen earlier copy of the model. The composite loss combines this group-relative reward, the KL term, contrastive prompt alignment, and the standard focal, L1, and GIoU losses.

What would settle it

Compute Equation 11 for two objectness score vectors and compare the result with the true KL divergence $\sum_i O_{\mathrm{ref},i} \log(O_{\mathrm{ref},i}/O_{\theta,i})$; also check whether the objectness vectors sum to 1. If the values differ or the vectors do not sum to 1, the term is not a KL divergence. A second check is to replace Equation 11 with a true KL after softmax normalization and rerun the 'only KL-Div' ablation: a substantially different outcome would show that the reported effect depends on the nonstandard formula.

Watch

Extended reading notes

Core claim

The central claim is that GRQO improves both out-of-domain and in-domain generalization for visual-prompt detection compared with standard supervised fine-tuning. The reward is computed per query as the negative of its minimum matching cost to any ground-truth box, normalized across all queries in the image as a relative advantage $(r_i - \mu_r)/\sigma_r$, then used as an auxiliary loss alongside regular detection losses. A KL-style regularization term on the objectness distribution, computed against a frozen reference model, is added to stabilize training. On COCO zero-shot transfer, DINO-R1-T improves over its SFT counterpart by +4.1 mAP; on LVIS rare categories, DINO-R1-B improves by +3.4 APr; and DINO-R1-L exceeds SFT by +8.8 and +4.4 mAP on ODinW13 and ODinW35. In the in-domain COCO fine-tuning setting, GRQO surpasses continued SFT by +4.3 mAP and, when the GRQO-pretrained model is the starting point, by +4.9 mAP.

Load-bearing premise

The argument assumes that the objectness scores in Equation 11 form a probability distribution and that the formula shown is a genuine KL divergence, but objectness is defined as a max-over-prompts similarity that is not normalized, and Equation 11 is not the standard categorical KL formula.

Editorial extensions

If this is right

  • GRQO can be applied to any DETR-style detector with query-based predictions, not only the visual-prompting variant tested here.
  • Training with one randomly sampled visual prompt per class outperforms training with many prompts, while inference benefits from using more prompts per class.
  • Group-relative rewards consistently beat absolute rewards in the ablations, suggesting that inter-query normalization is the key ingredient.
  • Layer-wise reward propagation improves results, implying that earlier decoder refinement stages also benefit from reinforcement-style supervision.
  • The best configuration uses a reward weight near 10e3 and a KL weight near 0.04, indicating that the method is sensitive to loss scaling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: A natural test outside the paper is whether GRQO also improves text-prompt grounding detectors or other dense prediction heads, since the group-relative reward does not depend on the prompt modality.
  • Editorial extension: If the objectness scores are not a normalized distribution, the regularization term in Equation 11 is not a true KL divergence; replacing it with an L2 penalty or a true KL after softmax would isolate what the stabilization actually does.
  • Editorial extension: The authors state the visual prompt encoder was kept deliberately simple; a stronger prompt encoder could further raise results, so architecture gains and optimization gains are entangled in the final system.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DINO-R1, a visual-prompt object detector built by adapting Grounding-DINO with a visual prompt encoder and visual-guided query selection (VIS-G-DINO), and a training scheme called GRQO. GRQO defines a query-level reward from the matching cost between each decoder query and ground truth, normalizes rewards within a group, and adds a KL-type penalty on objectness scores to stabilize training. The model is trained on Objects365 and evaluated zero-shot on COCO, LVIS-minival, ODinW13/35, and after fine-tuning on COCO. Across these settings, GRQO-trained models consistently outperform SFT-trained VIS-G-DINO baselines, with the largest gains on ODinW13 (+8.8 AP for the L variant).

Significance. If the empirical results hold, GRQO is a useful training recipe for visual-prompting detectors: the ablations in Tables 2 and 3 show consistent gains from both the reward and the penalty, and the relative form of the reward outperforms the absolute form. The held-out benchmarks are independent of the training objective, so the generalization claim is meaningful, and the layer-wise reward variant gives a clear additional gain. However, the paper does not demonstrate a new 'reasoning capability'; it improves supervised fine-tuning for a specific visual-prompting architecture. The main conceptual contribution is an auxiliary training loss that reuses the detector's own matching costs, so the RL framing should be tempered. No code or checkpoints are mentioned, which limits reproducibility, but the empirical pattern is internally consistent across model sizes and datasets.

major comments (4)
  1. [Sec. 3.3, Eqs. (9)-(12)] As written, the reward term in Eq. (12) is identically zero for any reward vector when α is a scalar: Eq. (9) standardizes Â_i over all N_q queries, so Σ_i Â_i = 0 and the term −(α/N_q) Σ_i Â_i has no gradient. The sentence 'we set α to zero for the final set of queries exhibiting low objectness probability' implicitly makes α per-query, but the threshold, fraction, and how 'final set' is defined are not given, and Eq. (12) still writes α as a scalar. This must be fixed: define the mask explicitly, specify how it is computed, and report its sensitivity, because otherwise the 'only reward' row of Table 2 measures reward plus an objectness-based hard-example selection rather than a pure group-relative reward.
  2. [Sec. 3.3, Eqs. (10)-(11)] Equation (11) is not the KL divergence. The objectness scores in Eq. (10) are max-over-prompt similarities of unnormalized dot products; they are not a probability distribution, and there is no normalization or positivity guarantee, so O_ref/O_θ may be negative or undefined, and O_θ log(O_θ/O_ref) is not the quantity shown. The right-hand side O_ref/O_θ − log(O_ref/O_θ) − 1 is a Bregman penalty on the ratio, not D_KL[O_θ ∥ O_ref]. Please define the exact regularizer used, including whether a softmax or temperature is applied, how the reference indices are computed, and whether reference features are detached, and then re-state what the 'only KL-Div' ablation in Table 2 actually evaluates.
  3. [Sec. 3.3, Eq. (7) vs. Eq. (1)] The reward reuses the same focal, L1, and GIoU costs as the supervised detection loss, so the 'reward' is partly a reweighted auxiliary detection loss rather than an independent verifiable reward. This does not invalidate the held-out improvements, but it weakens the RL interpretation and the claim of incentivizing a distinct reasoning capability. I ask the authors to state this explicitly and to justify why group-relative normalization plus masking, rather than simply extra gradient pressure on unmatched queries, is the source of the observed gains.
  4. [Sec. 4, Tables 1-5] The paper reports no error bars, multiple seeds, or statistical significance tests. Because RL-style training is typically high-variance and some reported differences are small (e.g., 22.7 vs. 23.5 in Table 3), the consistency claim needs at least a few seeds or a variance estimate. This is especially load-bearing for the fine-tune numbers, where the GRQO-pretrained model is compared against continued SFT.
minor comments (4)
  1. [Sec. 4.3, Table 4] The text says the reward weight is swept over 1.0, 10.0, 10e2, 10e3, and 10e4, but the table omits the 10e2 row; please align the text and table.
  2. [Sec. 3.2, Eq. (5)] The sentence 'The remaining stages mirror the G-DINO pipeline' leaves unspecified how final class predictions are obtained from visual prompts; please clarify whether text embeddings are replaced by visual prompt embeddings in the final contrastive classifier.
  3. [Sec. 3.3, Eq. (10)] The reference objectness uses the same top-N_q indices Idx^v_Nq as the current model; please clarify whether these indices come from the current or reference model and whether gradients are blocked through the reference features.
  4. [Throughout] The term 'visual in-context reasoning capability' is used as an invented entity without a formal definition or metric; please replace it with an operational description such as 'zero-shot visual-prompt detection accuracy' or define 'reasoning' precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical benchmark comparison on held-out datasets, not a derivation whose outputs are equivalent to its inputs.

full rationale

The paper's central claim is that DINO-R1, trained with GRQO, generalizes better than SFT on COCO, LVIS, and ODinW. These are held-out external benchmarks, so the reported improvements do not reduce by construction to the training objective. The reward in Eq. (7)-(8) reuses the same focal, L1, and GIoU costs as the supervised loss in Eq. (1), which makes GRQO partly a reweighted auxiliary loss rather than a fully independent RL signal; however, this is a design overlap, not a circular derivation, because the evaluation is not defined in terms of those costs. The KL term in Eq. (11) is not a standard categorical KL divergence and the objectness scores in Eq. (10) are not a normalized distribution, but these are mathematical correctness concerns rather than circularity. Similarly, the fact that the group-standardized advantage in Eq. (9) sums to zero over all queries, so that the reward term in Eq. (12) is degenerate unless alpha is per-query, is an underspecification bug in the stated loss; it does not make the benchmark predictions equivalent to the inputs. The paper contains no load-bearing self-citations: GRPO, Grounding-DINO, and MM-Grounding-DINO are external prior work, and no 'uniqueness' result is imported from the authors. The stated limitations are about architectural simplicity, not about circular reasoning. The empirical comparisons are self-contained against independent data, so the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the pretrained Grounding-DINO backbone, on an unproven treatment of objectness scores as a probability distribution, and on several hyperparameters selected through COCO ablations. No new physical entities are introduced, but the paper's 'reasoning capability' framing is not independently evidenced.

free parameters (5)
  • Reward loss weight alpha = 10e3
    Chosen from ablation in Table 4 on COCO zero-shot; the best value among 1, 10, 10e2, 10e3, 10e4.
  • KL regularization weight beta = 0.04
    Chosen from Table 4; best among 0.4, 0.04, 0.004.
  • Number of sampled prompts per class M = 1
    Table 5 shows training with M=1 strongly outperforms M=8, 16, 32, and 64 on COCO zero-shot, so it is selected by validation results.
  • Matching cost weights lambda_focal, lambda_l1, lambda_GIoU = unspecified
    Eq. 7 defines the reward as a weighted sum of focal, L1, and GIoU costs, but the weights are not reported in the paper, leaving a hidden training choice.
  • Masking threshold for low objectness queries = unspecified
    Section 3.3 sets alpha to zero for queries with low objectness probability, but the threshold is not defined, so the effective reward signal is implementation-dependent.
assumptions (4)
  • domain assumption Grounding-DINO / MM-Grounding-DINO provides a strong pretrained base detector and text encoder.
    The method builds on an existing detector and relies on its representations; if the base is weak, the observed gains may not transfer.
  • ad hoc to paper Objectness scores form a probability distribution suitable for KL-divergence.
    Eq. 10 takes the max over the prompt axis of a similarity matrix, which is not normalized, and Eq. 11 uses a nonstandard divergence formula, so this assumption is unproven.
  • domain assumption The focal, L1, and GIoU matching cost is a valid proxy for detection quality in the reward.
    The reward in Eq. 8 is the negative minimum matching cost, so the training signal inherits any bias in that cost.
  • domain assumption A frozen copy of the model after 1 SFT epoch is a stable reference for objectness.
    KL regularization anchors the current model to this reference; the choice of 1 epoch is not justified beyond experimental convenience.
invented entities (1)
  • Visual in-context reasoning capability
    purpose: The paper claims GRQO incentivizes this capability in vision foundation models.
    No direct measurement of reasoning is provided; detection mAP is used as a proxy, so the entity is rhetorical rather than empirically grounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models." pith.science (2026). https://pith.science/paper/KBXCCZLA

@misc{pith2026250524025,
  author       = {Pith},
  title        = {Pith review of: DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KBXCCZLA}},
  note         = {Machine review of arXiv:2505.24025}
}
read the original abstract

The recent explosive interest in the reasoning capabilities of large language models, such as DeepSeek-R1, has demonstrated remarkable success through reinforcement learning-based fine-tuning frameworks, exemplified by methods like Group Relative Policy Optimization (GRPO). However, such reasoning abilities remain underexplored and notably absent in vision foundation models, including representation models like the DINO series. In this work, we propose \textbf{DINO-R1}, the first such attempt to incentivize visual in-context reasoning capabilities of vision foundation models using reinforcement learning. Specifically, DINO-R1 introduces \textbf{Group Relative Query Optimization (GRQO)}, a novel reinforcement-style training strategy explicitly designed for query-based representation models, which computes query-level rewards based on group-normalized alignment quality. We also apply KL-regularization to stabilize the objectness distribution to reduce the training instability. This joint optimization enables dense and expressive supervision across queries while mitigating overfitting and distributional drift. Building upon Grounding-DINO, we train a series of DINO-R1 family models that integrate a visual prompt encoder and a visual-guided query selection mechanism. Extensive experiments on COCO, LVIS, and ODinW demonstrate that DINO-R1 significantly outperforms supervised fine-tuning baselines, achieving strong generalization in both open-vocabulary and closed-set visual prompting scenarios.

Figures

Figures reproduced from arXiv: 2505.24025 by the authors.

Figure 1
Figure 1. SFT vs. GRQO. SFT leads to limited and homogeneous supervision signals, while GRQO produces richer and more diverse learning signals, encouraging queries to be more expressive. Open-Vocabulary and Prompt-Based Detection. Open-vocabulary detection [54](OVD) aims to recognize categories beyond the supervised training set by leveraging external knowledge sources such as pretrained text encoders or image-text pairs. Mos… view at source ↗
Figure 2
Figure 2. Overview of the proposed Group Relative Query Optimization (GRQO) framework. The query reward module enriches supervision by assigning group-relative rewards to enhance query learning. In parallel, KL-regularization constrains the trust region of query updates, enabling the detector to progressively absorb diverse visual prompts while preserving previously acquired detection behaviors. guidance during detection, G-D… view at source ↗
Figure 3
Figure 3. Qualitative comparison of visual prompting detection between SFT and GRQO. SFT results exhibit both false positives (row 2,3,4) and missed detections (row 1), reflecting limited query expressiveness and weak alignment with visual prompts. In contrast, GRQO produces more accurate and complete detections, better aligning with the prompted semantics. These results highlight GRQO’s ability to enhance query reasoning and… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) Training curves comparing SFT and GRQO. GRQO consistently achieves more stable training with higher final performance. (b)(c) Effect of the number of prompts per class during training/inference. # COCO Zero-Shot Train Inference 1 24.0 11.2 8 16.4 17.6 16 15.2 22.0 …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 18 canonical work pages

  1. [1]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  6. [6]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    arXiv 2024

  7. [7]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024

    arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 58 references
  1. [9]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  2. [10]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  3. [11]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  4. [12]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  5. [13]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022

    2022 arXiv

  6. [14]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  7. [15]

    Dino-x: A unified vision model for open-world object detection and understanding

    Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, et al. Dino-x: A unified vision model for open-world object detection and understanding. arXiv preprint arXiv:2411.14347, 2024

    2024 arXiv

  8. [16]

    Visual in-context prompting

    Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual in-context prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12861–12871, 2024. 10

    2024

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  11. [19]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings...

    2014

  12. [20]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019

    2019

  13. [21]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  14. [22]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR, 2020

    2020

  15. [23]

    G-simclr: Self-supervised contrastive learning with guided projection via pseudo labelling

    Souradip Chakraborty, Aritra Roy Gosthipaty, and Sayak Paul. G-simclr: Self-supervised contrastive learning with guided projection via pseudo labelling. In 2020 international conference on data mining workshops (ICDMW), pages 912–916. IEEE, 2020

    2020

  16. [24]

    T-rex: Counting by visual prompting

    Qing Jiang, Feng Li, Tianhe Ren, Shilong Liu, Zhaoyang Zeng, Kent Yu, and Lei Zhang. T-rex: Counting by visual prompting. arXiv preprint arXiv:2311.13596, 2023

    2023 arXiv

  17. [25]

    T-rex2: Towards generic object detection via text-visual prompt synergy

    Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision, pages 38–57. Springer, 2024

    2024

  18. [26]

    Cp-detr: Concept prompt guide detr toward stronger universal object detection

    Qibo Chen, Weizhong Jin, Jianyue Ge, Mengdi Liu, Yuchao Yan, Jian Jiang, Li Yu, Xuanjiang Guo, Shuchang Li, and Jianzhong Chen. Cp-detr: Concept prompt guide detr toward stronger universal object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volu...

    2025

  19. [27]

    Autovp: An automated visual prompting framework and benchmark

    Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Chen, Sijia Liu, and Tsung-Yi Ho. Autovp: An automated visual prompting framework and benchmark. arXiv preprint arXiv:2310.08381, 2023

    2023 arXiv

  20. [28]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024

    2024

  21. [29]

    Multimodal prompt perceiver: Empower adaptiveness generalizability and fidelity for all-in-one image restoration

    Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, and Ran He. Multimodal prompt perceiver: Empower adaptiveness generalizability and fidelity for all-in-one image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2543...

    2024

  22. [30]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024

    2024 arXiv

  23. [31]

    Prompt engineering for zero-shot and few-shot defect detection and classification using a visual-language pretrained model

    Gunwoo Yong, Kahyun Jeon, Daeyoung Gil, and Ghang Lee. Prompt engineering for zero-shot and few-shot defect detection and classification using a visual-language pretrained model. Computer-Aided Civil and Infrastructure Engineering, 38(11):1536–1554, 2023

    2023

  24. [32]

    Promptcharm: Text-to-image generation through multi-modal prompting and refinement

    Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. Promptcharm: Text-to-image generation through multi-modal prompting and refinement. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–21, 2024

    2024

  25. [33]

    Prompting industrial anomaly segment with large vision-language models

    jinheng zhou, Wu Liu, Guang Yang, He Zhao, and feiniu yuan. Prompting industrial anomaly segment with large vision-language models. In Proceedings of the 6th ACM International Conference on Multimedia in Asia, pages 1–1, 2024. 11

    2024

  26. [34]

    Visual prompting in multimodal large language models: A survey

    Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.15310, 2024

    2024 arXiv

  27. [35]

    Understanding and improving visual prompting: A label-mapping perspective

    Aochuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: A label-mapping perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19133–19143, 2023

    2023

  28. [36]

    Vp3d: Unleashing 2d visual prompt for text-to-3d generation

    Yang Chen, Yingwei Pan, Haibo Yang, Ting Yao, and Tao Mei. Vp3d: Unleashing 2d visual prompt for text-to-3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4896–4905, 2024

    2024

  29. [37]

    Grounding dino 1.5: Advance the" edge" of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300, 2024

    2024 arXiv

  30. [38]

    An open and comprehensive pipeline for unified object grounding and detection

    Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361, 2024

    2024 arXiv

  31. [39]

    Learning to prompt for open- vocabulary object detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14084–14093, 2022

    2022

  32. [40]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  33. [41]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  34. [42]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  35. [43]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  36. [44]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021

    2021 arXiv

  37. [45]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages ...

    2022

  38. [46]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  39. [47]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  40. [48]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  41. [49]

    Yolo-world: Real- time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024

    2024

  42. [50]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020. 12

    2020

  43. [51]

    Dn-detr: Accelerate detr training by introducing query denoising

    Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13619–13627, 2022

    2022

  44. [52]

    Dab-detr: Dynamic anchor boxes are better queries for detr

    Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329, 2022

    2022 arXiv

  45. [53]

    Open-vocabulary detr with conditional matching

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European conference on computer vision, pages 106–122. Springer, 2022

    2022

  46. [54]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14393–14402, 2021

    2021

  47. [55]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2021

    2021 arXiv

  48. [56]

    Open- vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023

    2023

  49. [57]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lan...

    2019

  50. [58]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 13

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.