Pith. sign in

REVIEW 3 major objections 4 minor 66 references

Referring Expression Instance Retrieval and A Strong End-to-End Baseline

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces REIR, a task that requires retrieving and localizing a specific object instance from a gallery based on a fine-grained referring expression, and proposes CLARE, an end-to-end dual-stream model that learns…

desk verdict Useful new benchmark and a plausible baseline, but the missing train/test disjointness statement between REIRCOCO and the pretraining data is a load-bearing omission that needs to be fixed before the SOTA claims can be trusted. read the letter →

arxiv 2506.18246 v4 pith:UQT5UM2U submitted 2025-06-23 cs.CV

Xiangzhao Hao , Kuan Zhu , Hongyu Guo , Haiyun Guo , Ning Jiang , Quan Lu , Ming Tang , Jinqiao Wang This is my paper
classification cs.CV
keywords ReferringExpressionInstanceRetrievalREIRCOCOcontrastivelanguage-instancealignmentcomprehensiontext-imagemixtureofexpertsend-to-endgroundingvision-languagepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Referring Expression Instance Retrieval (REIR) asks a model to take a fine-grained natural language description of a specific object and return both the gallery image containing that object and the object's bounding box. The paper argues that existing tasks fail this setting: text-image retrieval finds images but not instances, and referring expression comprehension grounds instances but cannot rank across a gallery. To support the task, the authors build REIRCOCO, a dataset of over 30,000 images and 613,548 instance-level expressions generated by prompting vision-language models and filtered for uniqueness. They also present CLARE, an end-to-end dual-stream model that aligns text queries with detected object instances through a contrastive objective and a mixture of relation experts. CLARE is reported to outperform all two-stage TIR-plus-REC baselines on REIRCOCO across IoU thresholds and ranking levels, and to generalize competitively to standard referring expression comprehension and text-image retrieval benchmarks.

What carries the argument

The load-bearing components are the Contrastive Language-Instance Alignment (CLIA) loss and the dual-stream architecture that makes it possible. CLARE uses SigLIP vision and text encoders to produce independent image and text features, a Deformable-DETR-based object extractor to propose up to 900 candidates per image with instance embeddings, and a Mix of Relation Experts (MORE) module that refines the text embedding through one shared expert plus top-2 routing among four specialized experts. CLIA treats each expression-object pair across the whole batch as a binary classification problem with a learnable temperature and bias, so the model learns to separate the true referent from distractors in other images. The same object features feed a position decoder that outputs bounding boxes, so retrieval and localization share one representation and one score.

What would settle it

Rebuild the REIRCOCO evaluation with a split whose test images are guaranteed absent from every pretraining stage, and check whether CLARE's BR@1 at IoU 0.5 stays near 29.5; a large drop would indicate the reported numbers depended on images and boxes seen during pretraining.

Watch

Extended reading notes

Core claim

The paper's central claim is that instance-level retrieval and localization can be solved jointly by an end-to-end contrastive model rather than by cascading a text-image retriever with a referring-expression comprehender. CLARE encodes the expression into a query embedding, detects object candidates in each gallery image, and scores every candidate by dot-product similarity with the query; the top-scoring bounding box is the answer. A contrastive objective, CLIA, extends the SigLIP sigmoid loss to align expressions with objects across images and batches, while a Mix of Relation Experts (MORE) routes the text embedding through shared and specialized experts to capture spatial and relational cues. On the REIRCOCO benchmark the authors report state-of-the-art BoxRecall@k across IoU thresholds 0.5, 0.7, and 0.9, with the ViT-L variant reaching 29.53 BR@1 at IoU 0.5, and also report strong transfer to REC and TIR benchmarks. The conclusion the authors draw is that cross-image instance-level contrastive alignment is the right inductive bias for grounding fine-grained expressions in open galleries.

Load-bearing premise

The REIRCOCO test images are not the same images used in pretraining, a fact the paper does not explicitly state; if the split overlaps with COCO or RefCOCO training data, the reported retrieval-and-localization numbers could be inflated.

Editorial extensions

If this is right

  • If CLARE's results hold, gallery-scale instance search becomes feasible without per-image cross-modal fusion: object features are precomputed once, and a query is encoded once and matched by dot product.
  • The REIRCOCO benchmark gives the community a way to measure retrieval and grounding jointly, so future models can be compared on a single BoxRecall@k metric instead of separate recall and precision scores.
  • The reported transfer results suggest that instance-level contrastive alignment is a competitive alternative to early-fusion REC models, potentially simplifying training pipelines.
  • Because CLARE outperforms cascades of strong TIR and REC models, the paper argues that end-to-end training avoids error accumulation in the retrieve-then-localize pipeline.
  • The staging of pretraining on detection/REC data followed by REIRCOCO fine-tuning is shown to be necessary; removing either stage drops BR@1 from 26.39 to 13.44 or 4.02.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the test images in REIRCOCO are not disjoint from the COCO/RefCOCO images used in pretraining, the reported gains could be inflated by the model having seen the exact test images and their ground-truth boxes; a clean split would settle this.
  • REIRCOCO's expressions are generated and filtered by LLMs, so the benchmark inherits whatever distributional biases GPT-4o and DeepSeek-R1 have; human verification of a sample would test whether the 'unambiguous' property really holds at gallery scale.
  • The success of instance-level contrastive alignment suggests a natural extension: using REIRCOCO as a pretraining corpus for downstream REC or TIR tasks, or as a training signal for open-vocabulary detection.
  • A testable extension would be to vary the number of routed experts and the gallery size to measure where relational reasoning saturates, since the paper only ablates one configuration at 4 routed experts with top-2.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a new visual-language task, Referring Expression Instance Retrieval (REIR), which requires retrieving the correct image from a gallery and localizing the referred object instance given a fine-grained natural-language expression. To support the task, the authors construct a large benchmark, REIRCOCO, by prompting GPT-4o to generate five referring expressions per object instance from MSCOCO and RefCOCO annotations, then filtering with DeepSeek R1. They propose an end-to-end baseline, CLARE, a dual-stream SigLIP + Deformable-DETR architecture trained in two stages (detection/REC pretraining, then REIRCOCO finetuning with a contrastive language-instance alignment loss and a Mix of Relation Experts module). Experiments compare CLARE with two-stage TIR+REC baselines on REIRCOCO, report REC results on RefCOCO/+/g, and TIR results on REIRCOCO, concluding that CLARE achieves state-of-the-art REIR performance and strong generalization.

Significance. The task definition is timely and addresses a real gap between image-level retrieval and within-image localization. The dataset construction pipeline is scalable and the decision to release code and benchmark is a concrete contribution. If the evaluation is sound, CLARE provides a credible end-to-end architecture that avoids per-image cross-modal fusion, which could be useful for gallery-scale instance search. The ablation studies (Tables 4-7) directly test several design choices and are informative. However, the central state-of-the-art claim depends on two unverified assumptions: that REIRCOCO test images are disjoint from pretraining images, and that the comparison protocols against two-stage baselines are fair. These issues, if unresolved, materially weaken the empirical contribution.

major comments (3)
  1. [Section 3.2 and Appendix A.1] The paper does not document whether the REIRCOCO test set images are disjoint from the MSCOCO and RefCOCO images used in Stage 1 pretraining (Appendix A.1 describes pretraining on MSCOCO for detection and RefCOCO/+/g for REC). Because REIRCOCO is constructed from these exact datasets (Section 3.2), the model may have seen the test images and their ground-truth boxes during pretraining, which would inflate BoxRecall numbers in Tables 1 and 3. The authors must state how the REIRCOCO train/test split is defined, verify that no pretraining image appears in the REIRCOCO test set, and report the degree of overlap. Without this, the headline comparison in Section 5.3 is not trustworthy.
  2. [Section 5.3, Table 1] The two-stage baselines (CLIP/EVA-CLIP/SigLIP combined with DeepSeek-VL2 or SimVG) are used off-the-shelf, with no indication that they were trained or fine-tuned on REIRCOCO. In contrast, CLARE is explicitly fine-tuned on REIRCOCO in Stage 2 (Section A.1, Eq. 6). This is a zero-shot-versus-supervised comparison, and the claim that CLARE 'outperforms all combinations of two-stage baselines' is therefore not a meaningful state-of-the-art result for a newly proposed task. The authors should either fine-tune the two-stage components on REIRCOCO (e.g., fine-tune SimVG on REIRCOCO expressions, or train a contrastive adapter on top of TIR features) and report those results, or evaluate CLARE in a zero-shot setting without REIRCOCO finetuning.
  3. [Sections 1, 5.3, and 6] The paper repeatedly calls CLARE 'state-of-the-art' on REIR, but REIR is a new benchmark introduced in the same paper and the model is trained and tuned on that benchmark. This is a self-referential evaluation loop: the benchmark is authored by the same group, and the baseline is a method they developed and tuned. The authors should temper this claim and clarify that this is a first baseline on a new task, not a claim of superiority on an established benchmark. Independent validation (e.g., by external groups, or by testing CLARE on a held-out set constructed without the authors' pipeline) would be needed to substantiate the 'state-of-the-art' wording.
minor comments (4)
  1. [Table 6 heading] The heading 'Ablation study on CILA' uses 'CILA' while the model component is consistently named CLIA elsewhere; please fix this typo.
  2. [Section 5.1, Eq. 9] The definition of BoxRecall@k says 'given a batch of N referring expressions' but the gallery of instance candidates is not explicitly defined. It would be helpful to state whether the gallery is the full test set, a randomly sampled subset, or the batch itself, and to specify how the top-k candidates are formed across images during inference.
  3. [Section B.2] The filtering stage uses DeepSeek R1, which does not process images; it evaluates descriptions using only captions and the GPT-generated text. The paper should discuss the possibility that this text-only filter may retain expressions that misdescribe the actual visual content, and quantify the impact on dataset quality (e.g., by reporting a human-annotated quality sample).
  4. [Abstract and Section 3.2] The abstract states 'over 30,000 images and 200,000 uniquely annotated object instances,' while Section 3.2 gives exact numbers '30,106 images and 215,835 object instances' and later '613,548 fine-grained descriptions.' Please align these numbers for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CLARE's REIR results are a standard supervised evaluation, with independent external validation on RefCOCO benchmarks; the self-built benchmark raises evaluation-risk, not circularity.

full rationale

The paper's derivation chain is empirical rather than definitional. The fine-tuning objective (Eq. 6, L_finetune = L_CLIA + L_focal + L_bbox) supervises CLARE with REIRCOCO expression-instance pairs, and the reported metric BoxRecall@k (Eq. 9) requires both ranking and IoU agreement with annotated boxes; no equation reduces to another by construction, and the 'prediction' is not a renamed training target. The two-stage baselines in Table 1 are adapted external models without REIRCOCO fine-tuning, which is a comparison-fairness issue, not circularity. The absence of an explicit statement that REIRCOCO test images are disjoint from the COCO/RefCOCO images used in Stage 1 pretraining (Appendix A.1) is an omitted safeguard and a correctness risk, but it is not evidence that the results are equivalent to the inputs by construction. Independent support exists: Table 2 reports strong results on the external RefCOCO/RefCOCO+/RefCOCOg benchmarks (e.g., CLARE-ViT-L 91.40 RefCOCO val, 86.70 RefCOCOg test-u), so the method is not validated only on its own benchmark. No load-bearing self-citations are present; the cited bases (SigLIP, Deformable DETR, GPT-4o, DeepSeek, RefCOCO, COCO) are external systems/datasets. The appended note that DeepSeek filters without seeing images (Appendix B.2) and the qualitative note about test-set figures being slightly misaligned are data-quality/formatting limitations, not circular steps.

Assumptions & free parameters 5 free parameters · 4 assumptions · 4 invented entities

The central claims rest on assumptions about dataset quality and split cleanliness rather than on mathematical axioms. The model is empirical, so the ledger lists hand-chosen hyperparameters and domain assumptions about the LLM-generated benchmark. The key unverified assumption is that REIRCOCO test images are disjoint from pretraining images.

free parameters (5)
  • loss weight lambda_box = 5.0
    Set manually in Stage 2 finetuning; balances retrieval and localization, affects BoxRecall results.
  • loss weight lambda_retrieve = 1.0
    Set empirically in Appendix A.1; weighted with lambda_box in the final objective.
  • number of object queries M = 900
    Chosen by hand for the Deformable-DETR extractor; determines the candidate set size for retrieval ranking.
  • routed expert count and top-k selection = 4 routed experts, top-2
    Ablation in Table 5 shows these settings improve BR@10 from 53.28 to 56.85; selected based on validation performance.
  • learnable temperature t and bias b in CLIA = learned
    Introduced in the CLIA loss, inherited from SigLIP; affect contrastive scaling and decision threshold.
assumptions (4)
  • domain assumption REIRCOCO test images are disjoint from images used in pretraining on MSCOCO and RefCOCO.
    Nowhere stated in the paper; if false, pretraining sees test images and inflates REIR results. This assumption enters at Section 3.2 and Appendix A.1.
  • domain assumption GPT-4o-generated expressions are accurate, unambiguous, and grounded in the image.
    Dataset quality rests entirely on LLM generation and DeepSeek R1 filtering; no human validation is reported (Section 3.2, Appendix B.2).
  • domain assumption Deformable-DETR object proposals provide a fair candidate set covering the target instance.
    If the target is not among the 900 proposals, retrieval fails by construction; this is assumed throughout Section 4.1.
  • domain assumption DeepSeek R1 filtering correctly removes ambiguous expressions without bias.
    Filtering uses captions and generated descriptions only, since DeepSeek cannot accept images; mis-filtering changes benchmark difficulty (Appendix B.2).
invented entities (4)
  • REIR task independent evidence
    purpose: Defines combined instance retrieval and localization evaluation
    The task comes with a metric (BoxRecall@k) and a benchmark that other groups can run, so it is externally checkable once released.
  • REIRCOCO dataset
    purpose: Large-scale benchmark for REIR
    Currently promised but not released; quality rests on LLM generation and filtering, with no human validation or public artifacts yet.
  • MORE module
    purpose: Mix-of-experts text encoder to capture relational cues
    Evaluated only through ablations inside this paper; no external replication yet.
  • CLIA loss independent evidence
    purpose: Contrastive objective aligning expressions with object features
    The model's utility is corroborated on external REC benchmarks (RefCOCO family), giving an external handle beyond the authors' benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Referring Expression Instance Retrieval and A Strong End-to-End Baseline." pith.science (2026). https://pith.science/paper/UQT5UM2U

@misc{pith2026250618246,
  author       = {Pith},
  title        = {Pith review of: Referring Expression Instance Retrieval and A Strong End-to-End Baseline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQT5UM2U}},
  note         = {Machine review of arXiv:2506.18246}
}
read the original abstract

Using natural language to query visual information is a fundamental need in real-world applications. Text-Image Retrieval (TIR) retrieves a target image from a gallery based on an image-level description, while Referring Expression Comprehension (REC) localizes a target object within a given image using an instance-level description. However, real-world applications often present more complex demands. Users typically query an instance-level description across a large gallery and expect to receive both relevant image and the corresponding instance location. In such scenarios, TIR struggles with fine-grained descriptions and object-level localization, while REC is limited in its ability to efficiently search large galleries and lacks an effective ranking mechanism. In this paper, we introduce a new task called \textbf{Referring Expression Instance Retrieval (REIR)}, which supports both instance-level retrieval and localization based on fine-grained referring expressions. First, we propose a large-scale benchmark for REIR, named REIRCOCO, constructed by prompting advanced vision-language models to generate high-quality referring expressions for instances in the MSCOCO and RefCOCO datasets. Second, we present a baseline method, Contrastive Language-Instance Alignment with Relation Experts (CLARE), which employs a dual-stream architecture to address REIR in an end-to-end manner. Given a referring expression, the textual branch encodes it into a query embedding. The visual branch detects candidate objects and extracts their instance-level visual features. The most similar candidate to the query is selected for bounding box prediction. CLARE is first trained on object detection and REC datasets to establish initial grounding capabilities, then optimized via Contrastive Language-Instance Alignment (CLIA) for improved retrieval across images. We will release our code and benchmark publicly.

Figures

Figures reproduced from arXiv: 2506.18246 by the authors.

Figure 1
Figure 1. Comparison of three vision-language tasks and their datasets. REIR enables end-to-end instance-level retrieval and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the REIRCOCO dataset construction pipeline. GPT-4o produces context-rich and uniquely grounded [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An overview of CLARE. The model encodes texts and images in parallel, then aligns referring expressions with [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison between CLARE and a two [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The data construction process for all instances in a single image [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Some failure cases of CLARE on the REIRCOCO dataset [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: The qualitative results of CLARE on REIRCOCO. This result demonstrates that CLARE can effectively retrieve and [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompt used for generating textual descriptions with GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt used for filtering textual descriptions with DeepSeek R1 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 38 canonical work pages

  1. [1]

    OpenAI:Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, FlorenciaLeoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shya- mal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Bal- tescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Le...

    work page 2023

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  3. [3]

    Amulya Arun Ballakur and Arti Arya. 2020. Empirical Evaluation of Gated Recurrent Neural Network Architectures in Aviation Delay Prediction. In 2020 5th International Conference on Computing, Communication and Security (ICCCS) . 1–7. doi:10.1109/icccs49678.2020.9276855

    work page arXiv 2020

  4. [4]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In ECCV

    work page 2020

  5. [5]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao

  6. [6]

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Pi- otr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)

    arXiv 2015

  7. [7]

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TExt Representation Learning. 104–120. doi:10.1007/978-3-030-58577-8_7

    doi:10.1007/978-3-030-58577-8_7 2020

  8. [8]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24185–24198

    2024

Show all 66 references
  1. [9]

    Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang. 2024. Simvg: A simple framework for visual grounding with decoupled multi-modal fusion. Advances in neural information processing systems 37 (2024), 121670– 121698

    2024

  2. [10]

    DeepSeek-AI DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z.F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bocha...

    2025

  3. [11]

    Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li

  4. [12]

    Fleet, Jamie Kiros, and Sanja Fidler

    Fartash Faghri, DavidJ. Fleet, Jamie Kiros, and Sanja Fidler. 2017. VSE++: Improv- ing Visual-Semantic Embeddings with Hard Negatives. arXiv: Learning,arXiv: Learning (Jul 2017)

    2017

  5. [13]

    Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov

    Andrea Frome, GregS. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A Deep Visual-Semantic Embedding Model. Neural Information Processing Systems,Neural Information Processing Systems (Dec 2013)

    2013

  6. [14]

    Zhe Gan, Yen-Chun Chen, Pingqing Fu, Chen Zhu, Yu Cheng, and Jingjing Liu

  7. [16]

    Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang

  8. [17]

    Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingx- ing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. 2019. Searching for mobilenetv3. In ICCV

    2019

  9. [18]

    Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 2021. Seeing Out of tHe bOx: End-to-End Pre-training for Vision- Language Representation Learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . doi:10.1109/cvpr...

    2021

  10. [19]

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. MDETR-modulated detection for end-to-end multi- modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 1780–1790

    2021

  11. [20]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 787–798

    2014

  12. [21]

    Won-Jae Kim, Bokyung Son, and Ildoo Kim. 2021. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. International Con- ference on Machine Learning,International Conference on Machine Learning (Feb 2021)

    2021

  13. [22]

    Thomas Kipf and Max Welling. 2016. Semi-Supervised Classification with Graph Convolutional Networks. arXiv: Learning,arXiv: Learning (Sep 2016)

    2016

  14. [23]

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked Cross Attention for Image-Text Matching. 212–228. doi:10.1007/978-3-030- 01225-0_13 Conference acronym ’25, June 03–05, 2025, Woodstock, NY Xiangzhao et al

    2018 doi

  15. [24]

    Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. 2022. DN-DETR: Accelerate detr training by introducing query denoising. In CVPR

    2022

  16. [25]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

    2022

  17. [26]

    Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. 2019. Visual Semantic Reasoning for Image-Text Matching. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . doi:10.1109/iccv.2019.00475

    2019

  18. [27]

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-image pre-training. In CVPR

    2022

  19. [28]

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In ICCV

    2017

  20. [29]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. 740–755. doi:10.1007/978-3-319-10602-1_48

    2014 doi

  21. [30]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Gir- shick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In ECCV

    2014

  22. [31]

    Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, and Yong- dong Zhang. 2019. Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching. Cornell University - arXiv,Cornell University - arXiv (Sep 2019)

    2019

  23. [32]

    Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. 2019. Learning to assemble neural module tree networks for visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . 4673–4682

    2019

  24. [33]

    Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. 2019. Improving referring expression grounding with cross-modal attention-guided erasing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1950–1959

    2019

  25. [34]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled Weight Decay Regularization. Learning,Learning (Nov 2017)

    2017

  26. [35]

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. 2016. Modeling context between objects for referring expression understanding. In Proceedings of the European Conference on Computer Vision (ECCV) . Springer, 792–807

    2016

  27. [36]

    Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. 2017. Dual Attention Net- works for Multimodal Reasoning and Matching. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . doi:10.1109/cvpr.2017.232

    2017 doi

  28. [37]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Askell Amanda, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, JongWook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Askell Amanda, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. Cornell University ...

    2021

  29. [38]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  30. [39]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. InAdvances in Neural Information Processing Systems (NIPS)

    2015

  31. [40]

    Fengyuan Shi, Ruopeng Gao, Weilin Huang, and Limin Wang. 2022. Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding. (Sep 2022)

    2022

  32. [41]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv preprint arXiv:2303.15389 (2023)

    2023 arXiv

  33. [42]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017)

    2017

  34. [43]

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al

  35. [44]

    Yaxiong Wang, Hao Yang, Xueming Qian, Lin Ma, Jing Lu, Biao Li, and Xin Fan

  36. [45]

    Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. 2019. CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . doi:10.1109/iccv.2019.00586

    2019

  37. [46]

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 11686–11695

    2022

  38. [47]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024 arXiv

  39. [48]

    Sibei Yang, Guanbin Li, and Yizhou Yu. 2019. Dynamic graph attention for referring expression comprehension. In Proceedings of the IEEE/CVF international conference on computer vision . 4644–4653

    2019

  40. [49]

    arXiv preprint arXiv:2208.10442 (2022)

    Image as a Foreign Language: BEiT Pretraining for All Vision and Vision- Language Tasks. arXiv preprint arXiv:2208.10442 (2022)

    2022 arXiv

  41. [50]

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. 2022. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 18155–18165

    2022

  42. [51]

    Cornell University - arXiv,Cornell University - arXiv (Jul 2019)

    Position Focused Attention Network for Image-Text Matching. Cornell University - arXiv,Cornell University - arXiv (Jul 2019)

    2019

  43. [52]

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. Mattnet: Modular attention network for referring ex- pression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1307–1315

    2018

  44. [53]

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg

  45. [54]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sig- moid loss for language image pre-training. In Proceedings of the IEEE/CVF inter- national conference on computer vision . 11975–11986

    2023

  46. [55]

    Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang. [n. d.]. Negative- Aware Attention Framework for Image-Text Matching. ([n. d.])

  47. [56]

    Sibei Yang, Guanbin Li, and Yizhou Yu. 2019. Dynamic graph attention for referring expression comprehension. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . 4644–4653

    2019

  48. [57]

    Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. 2022. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision (ECCV). Springer, 598–615

    2022

  49. [58]

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 (2023)

    2023 arXiv

  50. [64]

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra

  51. [65]

    Detecting twenty-thousand classes using image-level supervision. InECCV

  52. [67]

    Instance quality is poor

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020. Deformable detr: Deformable transformers for end-to-end object detection.arXiv preprint arXiv:2010.04159 (2020). Referring Expression Instance Retrieval and A Strong End-to-End Baseline Conference ac...

    2020 arXiv

  53. [2016]

    In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14

    Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 . Springer, 69–85

    2016

  54. [2019]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2019)

    Learning to compose and reason with language tree structures for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2019)

    2019

  55. [2020]

    Neural Information Processing Systems,Neural Information Processing Systems (Jun 2020)

    Large-Scale Adversarial Training for Vision-and-Language Representation Learning. Neural Information Processing Systems,Neural Information Processing Systems (Jun 2020)

    2020

  56. [2021]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . 1769–1779

  57. [2022]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (Feb 2022), 684–696

    Learning to Compose and Reason with Language Tree Structures for Visual Grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence (Feb 2022), 684–696. doi:10.1109/tpami.2019.2911066

    2022

  58. [2023]

    arXiv preprint arXiv:2306.15195 (2023)

    Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.