Composed Object Retrieval: Object-level Retrieval via Composed Expressions

Fahad Shahbaz Khan; Guanyu Yang; Jinxing Zhou; Nian Liu; Salman Khan; Tong Wang; Zongyan Han

REVIEW 4 major objections 5 minor 50 references

Composed Object Retrieval: Object-level Retrieval via Composed Expressions

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes Composed Object Retrieval, a task in which a model must output pixel-level masks of every target object matching a reference object plus a textual edit, and presents CORE, an end-to-end model that reports large gains ove

desk verdict The paper is a well-built new task/benchmark with a coherent model, but its headline superiority is not established because the CIR baselines were never trained on the target task. read the letter →

arxiv 2508.04424 v3 pith:XY2FZJZL submitted 2025-08-06 cs.CV

Tong Wang , Guanyu Yang , Nian Liu , Zongyan Han , Jinxing Zhou , Salman Khan , Fahad Shahbaz Khan This is my paper · ORCID

classification cs.CV

keywords composedobjectretrievalimageobject-levelandsegmentationregion-levelcontrastivelearningvision-languagemodelinstancebenchmarkcategorygeneralization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Composed image retrieval returns whole images, so a query like "change the color to light" cannot say which object in a cluttered scene the edit refers to. The paper sets out to fix that with Composed Object Retrieval (COR): given a reference image, a mask of the reference object, a target image, and a retrieval text that describes only attribute changes, the model must output a binary mask of every object in the target that matches. To support the task, the authors built COR127K, a 127,166-triplet benchmark drawn from public image datasets and generated by a large multimodal model, with base/novel splits to test generalization. Their model CORE fuses the masked reference region with the text, then uses a region-level contrastive loss to align the composed query with target objects and repel distractors. On the benchmark, CORE reports Dice gains of 35% on base and 21% on novel categories over the strongest image-level pipeline, which would make object-level retrieval practical for fine-grained search and editing.

What carries the argument

The load-bearing machinery is the composed-query representation itself. A Reference Region Embedding (RRE) module fuses the mask feature with image features through stacked semantic-enhancing blocks and produces an averaged semantic activation map, describing the reference object without class names. An Adaptive Vision-Text Interaction (AVTI) module computes per-modality attention weights and a scalar blend $\alpha$ to combine the reference embedding $F_{\text{rre}}$ with the text embedding $F_{\text{txt}}$ into one composed vector $F_{\text{avit}}$. A COR-oriented contrastive loss $L_{\text{cor}} = L_{\text{fg}} + L_{\text{bg}}$ then aligns that vector with masked-pooled foreground features

What would settle it

An independent audit of a random sample of COR127K triplets—say 1,000 pairs—where a different vision-language model or human annotators repeat the pipeline's verification steps (given only the reference image, mask, and retrieval text, find the target object and check that no distractor matches) would settle whether the labels are as clean and specific as claimed. If a substantial fraction of triplets are ambiguous or fail the audit, the benchmark's difficulty and the model rankings built on it fall apart.

Watch

Extended reading notes

Core claim

The paper's central claim is that object-level retrieval should be formulated as a segmentation problem driven by a composed expression, not as image-level ranking. The authors argue that a mask of the reference object plus a retrieval text that describes only attribute-level changes is enough to identify one or more target objects in another image, and that this formulation makes retrieval work for categories the model has never seen, because no class name is needed. To make the claim testable, they construct COR127K, a 127,166-triplet benchmark with base/novel splits, and train CORE, which reports 0.7703 Dice / 0.6955 IoU on Test-Base and 0.7102 / 0.6290 on Test-Novel, outperforming all se

Load-bearing premise

The correctness of the benchmark labels rests on one multimodal language model both writing every retrieval text and then verifying that the text is correct; if that loop is biased, every downstream comparison inherits the bias.

Editorial extensions

If this is right

If the reported results hold, composed retrieval changes from returning a candidate image that still needs manual screening to returning the exact object region the edit refers to.
The noun-free, mask-based query design means a user can retrieve by pointing at a reference object and typing an attribute change, without knowing or naming the object's category; the novel-category split is the direct test.
CORE handles multiple target objects and visually similar negatives in one forward pass, which image-level CIR cannot do without post-hoc detection and matching.
The ablations attribute the largest performance drop to removing the region-level contrastive loss, so foreground/background repulsion is the component carrying the task's discriminative power.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark's validation loop (Steps 8–10 of Appendix B) is self-referential: the same multimodal language model writes the retrieval texts and then certifies that they work. Before treating the leaderboard gaps as settled fact, a reader should ask for an independent human or second-model audit of a random triplet sample.
A strong test of the generalization claim would be to re-run evaluation with retrieval texts stripped of all category-related words; if performance collapses, the model may be relying on implicit category cues in the vision-language backbone rather than compositional attribute reasoning.
The background-repulsion loss is a generic distractor-suppression mechanism; the paper leaves unstated that it could transfer to open-vocabulary segmentation or referring-expression tasks.
Since the reference mask is the only identity anchor, COR could be driven interactively by clicks or boxes; that practical path—mask from user interaction rather than from dataset annotations—is implied but not explored.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Core thing to know: this is a real new task definition and a large dataset, and the model design is reasonable. But the main empirical claim—CORE significantly beats CIR pipelines—rests on a comparison where the baselines never see the training set. That is a load-bearing flaw, not a minor artifact.

The task is nicely conceived: reference mask + noun-free text + target image to produce a mask. That genuinely extends CIR to object-level. The dataset, COR127K, is big—127k triplets, 408 categories, with base/novel splits. The ablation studies are sensible, and the component contributions (RRE, AVTI, Lcor) are each justified by clear drops. The paper is mostly honest in its setup description.

The problem is Section 5.2. The seven CIR baselines use pretrained CIRR weights and are evaluated without fine-tuning; Detic+SAM is fixed. CORE is trained on 85,928 COR127K triplets. So the +35% Dice / +37% IoU likely reflects task adaptation plus architecture, not architecture alone. The paper calls them 'strong baselines,' but they've never been adapted to region-level retrieval. That conflation is more serious than the Qwen self-validation loop, though that loop is also worth noting: the same model writes the retrieval texts and filters them, with no human or independent-model audit. The paper does report that Step 10 masks out the target and checks false matches, which is a thoughtful control, but it is still the same judge. Also, no code or data is released (only 'will be released'), so reproducibility is currently limited.

Bottom line: this is a useful contribution to a subfield that needs object-level retrieval, but the evaluation needs to give baselines a fair chance—either fine-tune the CIR methods on COR127K or at least use a strong open-vocabulary detector/segmenter tuned to the task. The dataset itself should be released and ideally independently audited. I would send it to peer review with major revision demands, and I would want to see the revised comparison before trusting the headline numbers.

Referee Report

4 major / 5 minor

Summary. The paper introduces Composed Object Retrieval (COR), a task in which a model receives a reference image, a reference-object mask, a target image, and a retrieval text, and must output a segmentation mask for the target object(s) in the target image. The authors construct COR127K, a large automatically generated benchmark from COCO/LVIS and Qwen2.5-VL, with base/novel category splits, and propose CORE, an end-to-end model combining reference region embedding, adaptive vision-text interaction, and a region-level contrastive loss. Experiments report large gains over seven composed-image-retrieval baselines on Test-Base and Test-Novel.

Significance. The task itself is a meaningful extension of composed image retrieval to the object level, and a reusable benchmark plus a simple end-to-end baseline would be a useful community asset. The proposed architecture is reasonable, the dataset construction pipeline is described in unusual detail, and the authors commit to releasing code. If the empirical claims are supported by fair comparisons, the contribution is solid. However, the main evidence for superiority currently rests on comparisons that are unfair in at least two concrete ways, and the benchmark's automatic self-validation needs independent audit.

major comments (4)

[§5.1–§5.2, Table 2] The headline comparison is not a controlled test of the proposed architecture. CORE is trained on the 85,928 COR127K training triplets, while all seven CIR baselines are evaluated using only open-source CIRR-pretrained weights and are never fine-tuned on COR127K. The reported +35% Dice / +37% IoU gains therefore conflate task-specific training with architectural merit. To support the claim that CORE outperforms existing CIR-based pipelines, the authors should fine-tune the same baselines on COR127K (or otherwise give them access to the same training data) and also disentangle the contribution of Detic/SAM failures in the modular baseline.
[§3.2.2, Appendix B.2.3–B.2.4] The corpus is self-confirming: Qwen2.5-VL generates the retrieval texts in Step 8 and then filters them for semantic validity in Steps 9 and 10, and the same model is used in Steps 6 and 7 for attribute-distinguishability checks. No independent human or alternative-model audit is reported. Since the difficulty and noise level of every label propagate into all downstream numbers, the paper should report a human-validated sample (for example 500–1000 triplets) with inter-annotator agreement, and ideally a secondary VLM audit.
[§3.2.1, Table 1 and Appendix B.2.2 Step 4] The splits are claimed to be disjoint, but the reported image counts imply substantial overlap: Train + Test-Base + Test-Novel all-images counts sum to 36,824 while the total unique image count is 28,183. Because the split is category-level and COCO images contain multiple categories, the same image may appear both in Train and in Test-Novel through different objects. This can create image-level leakage and inflate the claimed novel-category generalization. The authors should quantify the overlap and enforce an image-disjoint split or analyze the effect of shared images.
[§5.1 and Table 3] In the multi-object settings, the Detic+CIR+SAM pipeline selects only the single most similar candidate region, so 2p0n, 2p1n, and 3p0n are structurally impossible for that baseline to solve. The large relative gains in these settings (+83.9%, +88.5%, +95.5% on Test-Base) are therefore not evidence of CORE's multi-object superiority. A fair comparison needs a multi-object-capable variant of the baseline, such as top-k or threshold-based selection, and the paper should state whether the reference-object mask is used at all by the baselines.

minor comments (5)

[Abstract / §3.2] The abstract states COR125K with 125,541 triplets, while the body consistently reports COR127K with 127,166 triplets. Please unify the name and statistics.
[§5.1] The metrics mDice and mIoU are reported but never defined. Specify how multi-object settings are aggregated (per object, per image, or per positive instance).
[§5.4 / Eq. (3)] The number of semantic subspaces K in the RRE module is a design choice with no sensitivity analysis. A short ablation on K would strengthen the paper.
[§5.1] Detic candidate-generation thresholds (confidence > 0.3, NMS threshold < 0.8) are presented without commentary. State whether any hyperparameter search was performed for the baseline pipeline.
[Table 4] The scaling results show small differences for some configurations; reporting results over multiple seeds with variance would make the comparisons more robust.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the model's predictions and benchmark comparisons are grounded in held-out masks and independent training, not derived from the paper's own claims.

full rationale

The paper's central claim is that CORE outperforms existing CIR pipelines on the COR127K benchmark. The model is trained on a train split and evaluated on held-out test splits with ground-truth masks from COCO/LVIS, so the reported Dice/IoU numbers are not constructed from the model's own inputs. The baseline comparison uses pretrained CIR models without fine-tuning, which is an experimental-design weakness rather than a circular derivation. The only potential circular element is the data-generation loop (Steps 8–10): Qwen2.5-VL generates the retrieval texts and the same model verifies them. However, this does not make the task's prediction circular—the semantic validity of the texts is an annotation-quality issue, not a mathematical equivalence between model outputs and training targets. There are no self-citations that carry the argument, no fitted parameters renamed as predictions, and no uniqueness theorems imported from the authors. The dataset construction pipeline, though automated and self-validated, does not feed back into the model's loss or evaluation in a way that forces the claimed results. Therefore, the derivation is self-contained with respect to the core claims.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims depend on the annotation pipeline and on the split/transfer assumptions listed above; these are the pieces the paper does not independently verify.

free parameters (5)

Semantic subspace count K = not reported
RRE uses K activation maps in Eq. 3; K is chosen implicitly and not ablated.
DINOv2 cosine similarity threshold = 0.8
Step 6 discards negative candidates above this threshold; controls distractor difficulty.
Object size filters = target 3%-80% area; reference >=5%; mask >20% bbox
Step 1 and Step 5 filtering criteria define dataset composition.
Category cap = 300 samples
Step 1 caps large categories, changing frequency distribution.
Detic baseline hyperparameters = confidence 0.3, NMS 0.8, 30 candidates
Detection-based baseline setup in Section 5.1; not part of CORE but part of comparison.

assumptions (5)

domain assumption Qwen2.5-VL judgments about object quality, attribute differences, and text-target match are correct.
Steps 2, 6, 7, 9, and 10 use Qwen's 0/1 responses as ground truth.
domain assumption COCO2017 and LVIS instance masks are accurate.
All masks used for training and evaluation are inherited from these datasets.
domain assumption Frozen SigLIP and SAM features transfer to object-level composed retrieval and novel categories.
CORE freezes both backbones; novel-category performance depends on their generalization.
ad hoc to paper Category-level base/novel split is a valid generalization test with no image-level leakage.
The paper reports category disjointness but does not show image overlap between Train and Test; the same COCO image may contain objects belonging to different splits.
ad hoc to paper DINOv2 similarity threshold 0.8 defines a meaningful hard-negative regime.
This cutoff shapes the negative-object difficulty of the benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Composed Object Retrieval: Object-level Retrieval via Composed Expressions." pith.science (2026). https://pith.science/paper/XY2FZJZL

@misc{pith2026250804424,
  author       = {Pith},
  title        = {Pith review of: Composed Object Retrieval: Object-level Retrieval via Composed Expressions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XY2FZJZL}},
  note         = {Machine review of arXiv:2508.04424}
}

read the original abstract

Retrieving fine-grained visual content based on user intent remains a challenge in multimodal systems. Although current Composed Image Retrieval (CIR) methods combine reference images with retrieval texts, they are constrained to image-level matching and cannot localize specific objects. To this end, we propose Composed Object Retrieval (COR), a new object-level retrieval task that retrieves target object(s) from candidate objects in a target image and grounds the retrieved result with pixel-level masks. Given a reference object, its mask, a target image, and a retrieval text describing the desired modification, COR requires models to perform composed visual-textual reasoning rather than relying on explicit category names. This setting introduces several challenges, including fine-grained compositional matching, negative-object filtering under visually similar distractors, and flexible single- or multi-object retrieval. We construct COR125K, the first large-scale COR benchmark, containing 125,541 retrieval triplets across 408 categories with base/novel splits for evaluating category-level generalization. We also present CORE, a unified end-to-end model that integrates reference region encoding, adaptive vision-text interaction, and region-level contrastive learning to align composed representations with target objects while suppressing background and distractors. Extensive experiments demonstrate that CORE significantly outperforms existing CIR-based pipelines and strong baselines in both base and novel categories, establishing a simple and effective foundation for fine-grained object-level multimodal retrieval. Code will be released publicly at https://github.com/wangtong627/COR.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 46 canonical work pages

[1]

Probabilistic embeddings for cross-modal retrieval

Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8415–8424, 2021. 1

work page 2021
[2]

A decade survey of content based image retrieval using deep learning

Shiv Ram Dubey. A decade survey of content based image retrieval using deep learning. IEEE Transactions on Circuits and Systems for Video Technology, 32(5):2687–2704, 2021. 1

work page 2021
[3]

Image retrieval on real-life images with pre-trained vision-and-language models

Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2125–2134, 2021. 1, 6

work page 2021
[4]

Composed image retrieval using contrastive learn- ing and task-oriented clip-based features

Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Composed image retrieval using contrastive learn- ing and task-oriented clip-based features. ACM Transactions on Multimedia Computing, Communications and Applications, 20(3):1–24, 2023. 1, 2, 5, 6

work page 2023
[5]

Dy- namic weighted combiner for mixed-modal image retrieval

Fuxiang Huang, Lei Zhang, Xiaowei Fu, and Suqi Song. Dy- namic weighted combiner for mixed-modal image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2303–2311, 2024. 1, 5

work page 2024
[6]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Euro- pean conference on computer vision, pages 740–755. Springer,

work page
[7]

LVIS: A dataset for large vocabulary instance segmentation

Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019. 2, 3, 6

work page 2019
[8]

Qwen2.5- vl technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report. ...

arXiv 2025

Show all 50 references

[9]

Candidate set re-ranking for composed image retrieval with dual multi-modal encoder

Zheyuan Liu, Weixuan Sun, Damien Teney, and Stephen Gould. Candidate set re-ranking for composed image retrieval with dual multi-modal encoder. arXiv preprint arXiv:2305.16304,

arXiv
[10]

Sentence- level prompts benefit composed image retrieval

Xinxing Xu, Yong Liu, Salman Khan, Fahad Khan, Wangmeng Zuo, Rick Siow Mong Goh, Chun-Mei Feng, et al. Sentence- level prompts benefit composed image retrieval. In The Inter- national Conference on Learning Representations, 2024. 2, 6

2024
[11]

Improving composed image retrieval via contrastive learning with scaling positives and negatives

Zhangchi Feng, Richong Zhang, and Zhijie Nie. Improving composed image retrieval via contrastive learning with scaling positives and negatives. In Proceedings of the ACM Interna- tional Conference on Multimedia, pages 1632–1641, 2024. 2, 6

2024
[12]

Modality-agnostic attention fusion for visual search with text feedback

Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye. Modality-agnostic attention fusion for visual search with text feedback. arXiv preprint arXiv:2007.00145,

2007 arXiv
[13]

Dual compositional learning in interactive image retrieval

Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim. Dual compositional learning in interactive image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1771–1779, 2021. 2 8

2021
[14]

Composing text and image for im- age retrieval-an empirical odyssey

Nam V o, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei- Fei, and James Hays. Composing text and image for im- age retrieval-an empirical odyssey. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 6439–6448, 2019. 2

2019
[15]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...
[16]

Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In International con- ference on machine learning , pages 12888–12900. PMLR,
[17]

Target-guided composed image retrieval

Haokun Wen, Xian Zhang, Xuemeng Song, Yinwei Wei, and Liqiang Nie. Target-guided composed image retrieval. In Pro- ceedings of the ACM International Conference on Multimedia, pages 915–923, 2023. 2

2023
[18]

Data roaming and quality assessment for composed image retrieval

Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data roaming and quality assessment for composed image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2991–2999, 2024. 2

2024
[19]

Progressive learning for image retrieval with hybrid-modality queries

Yida Zhao, Yuqing Song, and Qin Jin. Progressive learning for image retrieval with hybrid-modality queries. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1012–1021, 2022. 2

2022
[20]

Compodiff: Versatile com- posed image retrieval with latent diffusion

Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun. Compodiff: Versatile com- posed image retrieval with latent diffusion. arXiv preprint arXiv:2303.11916, 2023. 2

2023 arXiv
[21]

Align before fuse: Vision and language representation learning with momen- tum distillation

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momen- tum distillation. Advances in neural information processing systems, 34:9694–9705, 2021. 2

2021
[22]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International confer- ence on machine learning, pages 19730–19742. PMLR, 2023. 2

2023
[23]

Sigmoid loss for language image pre-training

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lu- cas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 2, 5

2023
[24]

Segment anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 2, 5, 6

2023
[25]

De- coupling zero-shot semantic segmentation

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11583–11592, 2022. 4

2022
[26]

Open-vocabulary semantic segmentation with mask-adapted clip

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pa...

2023
[27]

Scaling open-vocabulary image segmentation with image-level labels

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European conference on computer vision, pages 540–557. Springer, 2022. 4

2022
[28]

Mask-adapter: The devil is in the masks for open-vocabulary segmentation

Yongkang Li, Tianheng Cheng, Bin Feng, Wenyu Liu, and Xinggang Wang. Mask-adapter: The devil is in the masks for open-vocabulary segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 14998–15008, 2025. 4

2025
[29]

Detecting twenty-thousand classes using image-level supervision

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨uhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European conference on computer vision, 2022. 6

2022
[30]

Bi-directional training for composed image retrieval via text prompt learning

Zheyuan Liu, Weixuan Sun, Yicong Hong, Damien Teney, and Stephen Gould. Bi-directional training for composed image retrieval via text prompt learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5753–5762, 2024. 6 9 Supplementary ...

2024
[31]

Contains at least {ins len} identifiable in- stances of the {cat name} category
[32]

Return 1 if all the conditions are met; otherwise, return 0

The {cat name} objects are complete, clear, and not occluded or blurry. Return 1 if all the conditions are met; otherwise, return 0. Only return 1 or 0, with no other content. where [IMAGE] is the image token, {cat name} is the category of the object, and {ins len} is the num-...
[33]

This ensures clear and discriminative reference objects for retrieval tasks

the image contains only one instance of the given category, and 2) the object occupies at least 5% of the image area for sufficient visibility. This ensures clear and discriminative reference objects for retrieval tasks. • Step 6: Target Object Selection. Target candidate ob- ...
[34]

The reference and target objects are distinguish- able in attributes such as color, shape, or action
[35]

Both objects are complete, clear, and free from motion blur or occlusion
[36]

reference

The target object in the target image is distin- guishable from same-category interfering objects. Return 1 if all the conditions are met; otherwise, return 0. Only return 1 or 0, with no other content. • Step 8: Retrieval Text Generation. We use QWen2.5- VL and a specific pro...
[37]

For animals: use dynamic attributes (e.g., pose, action, appearance, color, pattern, direction, style)
[38]

reference,

For inanimate objects: use static attributes (e.g., shape, position, color, direction, style). Avoid using the words “reference, ” “target, ”{cat - name}, or uncommon terms. Use simple lan- guage and follow the format: [(change1), (change2), (change3)] , with each change no lo...
[39]

change the color to bright

whether the retrieval text accurately describes the attribute changes. If the triplet passes this initial check, we remove the target object’s bounding box, leaving only the background, and reevaluate using the multi-modal large model to deter- mine whether the target object c...
[40]

In examples a and b, our model successfully retrieves objects that are challenging to describe textually, show- casing its strong semantic understanding
[41]

In examples c and e, the model accurately distinguishes negative objects within the same category, demonstrating precise discriminative capabilities
[42]

In examples d, e, and f, it effectively retrieves multi- ple target objects within a single scene, highlighting its ability to handle complex multi-object scenarios. C.2 Additional Ablation Studies In the setting of COR, we use the reference image Iref , the reference object m...
[43]

Only remove the retrieval text while retaining the ref- erence image and reference object mask ( i.e., Iref + Mref ), in which case the A VTI module will be discarded, and only the RRE module will be used
[44]

Only remove the reference object mask while retaining the reference image and retrieval text (i.e., Iref + Tret), in which case the RRE module will be discarded, and only the A VTI module will be used
[45]

Remove both the reference object mask and the retrieval text while retaining only the reference image (i.e., Iref ), in which case both the RRE and A VTI modules will be discarded
[46]

Remove both the reference image and reference object mask while retaining only the retrieval text (i.e., Tret), in which case both the RRE and A VTI modules will be discarded; The experimental results are presented in Tab. 5. The re- sults demonstrate that retrieval using comp...
[47]

Removing Retrieval Text ( Tret): When only the re- trieval text is excluded from the composed expression, the Dice score on Test-base decreases from 0.7703 to 0.7411 (a 3.79% drop), and on Test-novel, it falls from 0.7102 to 0.6664 (a 6.17% drop)
[48]

Removing Reference Object Mask (Mref ): When only the reference object mask is removed, the Dice score on Test-base drops from 0.7703 to 0.7319 (a 4.98% de- crease), and on Test-novel, it decreases from 0.7102 to 0.6712 (a 5.49% decrease)
[49]

The Dice score on Test-base falls from 0.7703 to 0.6770 (a 12.11% drop), and on Test-novel, it drops from 0.7102 to 0.6137 (a 13.59% drop)

Removing Both Retrieval Text (Tret) and Reference Object Mask (Mref ): When both the retrieval text and reference object mask are excluded (retaining only the reference image Iref ), performance degrades signifi- cantly. The Dice score on Test-base falls from 0.7703 to 0.6770 ...
[50]

Removing Both Reference Image ( Iref ) and Refer- ence Object Mask ( Mref ): When both the reference image and reference object mask are removed (retaining only the retrieval text Tret), the Dice score on Test-base decreases to 0.6767 (a 12.15% drop), and on Test-novel, it fal...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Probabilistic embeddings for cross-modal retrieval

Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8415–8424, 2021. 1

work page 2021

[2] [2]

A decade survey of content based image retrieval using deep learning

Shiv Ram Dubey. A decade survey of content based image retrieval using deep learning. IEEE Transactions on Circuits and Systems for Video Technology, 32(5):2687–2704, 2021. 1

work page 2021

[3] [3]

Image retrieval on real-life images with pre-trained vision-and-language models

Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2125–2134, 2021. 1, 6

work page 2021

[4] [4]

Composed image retrieval using contrastive learn- ing and task-oriented clip-based features

Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Composed image retrieval using contrastive learn- ing and task-oriented clip-based features. ACM Transactions on Multimedia Computing, Communications and Applications, 20(3):1–24, 2023. 1, 2, 5, 6

work page 2023

[5] [5]

Dy- namic weighted combiner for mixed-modal image retrieval

Fuxiang Huang, Lei Zhang, Xiaowei Fu, and Suqi Song. Dy- namic weighted combiner for mixed-modal image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2303–2311, 2024. 1, 5

work page 2024

[6] [6]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Euro- pean conference on computer vision, pages 740–755. Springer,

work page

[7] [7]

LVIS: A dataset for large vocabulary instance segmentation

Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019. 2, 3, 6

work page 2019

[8] [8]

Qwen2.5- vl technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report. ...

arXiv 2025