Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a vision-language model on two MLLM-written retrieval tasks, using only unlabeled images, is enough to give it zero-shot composed image retrieval, raising CIRR recall@1 from 28.3 to 39.30.

desk verdict Strong empirical gains in ZS-CIR, but the theoretical justification is overclaimed and the training objectives do not match the derivation. read the letter →

arxiv 2505.19707 v1 pith:SRA4W4BU submitted 2025-05-26 cs.CV cs.IR

Rong-Cheng Tu , Zhao Jin , Jingyi Liao , Xiao Luo , Yingjie Wang , Li Shen , Dacheng Tao This is my paper · ORCID
classification cs.CVcs.IR
keywords zero-shotcomposedimageretrievalvision-languagemodelfine-tuningmultimodallargelanguagesynthetictextsupervisiontargettext-to-imagejointinferenceQ-Former
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a vision-language model (VLM) can acquire zero-shot composed image retrieval — finding a target image from a reference image plus a text modification — without a single labeled training triplet. The supervision is written by a multimodal large language model (MLLM) from unlabeled images: a modification text describing some change, a target text describing the imagined modified image, and a plain caption of the original. The VLM is then fine-tuned on two contrastive tasks built from these synthetic texts: matching a composed query (image plus modification) to its target text, and matching a caption to its image. At inference the retrieval score averages the VLM's own composed-query-to-image similarity with the MLLM-generated target text's similarity to candidate images. If the claim holds, the prevailing adapter-based route to zero-shot composed retrieval is replaceable by direct fine-tuning, and the payoff is concrete: an 11-point absolute gain in recall@1 on CIRR over the strongest baseline.

What carries the argument

The load-bearing identity is a probability decomposition. Assuming a latent textual intent $t_i$ fully captures what a composed query $(x_i, m_i)$ is asking for, and that the target image is conditionally independent of the query given $t_i$, the retrieval probability factorizes as $P(x^t_i \mid x_i, m_i) = \sum_t P(x^t_i \mid t) P(t \mid x_i, m_i)$: one term is a text-to-image retrieval problem and the other a target-text retrieval problem. The paper turns each term into an InfoNCE training task over a shared set of learnable query tokens in a Q-Former, a transformer module that condenses multimodal inputs into fixed-size feature vectors, and measures similarity as the average of the best token-pair cosine matches. A multimodal LLM (MiniCPM-V-2.6) supplies all the texts — modifications, target descriptions, and captions — from unlabeled images, and inference combines the two signals by averaging the VLM's composed-query-to-image similarity with the similarity between the MLLM-generated target text and the candidate image.

What would settle it

Shuffle the MLLM-generated target texts among different reference images before fine-tuning, so each composed query is trained to align with a target text that does not describe its modification, and measure CIRR recall@1. If the score stays near the reported 39.30, the target-text alignment task is not actually doing the work the paper attributes to it; if it collapses toward the roughly 10.55 of the Image+Text baseline, the MLLM-written supervision is load-bearing exactly as claimed.

Watch

Extended reading notes

Core claim

The paper's central claim is that optimizing two complementary retrieval objectives on MLLM-generated data directly fine-tunes a pretrained VLM into a composed-image retriever, with no pseudo-text adapter and no annotated triplets. The first objective, target text retrieval, aligns the representation of the composed query $(x_i, m_i)$ with the representation of the MLLM-written target text $t_i$; the second, text-to-image retrieval, aligns the representation of an MLLM-generated caption $c_i$ with the image it describes. The paper reports state-of-the-art zero-shot results on all three benchmarks it evaluates (FashionIQ, CIRCO, and CIRR), with the largest single gain on CIRR, where recall@1 reaches 39.30 against 28.3 for the MLLM-I2W baseline.

Load-bearing premise

The MLLM-generated modification texts, target texts, and captions have to be accurate and consistent descriptions of the intended retrieval semantics, because they are the only supervision the VLM receives; the paper itself notes in its Appendix A that MLLMs still hallucinate, and any wrong or inconsistent synthetic text trains the VLM to align composed queries with the wrong meaning.

Editorial extensions

If this is right

  • Adapter-based zero-shot composed retrieval is not the only route: direct VLM fine-tuning on MLLM-synthesized data outperforms pseudo-text-token adapters on every benchmark reported.
  • Retrieval quality is jointly bounded by the VLM's cross-modal alignment and the MLLM's text fidelity, so gains should compound with either better VLMs or better MLLMs.
  • Both inference signals carry weight: removing the MLLM target-text similarity drops FashionIQ R@10 from 34.84 to 32.52, and removing the VLM composed-query similarity drops it to 26.66.
  • Both training objectives contribute: ablating the target-text retrieval task hurts on all three benchmarks (e.g., CIRCO mAP@5 from 21.69 to 15.18), and ablating the caption-to-image task also lowers most scores.
  • Because the whole recipe consumes only unlabeled images and an MLLM, the same two-task pipeline can in principle be pointed at a new domain with no annotation effort, as long as the MLLM can describe that domain's images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: sample several target texts per composed query during training and treat them as positives, which the probability decomposition in Eq. (4) invites since the sum over latent $t$ is approximated by a single sample.
  • Nothing in the derivation is specific to the Q-Former architecture, so the same two-task, MLLM-supervised recipe could plausibly be transferred to other dual-encoder VLMs, though the paper only demonstrates it on BLIP-2.
  • If MLLM hallucination is the binding constraint, a cheap untested filter would be self-consistency: keep only target texts that two independent MLLM samples agree on, which should raise the quality of the synthetic supervision.
  • The same latent-intent decomposition has a natural home outside retrieval — for instance, judging text-guided image edits by whether an edited image matches an unannotated target description.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MVFT-JI, a zero-shot composed image retrieval method. Instead of training adapters that map reference images to pseudo-text tokens, the method fine-tunes a BLIP-2-style Q-Former VLM directly on two MLLM-generated training tasks: (i) target text retrieval, where a composed query (reference image plus modification text) is aligned with an MLLM-generated description of the modified target, and (ii) text-to-image retrieval, where MLLM-generated image captions are aligned with their images. At inference, the method computes an equally weighted combination of the VLM's composed-query-to-image similarity and the similarity between an MLLM-generated target-text description and each candidate image. The authors report large improvements over prior ZS-CIR baselines, e.g., CIRR R@1 of 39.30 versus 28.3 for MLLM-I2W, and CIRCO mAP@5 of 21.69 versus 18.57 for CIReVL. The paper also provides a theoretical derivation in Section 3.1 aimed at showing that jointly optimizing the two tasks optimizes the composed retrieval probability.

Significance. If the empirical results hold, this is a practically significant contribution: the method uses only unlabeled images plus an MLLM, avoids pseudo-text adapter training, and reports consistent gains across three diverse benchmarks. The ablation study is valuable: it shows that both training objectives contribute on at least two of the three datasets, and that both inference-time similarity terms contribute on all datasets. The paper also reports three-seed averages and fixed random seeds, which is a useful reproducibility practice. The main weakness is that the theoretical justification in Section 3.1 does not actually match the training objectives as constructed; this is a load-bearing flaw because the abstract, introduction, and conclusion all invoke the theory as support for the method. The empirical contribution could stand on its own if the theory is corrected or explicitly reframed as a heuristic.

major comments (3)
  1. [Section 3.1, Eq. (4)] The derivation requires the same latent text t to appear in both factors P(x_t | t) and P(t | x_i, m), but Section 3.2 states that the latent textual intents for the two tasks 'are generated separately and thus differ between these two tasks.' The text-to-image objective L_c in Eq. (13) aligns image x_i with its caption c_i = MLLM(x_i), not with the target text t_i = MLLM(x_i, m_i) used in the target text retrieval objective L_t. Consequently, the right-hand side of Eq. (4) does not factor into the two conditional probabilities that the losses actually optimize, so the claimed theoretical justification does not follow from the stated assumptions. Please either supply a corrected derivation with a shared latent text (e.g., a proper two-stage variational bound) or explicitly relabel the joint-training rationale as a heuristic.
  2. [Section 3.1-3.3] Equation (4) sums over all latent texts t, but the method samples a single text t_i per training instance via the MLLM and uses a batch-wise InfoNCE loss. No variational lower bound, importance weighting, or other argument is provided to justify replacing the sum over t with one sample; the softmax over the batch is not a Monte Carlo estimate of Eq. (4). The paper should state this approximation explicitly and discuss its bias, or remove the claim in Section 3.3 that minimizing L in Eq. (14) 'effectively' maximizes P(x_t | x_i, m).
  3. [Appendix A and Section 3.2] The entire training supervision comes from MLLM-generated text, and the paper acknowledges in Appendix A that MLLMs 'are still prone to hallucinations and other generation errors.' Because this synthetic text is the sole source of training targets, the zero-shot generalization claim rests on the quality of these texts. The ablations with InternVL25 and prompt set P' partially address sensitivity, but the paper does not quantify annotation noise (e.g., a human evaluation or an automatic consistency check between generated target texts and actual target images). I consider this a correctness-risk concern rather than a rejection basis, but it should be acknowledged and ideally measured in the main paper.
minor comments (5)
  1. [Section 4.1.3] The text says 'a signle NVIDIA A100 40GB GPU'; 'signle' should be 'single'.
  2. [Table 2] The header 'R s@2' has a stray space, and the Rs@1/Rs@2 values for MLLM-I2W are missing, so the 'Avg' column cannot be compared across all baselines.
  3. [Table 4] The column header 'CIRO' is inconsistent with 'CIRCO' used elsewhere in the paper; the abbreviation should be unified.
  4. [Eq. (19)] The fusion weight is fixed at 1/2; a small sensitivity study over the fusion coefficient would help clarify robustness, since this is the only inference-time hyperparameter introduced by the method.
  5. [Section 3.4, Eq. (17)] The same prompt Ptt is used both for generating training target texts and for generating the inference-time target text, so the same MLLM supplies both the training signal and the additional retrieval signal; this potential train/inference coupling should be stated more explicitly.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: benchmark gains are externally measured, but the theoretical justification contains a load-bearing mismatch between Eq. (4)'s shared latent text and the separately generated MLLM texts.

full rationale

The paper's concrete contribution is an empirical pipeline: MLLM-generated pseudo-labels from unlabeled images, direct VLM fine-tuning with two InfoNCE losses (Lt, Lc), and evaluation on FashionIQ, CIRCO, and CIRR against published baselines. These results are externally falsifiable and are not produced by fitting a parameter and then relabeling it as a prediction; the inference fusion weight 1/2 is fixed by hand, and the ablations (w/o Lt, w/o Lc, w/o hat-s, w/o tilde-s) show sensitivity to each component. Self-citations in the reference list are not load-bearing, and no uniqueness theorem is imported from the authors' prior work. The flagged issue is a rigor gap in the theoretical claim, not circularity: Eq. (4) decomposes P(x_t|xi,mi) = sum_t P(x_t|t) P(t|xi,mi) with one shared latent text t, but Section 3.2 states 'the latent textual intents ti for each task are generated separately and thus differ between these two tasks.' L_t is built from t_i = MLLM(xi, mi) while L_c is built from c_i = MLLM(xi), so L_c optimizes P(x_i|c_i) rather than P(x_t_i|t_i) for the same t_i; the joint loss is therefore not literally the objective in Eq. (4). Replacing the sum over t by a single MLLM sample (Section 3.2) is likewise presented without a variational or other bound. These are missing-support problems that weaken the 'theoretical justification' and should be corrected or reframed as a heuristic, but they do not make the benchmark measurements circular, since the reported gains are measured rather than deduced from Eq. (4). Appendix A's acknowledgement that MLLMs are 'still prone to hallucinations and other generation errors' is consistent with this reading: the method's risk is supervision noise, not by-construction equivalence. Overall the empirical core is self-contained and externally validated, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method's training signal is entirely synthetic, generated by an MLLM, so the central claim depends on three unproven assumptions: the latent-text decomposition (Section 3.1), the conditional-independence factorization (Eq. 3), and the quality of MLLM-generated supervision (Appendix A). The method also has a handful of unspecified hyperparameters (fusion weight, number of queries, epochs). No new entities are posited.

free parameters (3)
  • Fusion weight alpha for combining composed-query similarity and MLLM text similarity at inference = 1/2
    Eq. (19) sets sqj = (sqj_hat + sqj_tilde)/2. The weight is fixed by hand and no ablation sweeps over it, so the optimal balance is not established.
  • Number of Q-Former query tokens k
    Used in Eqs. (8) to (11); the value is never stated in the paper and must be inferred from the BLIP2 implementation (typically 32), which is a reproducibility gap.
  • Number of fine-tuning epochs
    Section 4.1.3 gives batch size and learning rate but not the epoch count used for the main results, so the training budget is a free choice.
assumptions (5)
  • domain assumption For every composed query (xi, mi) there exists a latent textual description ti capturing the retrieval intent (Section 3.1, latent semantic assumption).
    This is posited, not derived. The entire theoretical decomposition in Eq. (2) rests on it.
  • domain assumption Conditional independence P(xt | xi, mi, ti) = P(xt | ti), i.e., the target image is conditionally independent of the composed query given the latent text (Eq. 3).
    This strong assumption lets Eq. (4) factorize the retrieval probability into two separate tasks; it is not empirically validated.
  • domain assumption A single MLLM-generated ti per query is a sufficient surrogate for the full latent-text distribution in Eq. (2).
    Section 3.2 states that enumerating all latent intents is infeasible, so they sample one; this replaces a sum with a point estimate.
  • domain assumption BLIP2's pretrained representations provide a suitable initialization for zero-shot CIR after fine-tuning.
    The method relies on BLIP2 (ViT-L/14) as the backbone and does not test other VLMs, so the results may be specific to this initialization.
  • standard math Standard probability identities: law of total probability and conditioning used in Eqs. (2) and (4).
    These are standard and unproblematic, listed for completeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval." pith.science (2026). https://pith.science/paper/SRA4W4BU

@misc{pith2026250519707,
  author       = {Pith},
  title        = {Pith review of: MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRA4W4BU}},
  note         = {Machine review of arXiv:2505.19707}
}
read the original abstract

Existing Zero-Shot Composed Image Retrieval (ZS-CIR) methods typically train adapters that convert reference images into pseudo-text tokens, which are concatenated with the modifying text and processed by frozen text encoders in pretrained VLMs or LLMs. While this design leverages the strengths of large pretrained models, it only supervises the adapter to produce encoder-compatible tokens that loosely preserve visual semantics. Crucially, it does not directly optimize the composed query representation to capture the full intent of the composition or to align with the target semantics, thereby limiting retrieval performance, particularly in cases involving fine-grained or complex visual transformations. To address this problem, we propose MLLM-Guided VLM Fine-Tuning with Joint Inference (MVFT-JI), a novel approach that leverages a pretrained multimodal large language model (MLLM) to construct two complementary training tasks using only unlabeled images: target text retrieval taskand text-to-image retrieval task. By jointly optimizing these tasks, our method enables the VLM to inherently acquire robust compositional retrieval capabilities, supported by the provided theoretical justifications and empirical validation. Furthermore, during inference, we further prompt the MLLM to generate target texts from composed queries and compute retrieval scores by integrating similarities between (i) the composed query and candidate images, and (ii) the MLLM-generated target text and candidate images. This strategy effectively combines the VLM's semantic alignment strengths with the MLLM's reasoning capabilities.

Figures

Figures reproduced from arXiv: 2505.19707 by the authors.

Figure 1
Figure 1. The training and inference framework of MVFT-JI. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Prompt template Pm for modification text generation. # Task Description You are an expert in object image editing and visualization. Your task is to imagine how the object in the given image would look after being modified according to the description "{modification_text}". Write exactly one clear and concise sentence describing only the modified image, focusing on the most important object details, such as: - The t… view at source ↗
Figure 3
Figure 3. Prompt template Ptt for target text generation. # Task Description You are an expert in image analysis and description. Your job is to generate one precise and concise sentence that fully describes the content of the given image. Focus on the most important details, such as: - The primary objects or elements in the image. - The relationships, positions, or actions of these objects. - The overall setting, background,… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Prompt template Pc for image caption generation. # Task Description You are an expert in understanding and modifying image content. Your job is to generate one concise modification text for the given image based on its content. The modification should focus on one or m…
Figure 5
Figure 5. Figure 5: Prompt template P ′ m for modification text generation in the ablation study. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompt template P ′ tt for target text generation in the ablation study. # Task Description You are an expert in image analysis and description. Your job is to generate one precise and concise sentence that fully describes the content of the given image. Focus on the m…
Figure 7
Figure 7. Figure 7: Prompt template P ′ c for image caption generation in the ablation study. Additionally, we show some examples of the <Reference Image, Modification Text, Target Text> triplets generated by the multimodal large language model (MLLM) in [PITH_FULL_IMAGE:figures/full_fig…
Figure 8
Figure 8. Figure 8: Examples of <Reference Image, Modification Text, Target Text> triplets generated by the [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Examples of <Reference Image, Caption Text> generated by the MLLM. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Retrieved results on the three datasets. The green box indicates the target image. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoVR-R:Reason-Aware Composed Video Retrieval

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Zero-shot LMM reasoning over edit after-effects (states, phases, camera, tempo) plus a new CoVR-R benchmark yields large recall gains on implicit-effect composed video retrieval without task-specific training.

  2. SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SPAZER, a VLM-driven agent, combines 3D rendered views with 2D camera images in a progressive pipeline to achieve state-of-the-art zero-shot 3D visual grounding.

Reference graph

Works this paper leans on

59 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Lim, and Hung Keng Pung

    Tat-Seng Chua, S.-K. Lim, and Hung Keng Pung. Content-based retrieval of segmented images. In Meera Blattner and John O. Limb, editors, Proceedings of the Second ACM International Conference on Multimedia ’94, San Francisco, CA, USA, October 15-20, 1994, pages 211–218. ACM Press, 1994. doi: 10.1145/192593.192658. URL https://doi.org/10.1145/192593. 192658. 1, 3

    work page arXiv 1994

  2. [2]

    MLS3RDUH: deep unsupervised hashing via manifold based local semantic similarity structure reconstructing

    Rong-Cheng Tu, Xianling Mao, and Wei Wei. MLS3RDUH: deep unsupervised hashing via manifold based local semantic similarity structure reconstructing. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3466–3472. ijcai.org, 2020. doi: 10.24963/IJCAI.2020/479. URL https:...

    work page doi:10.24963/ijcai.2020/479 2020

  3. [3]

    Unsupervised hashing with semantic concept mining

    Rong-Cheng Tu, Xian-Ling Mao, Kevin Qinghong Lin, Chengfei Cai, Weize Qin, Wei Wei, Hongfa Wang, and Heyan Huang. Unsupervised hashing with semantic concept mining. Proc. ACM Manag. Data, 1(1):3:1–3:19, 2023. doi: 10.1145/3588683. URL https://doi.org/10. 1145/3588683. 1

    doi:10.1145/3588683 2023

  4. [4]

    Where does the performance improvement come from?: - A reproducibility concern about image-text retrieval

    Jun Rao, Fei Wang, Liang Ding, Shuhan Qi, Yibing Zhan, Weifeng Liu, and Dacheng Tao. Where does the performance improvement come from?: - A reproducibility concern about image-text retrieval. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR ’22: The 45th International ACM SIGIR Confer...

    arXiv 2022

  5. [5]

    Unsupervised cross-modal hashing with modality-interaction

    Rong-Cheng Tu, Jie Jiang, Qinghong Lin, Chengfei Cai, Shangxuan Tian, Hongfa Wang, and Wei Liu. Unsupervised cross-modal hashing with modality-interaction. IEEE Trans. Circuits Syst. Video Technol., 33(9):5296–5308, 2023. doi: 10.1109/TCSVT.2023.3251395. URL https://doi.org/10.1109/TCSVT.2023.3251395

    arXiv 2023

  6. [6]

    Deep cross-modal proxy hashing

    Rong-Cheng Tu, Xian-Ling Mao, Rongxin Tu, Bin-Bin Bian, Chengfei Cai, Hongfa Wang, Wei Wei, and Heyan Huang. Deep cross-modal proxy hashing. IEEE Trans. Knowl. Data Eng., 35 (7):6798–6810, 2023. doi: 10.1109/TKDE.2022.3187023. URL https://doi.org/10.1109/ TKDE.2022.3187023. 1

    arXiv 2023

  7. [8]

    Image retrieval on real-life images with pre-trained vision-and-language models

    Zheyuan Liu, Cristian Rodriguez Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 2105–2114. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00213. URL https://doi. org/10...

    arXiv 2021

  8. [9]

    Composed image retrieval using contrastive learning and task-oriented clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Composed image retrieval using contrastive learning and task-oriented clip-based features. ACM Trans. Multim. Comput. Commun. Appl. , 20(3):62:1–62:24, 2024. doi: 10.1145/3617597. URL https: //doi.org/10.1145/3617597

    doi:10.1145/3617597 2024

Show all 59 references
  1. [10]

    Dual-path semantic construction network for composed query-based image retrieval

    Shenshen Li. Dual-path semantic construction network for composed query-based image retrieval. In Ioannis Kompatsiaris, Jiebo Luo, Nicu Sebe, Angela Yao, Vasileios Mazaris, Symeon Papadopoulos, Adrian Popescu, and Zi Helen Huang, editors, Proceedings of the 2023 ACM Internatio...

    2023

  2. [11]

    Multi-modal transformer with global-local alignment for composed query image retrieval

    Yahui Xu, Yi Bin, Jiwei Wei, Yang Yang, Guoqing Wang, and Heng Tao Shen. Multi-modal transformer with global-local alignment for composed query image retrieval. IEEE Trans. Multim., 25:8346–8357, 2023. doi: 10.1109/TMM.2023.3235495. URL https://doi.org/ 10.1109/TMM.2023.3235495

    2023

  3. [12]

    Composed image retrieval via cross relation network with hierarchical aggregation transformer

    Qu Yang, Mang Ye, Zhaohui Cai, Kehua Su, and Bo Du. Composed image retrieval via cross relation network with hierarchical aggregation transformer. IEEE Trans. Image Process., 32: 4543–4554, 2023. doi: 10.1109/TIP.2023.3299791. URL https://doi.org/10.1109/TIP. 2023.3299791

    2023

  4. [13]

    Self- training boosted multi-factor matching network for composed image retrieval

    Haokun Wen, Xuemeng Song, Jianhua Yin, Jianlong Wu, Weili Guan, and Liqiang Nie. Self- training boosted multi-factor matching network for composed image retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 46(5):3665–3678, 2024. 2

    2024

  5. [14]

    Pic2word: Mapping pictures to words for zero-shot composed image retrieval

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada,...

    2023

  6. [15]

    Zero-shot composed image retrieval with textual inversion

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 15292–15301. IEEE,

    2023

  7. [16]

    Fine- grained textual inversion network for zero-shot composed image retrieval

    Haoqiang Lin, Haokun Wen, Xuemeng Song, Meng Liu, Yupeng Hu, and Liqiang Nie. Fine- grained textual inversion network for zero-shot composed image retrieval. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors,Proceedings of the 47th I...

    2024

  8. [17]

    Knowledge-enhanced dual-stream zero- shot composed image retrieval

    Yucheng Suo, Fan Ma, Linchao Zhu, and Yi Yang. Knowledge-enhanced dual-stream zero- shot composed image retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 26941–26952. IEEE,

    2024

  9. [18]

    Zero-shot composed image retrieval considering query-target relationship leveraging masked image- text pairs

    Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Zero-shot composed image retrieval considering query-target relationship leveraging masked image- text pairs. CoRR, abs/2406.18836, 2024. doi: 10.48550/ARXIV .2406.18836. URL https: //doi.org/10.48550/...

    2024 doi arXiv

  10. [19]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In Marina Meila an...

    2021

  11. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edit...

    2023

  12. [21]

    Selvaraju, Akhilesh Gotmare, Shafiq R

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, a...

    2021

  13. [22]

    Seeing what you miss: Vision-language pre-training with semantic completion learning

    Yatai Ji, Rongcheng Tu, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, and Wei Liu. Seeing what you miss: Vision-language pre-training with semantic completion learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, ...

    2023

  14. [23]

    Global and local semantic completion learning for vision-language pre-training

    Rong-Cheng Tu, Yatai Ji, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, and Wei Liu. Global and local semantic completion learning for vision-language pre-training. CoRR, abs/2306.07096, 2023. doi: 10.48550/ARXIV .2306.07096. URL https: //doi.org/1...

    2023 doi arXiv

  15. [24]

    Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024 arXiv

  16. [25]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 doi arXiv

  17. [26]

    Minicpm-v: A GPT-4V level MLLM on your phone.CoRR, abs/2408.01800,

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. M...

    arXiv

  18. [27]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, 12 Dahua Lin, and Jiaqi...

    2024 arXiv

  19. [28]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. CoRR, abs/2411.10442, 2024. doi: 10.48550/...

    2024 doi arXiv

  20. [29]

    URL https://doi.org/10.48550/arXiv.2408

    doi: 10.48550/ARXIV .2408.01800. URL https://doi.org/10.48550/arXiv.2408. 01800. 2, 7

    doi arXiv

  21. [30]

    Spagent: Adaptive task decomposition and model selection for general video generation and editing

    Rong-Cheng Tu, Wenhao Sun, Zhao Jin, Jingyi Liao, Jiaxing Huang, and Dacheng Tao. Spagent: Adaptive task decomposition and model selection for general video generation and editing. CoRR, abs/2411.18983, 2024. doi: 10.48550/ARXIV .2411.18983. URL https://doi.org/ 10.48550/arXiv...

    2024 doi arXiv

  22. [31]

    Yu, and Ming Zhang

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao...

    2025 doi arXiv

  23. [32]

    Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. CoRR, abs/2...

    2023 doi arXiv

  24. [33]

    Kankanhalli

    Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan S. Kankanhalli. Improving context understanding in multimodal large language models via multimodal composition learning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21- 27, 20...

    2024

  25. [34]

    Grounding language models to images for multimodal inputs and outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine L...

    2023

  26. [35]

    Multi- modal retrieval augmented multi-modal generation: A benchmark, evaluate metrics and strong baselines

    Zi-Ao Ma, Tian Lan, Rong-Cheng Tu, Yong Hu, Heyan Huang, and Xian-Ling Mao. Multi- modal retrieval augmented multi-modal generation: A benchmark, evaluate metrics and strong baselines. CoRR, abs/2411.16365, 2024. doi: 10.48550/ARXIV .2411.16365. URL https: //doi.org/10.48550/a...

    2024 doi arXiv

  27. [36]

    Partial-softmax loss based deep hashing

    Rong-Cheng Tu, Xian-Ling Mao, Jia-Nan Guo, Wei Wei, and Heyan Huang. Partial-softmax loss based deep hashing. In Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia, editors, WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2...

    2021

  28. [37]

    Weighted gaussian loss based hamming hashing

    Rong-Cheng Tu, Xian-Ling Mao, Cihang Kong, Zihang Shao, Ze-Lin Li, Wei Wei, and Heyan Huang. Weighted gaussian loss based hamming hashing. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran, editors, MM ’21: ACM...

    2021

  29. [38]

    Compo- sitional image retrieval via instruction-aware contrastive learning

    Wenliang Zhong, Weizhi An, Feng Jiang, Hehuan Ma, Yuzhi Guo, and Junzhou Huang. Compo- sitional image retrieval via instruction-aware contrastive learning. CoRR, abs/2412.05756, 2024. doi: 10.48550/ARXIV .2412.05756. URLhttps://doi.org/10.48550/arXiv.2412.05756. 2

    2024 doi arXiv

  30. [39]

    Data-aware proxy hashing for cross-modal retrieval

    Rong-Cheng Tu, Xian-Ling Mao, Wenjin Ji, Wei Wei, and Heyan Huang. Data-aware proxy hashing for cross-modal retrieval. In Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete, editors, Proceedings of the 46th International ACM...

    2023

  31. [40]

    Enhance composed image retrieval via multi-level collaborative localization and semantic activeness perception

    Gangjian Zhang, Shikui Wei, Huaxin Pang, Shuang Qiu, and Yao Zhao. Enhance composed image retrieval via multi-level collaborative localization and semantic activeness perception. IEEE Trans. Multim., 26:916–928, 2024. doi: 10.1109/TMM.2023.3273466. URL https: //doi.org/10.1109...

    2024

  32. [41]

    Unsupervised cross-modal hashing via semantic text mining

    Rong-Cheng Tu, Xian-Ling Mao, Qinghong Lin, Wenjin Ji, Weize Qin, Wei Wei, and Heyan Huang. Unsupervised cross-modal hashing via semantic text mining. IEEE Trans. Multim., 25: 8946–8957, 2023. doi: 10.1109/TMM.2023.3243608. URL https://doi.org/10.1109/TMM. 2023.3243608. 3

    2023

  33. [42]

    Language- only efficient training of zero-shot composed image retrieval

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun. Language- only efficient training of zero-shot composed image retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13225–13234. IE...

    2024

  34. [43]

    Vision-by- language for training-free compositional image retrieval

    Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. Vision-by- language for training-free compositional image retrieval. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2...

    2024

  35. [44]

    Newsam, and Kofi Boakye

    Yuxin Tian, Shawn D. Newsam, and Kofi Boakye. Image search with text feedback by additive attention compositional learning. CoRR, abs/2203.03809, 2022. doi: 10.48550/ARXIV .2203. 03809. URL https://doi.org/10.48550/arXiv.2203.03809. 3

    2022 doi arXiv

  36. [45]

    Target-guided composed image retrieval

    Haokun Wen, Xian Zhang, Xuemeng Song, Yinwei Wei, and Liqiang Nie. Target-guided composed image retrieval. In Proceedings of the ACM International Conference on Multimedia, pages 915–923. ACM, 2023

    2023

  37. [46]

    Sim- ple but effective raw-data level multimodal fusion for composed image retrieval

    Haokun Wen, Xuemeng Song, Xiaolin Chen, Yinwei Wei, Liqiang Nie, and Tat-Seng Chua. Sim- ple but effective raw-data level multimodal fusion for composed image retrieval. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors,Proceed- ings...

    2024

  38. [47]

    Com- posing text and image for image retrieval - an empirical odyssey

    Nam V o, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Com- posing text and image for image retrieval - an empirical odyssey. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6439–...

    2019 doi

  39. [48]

    isearle: Improving textual inversion for zero-shot composed image retrieval

    Lorenzo Agnolucci, Alberto Baldrati, Marco Bertini, and Alberto Del Bimbo. isearle: Improving textual inversion for zero-shot composed image retrieval. CoRR, abs/2405.02951, 2024. doi: 10.48550/ARXIV .2405.02951. URLhttps://doi.org/10.48550/arXiv.2405.02951. 3, 7

    2024 doi arXiv

  40. [49]

    MLLM-I2W: harnessing multimodal large language model for zero-shot composed image retrieval

    Tong Bao, Che Liu, Derong Xu, Zhi Zheng, and Tong Xu. MLLM-I2W: harnessing multimodal large language model for zero-shot composed image retrieval. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, 14 Proceedin...

    2025

  41. [50]

    URL https://doi.org/10.1145/3626772.3657727

    doi: 10.1145/3626772.3657727. URL https://doi.org/10.1145/3626772.3657727. 3

  42. [51]

    Sentence-level prompts benefit composed image retrieval

    Yang Bai, Xinxing Xu, Yong Liu, Salman Khan, Fahad Khan, Wangmeng Zuo, Rick Siow Mong Goh, and Chun-Mei Feng. Sentence-level prompts benefit composed image retrieval. arXiv preprint arXiv:2310.05473, 2023. 3

    2023 arXiv

  43. [52]

    Representation learning with contrastive predictive coding

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018. URL http://arxiv.org/abs/1807.03748. 6

    2018 arXiv

  44. [53]

    Fashion IQ: A new dataset towards retrieving images by natural language feedback

    Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grau- man, and Rogério Feris. Fashion IQ: A new dataset towards retrieving images by natural language feedback. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25...

    2021

  45. [54]

    Context- i2w: Mapping images to context-dependent words for accurate zero-shot composed image retrieval

    Yuanmin Tang, Jing Yu, Keke Gai, Jiamin Zhuang, Gang Xiong, Yue Hu, and Qi Wu. Context- i2w: Mapping images to context-dependent words for accurate zero-shot composed image retrieval. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty- Eighth AAAI...

    2024

  46. [55]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors,Proceedings of the 56th Annual Meeting of the Association for Computatio...

    2018 doi

  47. [56]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014...

    2014 doi

  48. [57]

    {modification_text}

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Comput...

    2019 doi

  49. [58]

    Bernstein, Alexander C

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhi- heng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115 (3):211–252, 2...

    2015 doi

  50. [59]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. 8

    2019

  51. [2023]

    URL https://doi.org/10.1109/ICCV51070

    doi: 10.1109/ICCV51070.2023.01407. URL https://doi.org/10.1109/ICCV51070. 2023.01407. 7, 16

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.