{"id":"10ede17f-beb0-4d24-81d8-1a23207dd98b","arxiv_id":"2412.11087","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CIR-LVLM fine-tunes Qwen-VL-Chat with LoRA and hybrid task and instance-specific prompts to produce query and target embeddings, achieving new state-of-the-art recall on Fashion-IQ, Shoes, and CIRR.","lead":"CIR-LVLM turns a large vision-language model into a single-pass encoder for composed image retrieval, combining a reference image and a relative caption into one query embedding. The authors report state-of-the-art results on three public benchmarks and argue that LVLMs can surpass traditional vision-language pretraining models on retrieval tasks that require reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'LVLM reasoning beats VLPM' claim rests on ablations that change scale, pooling, and encoder type simultaneously (Table 3 A.1-A.2); a scale-matched control is required before that claim can stand.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing gap: the encoder ablation changes scale and architecture together, so the paper does not isolate LVLM reasoning ability. I agree with the conditional verdict because the engineering contribution is plausible and internally consistent: the ablations in Table 3 B/C show that task prompts and instance-specific soft prompts each contribute, and the backbone comparison in Appendix C shows consistent behavior across LLaVA-1.5, LLaVA-1.6, and Qwen-VL-Chat. These are genuine, falsifiable design effects. However, the scientific claim as worded in contribution (4) requires a causal attribution to LVLM reasoning, and the current evidence cannot support that attribution. The scale-matched test I propose would settle whether the effect is due to reasoning/instruction-following or simply to a much larger and differently pretrained encoder. No further adjustment to the reader's verdict is needed: conditional acceptance with a request for this control is the right outcome. If the controlled experiment fails, the paper should be reframed as a strong engineering result without the 'first to show' scientific claim.","tokens_in":22000,"tokens_out":4214,"duration_ms":42744,"concrete_test":"Retrain CIR-LVLM on Fashion-IQ with the same full pipeline (same connector, task prompt, prompt pool, LoRA fine-tuning, position-weighted mean pooling, and training hyperparameters) but initialize the LLM from the base Qwen-7B checkpoint that Qwen-VL-Chat was built on, instead of from Qwen-VL-Chat. This keeps parameter count, architecture, and pooling identical and removes only the LVLM visual-instruction pretraining stage. If the Fashion-IQ Rmean stays near 66.17, the LVLM-specific reasoning ability is not the cause of the gains; if it drops toward the A.1/A.2 levels, the causal story is supported. Report results over at least three seeds with standard deviations, and repeat the same comparison on the CIRR benchmark to check generality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central scientific claim is contribution (4): 'the first to clearly show that in multimodal retrieval tasks that require reasoning, the LVLMs have the potential to surpass the VLPMs.' The evidence offered is (i) SOTA comparisons in Tables 1-2 and (ii) the encoder ablation in Table 3 rows A.1-A.2. Neither controls the factor the claim names. In A.1-A.2, Qwen-VL-Chat's 7B autoregressive LLM is replaced by the CLIP text encoder or the BLIP-2 text encoder; Appendix C states that an extra MLP is added after the connector and that the representation changes from position-weighted mean pooling to the [CLS] token. So parameter count, pooling strategy, and encoder architecture all change at once. The large Rmean gap (45.34 and 54.85 vs 66.17) could be substantially due to model scale or pooling, not to LVLM instruction-following or visual reasoning. The SOTA comparisons are also uncontrolled across training data, losses, backbones, and fine-tuning protocols. On CIRR, Table 2 shows CIR-LVLM is actually below SPRC on Recallsubset@1 and exceeds it on the reported 'Avg.' by only 0.06 points (81.44 vs 81.38), so the SOTA claim is marginal on that benchmark. The load-bearing missing piece is a same-scale, same-pipeline comparison that holds architecture decisions fixed while varying only the LVLM-specific pretraining and instruction-following capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CIR-LVLM, a framework that fine-tunes the Qwen-VL-Chat large vision-language model (LVLM) as a dual encoder for composed image retrieval (CIR). A Connector with learnable query embeddings converts the reference or target image into a sentence-level prompt, and a hybrid intent instruction module adds a task-level hard prompt and an instance-specific soft prompt selected from a learnable prompt pool using image and text keys. The model is trained with a batch-based contrastive loss and uses position-weighted mean pooling over the decoder outputs. The authors report state-of-the-art results on Fashion-IQ, Shoes, and CIRR, and claim to be the first to show that LVLMs can surpass vision-language pre-trained models (VLPMs) in multimodal retrieval tasks that require reasoning.","tokens_in":22322,"tokens_out":5443,"duration_ms":48015,"significance":"If the claims hold, the paper provides a practical recipe for adapting a 7B LVLM as a single-pass retriever with acceptable inference cost, and the hybrid prompt-pool module is a reasonable and interesting design. The paper is also transparent about failure cases and includes ablations over fine-tuning strategies, pooling strategies, and backbones. The key scientific claim, however, is not yet established: the evidence for LVLM superiority over VLPMs rests on comparisons that change model scale, pooling, and connector architecture at the same time. The empirical SOTA claim on CIRR is also marginal, with no error bars. The work is a solid systems/empirical contribution, but the headline conceptual claim needs either a scale-matched controlled experiment or a substantial tempering.","major_comments":[{"comment":"The ablation that replaces the LLM with CLIP or BLIP-2 text encoders simultaneously changes the parameter count (7B decoder vs. a much smaller text encoder), the pooling strategy (position-weighted mean pooling vs. [CLS] token), the projection head, and the connector design. The Rmean gap (66.17 vs. 45.34 and 54.85) is therefore not evidence that the LVLM's reasoning or instruction-following ability is the cause; it could be due to model scale, pooling, or the extra MLP described in Appendix C. To support the conclusion in Section 5 that 'compared to regular text encoders, the fine-tuned LLM is more advantageous in discerning user intent,' a scale-matched control is required, e.g., a 7B text-only encoder or a VLPM of comparable size using the same connector, pooling, and LoRA fine-tuning protocol.","section":"§5 Ablation study, Table 3 rows A.1-A.2; Appendix C"},{"comment":"On CIRR, CIR-LVLM's Avg. metric is 81.44 vs. SPRC's 81.38, a difference of 0.06 absolute points, while Recallsubset@1 is 1.53 points lower (79.12 vs. 80.65). No error bars, repeated runs, or significance testing are reported anywhere in the paper. Given the small margin, the abstract's claim of state-of-the-art performance 'across three prominent benchmarks' overstates the CIRR result. Please report variance over multiple seeds or a paired significance test, and adjust the claim to the metrics where the improvement is non-negligible.","section":"Table 2, CIRR results"},{"comment":"The cross-method comparisons in Tables 1 and 2 vary backbone, pretraining data, training loss, and fine-tuning protocol simultaneously, so the observed gains over VLPM-based methods cannot be attributed to the LVLM's user-intent reasoning per se. For example, the discussion in Section 4 claims the gain over TG-CIR is 'attributed to' the Connector and LLM capabilities, but no experiment isolates these factors. A baseline that uses the same training pipeline (LoRA, task prompts, prompt pool, position-weighted pooling) with a competitive VLPM of comparable scale is needed to make the mechanism claim credible. Without such a control, the SOTA numbers remain an empirical observation, not evidence for the 'LVLMs surpass VLPMs' conclusion stated in Contribution (4).","section":"§4 Comparison with SOTA; §5 Ablation study"}],"minor_comments":[{"comment":"The Rmean for the 'Detailed Prompt' setting is reported as 65.50 in Table 3 B.3 but 65.55 in Appendix Table 7 C.3; please reconcile these values or explain the rounding.","section":"Table 3 vs. Appendix Table 7"},{"comment":"The computation of q(I) and q(T) is only fully specified in Section 3; adding a brief sentence in the figure caption or a forward reference would improve readability.","section":"Figure 2(b)"},{"comment":"The caption reads 'Ablation studies on Fashion-IQ dataset with regard to different pooling strategies,' but the table compares fine-tuning strategies; please correct the caption.","section":"Appendix C, Table 4 caption"},{"comment":"The conclusion states LVLMs can 'surpass the VLMs,' while the rest of the paper uses 'VLPMs'; please use consistent terminology.","section":"Section 7 Conclusion"},{"comment":"There are several typos: 'loRA' should be 'LoRA,' 'retrival' in Figure 2, and 'Tabel' in Appendix C; a careful proofread is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's Contribution (4) is the kind of strong conceptual claim that will attract scrutiny. The current evidence is insufficient because the ablations confound scale and architecture, and the CIRR SOTA margin is tiny. This is fixable either by adding a scale-matched control or by substantially tempering the claim to an empirical demonstration that an LVLM-based encoder can achieve competitive results. The systems contribution (LoRA fine-tuning of a 7B LVLM as a CIR encoder, prompt pool design, and the efficient single-pass inference) is solid and within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Your stress-tester is right, and it's the one thing to know about this paper: the headline claim about LVLMs surpassing VLPMs is not supported by the experiments. The ablations in Table 3 (A.1–A.2) swap a 7B autoregressive LLM for a CLIP or BLIP-2 text encoder, and Appendix C says they also add an MLP and change pooling from position-weighted mean to [CLS]. So scale, pooling, and architecture all move at once. On CIRR the SOTA claim is marginal too: +0.06 Avg. over SPRC and a loss on Recallsubset@1.\n\nThat soft spot is real, but it shouldn't obscure what the paper does well. The engineering is solid: a single-pass LVLM bi-encoder for CIR is genuinely new, and the hybrid intent instruction module — task prompt plus instance-specific soft prompt pool — is a sensible combination that the ablations consistently support. The inference speed comparison is useful: 0.08s per query vs 1.38s for the generative pipeline, which is a meaningful practical advantage. The paper is also honest about its weaknesses: Appendix E discusses failure cases, and they explicitly note the CIRR Recallsubset@1 deficit.\n\nThe main missing piece is a controlled comparison at matched scale. Without that, the paper's contribution (4) — 'first to clearly show LVLMs surpass VLPMs' — is just an observation that a 7B model with instruction tuning beats encoders that are orders of magnitude smaller. That's not nothing, but it's not the claimed finding. The method itself may still be worth adopting; the prompt pool and single-pass design look like useful components for future work.\n\nNo code or data are released, and there are no error bars, so the precise numbers should be treated with some caution. But the experiments are extensive and the ablations are internally consistent.\n\nWho gets value? Researchers working on CIR or multimodal retrieval who want to see how LVLMs can be adapted as dense encoders. It deserves a serious referee: the architecture and the empirical study are worth publishing even if the strong claim needs to be rewritten. I'd recommend engaging with it, and requiring a scale-matched control or a revised claim before acceptance.","headline":"A useful single-pass LVLM encoder for composed image retrieval, but the advertised claim that LVLMs surpass VLPMs is not supported — the ablations change scale, pooling, and architecture at once.","tokens_in":22852,"tokens_out":2514,"would_cite":true,"duration_ms":22294,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CIR-LVLM turns a large vision-language model into a single-pass encoder for composed image retrieval, reporting state-of-the-art recall on Fashion-IQ, Shoes, and CIRR.","keywords":["composed image retrieval","large vision-language model","user intent","prompt pool","hybrid intent instruction","contrastive learning","Fashion-IQ","CIRR"],"falsifier":"A same-scale control experiment would settle the claim: run the same architecture with a comparable-size pure text encoder or VLPM of roughly 7B parameters, and also run the LVLM without the Connector and prompt pool. If recall parity persists, the reasoning claim fails; if the LVLM only wins when the full instruction module is present, the mechanism is the prompts rather than the backbone.","tokens_in":21817,"feed_emoji":"🖼️","tokens_out":3208,"duration_ms":26681,"temperature":0.7,"pith_summary":"This paper proposes CIR-LVLM, a framework that repurposes a large vision-language model (LVLM) as a user intent-aware encoder for composed image retrieval. The paper claims that the LVLM's reasoning and instruction-following abilities let it capture what a user wants changed across a reference image and a relative caption, which earlier vision-language pretraining approaches tend to miss. A hybrid intent instruction module supplies two levels of guidance: a task prompt that clarifies what the model should do, and an instance-specific soft prompt selected from a learnable prompt pool. On the Fashion-IQ, Shoes, and CIRR benchmarks, CIR-LVLM reports state-of-the-art recall while keeping inference fast enough for practical use. The paper also claims to be the first to show that LVLMs can surpass VLPMs in multimodal retrieval tasks that require reasoning.","feed_headline":"LVLM encoder beats VLPMs in composed image retrieval","feed_subtitle":"CIR-LVLM reaches state-of-the-art recall on Fashion-IQ, Shoes, and CIRR while keeping single-pass speed.","key_machinery":"The machinery is the LVLM-as-encoder design plus a hybrid intent instruction module. A Connector with learnable query embeddings maps the reference image into a sentence-level prompt; a task prompt tells the model what to do differently for queries versus target images; and a prompt pool with paired image/text keys selects the top-K soft prompts per instance. Because causal attention in a decoder-only LLM only lets later tokens see the whole sequence, the model uses position-weighted mean pooling to aggregate token representations, and a batch-based contrastive loss aligns query and target embeddings.","core_discovery":"The central claim is that fine-tuning an LVLM such as Qwen-VL-Chat to act as a bi-encoder retriever, rather than as a caption generator, makes it a stronger composed-image-retrieval model than cross-attention early fusion, textual inversion, or multi-pass LLM caption revision. The Connector condenses the reference image into a sentence-level prompt, the LLM produces a pooled representation that respects both the image content and the relative caption, and retrieval is done by cosine similarity to target embeddings. The authors report state-of-the-art results, including 56.21 Rmean on Fashion-IQ, 63.51 Rmean on Shoes, and 53.64 R@1 with 83.76 R@5 on CIRR, and attribute the gains to the LVLM's intent-aware reasoning combined with the hybrid instruction prompts.","pith_inferences":["If the LVLM's reasoning ability is the true driver, then future improvements in LVLM backbones should translate almost directly into retrieval gains without redesigning the fusion strategy.","The prompt pool behaves like a small learned routing memory, and the same selection mechanism could be applied to other retrieval tasks with complex, multimodal or multi-step queries.","A same-scale control is needed to separate reasoning ability from model scale, since the paper's main ablations compare the LVLM against much smaller CLIP and BLIP-2 text encoders while changing both architecture and parameter count."],"forward_implications":["Replacing conventional text encoders with an LVLM encoder improves composed image retrieval accuracy on both fashion-domain and open-domain benchmarks.","Single-pass encoding sidesteps the multi-pass decoding cost of generative LVLM retrieval, giving a per-query inference time of about 0.08 seconds compared with 1.38 seconds for a generative baseline.","Instance-specific soft prompts selected from a prompt pool outperform a universal soft prompt, suggesting that per-instance guidance is a measurable source of accuracy.","The framework transfers to different LVLM backbones, with stronger backbones yielding higher retrieval performance."],"supporting_citations":[{"why":"Provides SPRC, the sentence-level prompt textual inversion baseline that CIR-LVLM outperforms and whose user-intent limitations motivate the LVLM encoder.","marker":"Bai et al. 2023b"},{"why":"Provides CASE, the early-fusion baseline whose cross-attention failure to retain visual details motivates the Connector design.","marker":"Levy et al. 2023"},{"why":"Provides CIReVL, the generative multi-pass LLM baseline used for the inference efficiency comparison.","marker":"Karthik et al. 2023"},{"why":"Provides Qwen-VL-Chat, the LVLM backbone that the model is initialized from and fine-tuned on.","marker":"Bai et al. 2023a"},{"why":"Supplies the position-weighted mean pooling method that adapts decoder token outputs into a retrieval embedding.","marker":"Muennighoff 2022"},{"why":"Supplies the learnable prompt pool idea that the instance-specific soft prompt selection is based on.","marker":"Wang et al. 2022"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method that makes training the LVLM encoder memory-feasible.","marker":"Hu et al. 2021"}],"fun_headline_variants":["CIR-LVLM: intent-aware LVLM encoder beats VLPMs in composed retrieval","Fine-tuned LVLM as bi-encoder tops composed image retrieval benchmarks","Hybrid intent prompts + LVLM encoder set new SOTA on CIR tasks","LVLM encoder with hybrid prompts beats cross-attention in CIR","Qwen-VL fine-tuned as bi-encoder achieves SOTA composed retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline conclusion rests on the assumption that the LVLM's higher recall comes from its user-intent reasoning, because the ablation compares the LVLM encoder to much smaller CLIP and BLIP-2 text encoders while changing both model size and architecture.","fun_headline_variants_meta":{"raw":{"variants":["CIR-LVLM: intent-aware LVLM encoder beats VLPMs in composed retrieval","Fine-tuned LVLM as bi-encoder tops composed image retrieval benchmarks","Hybrid intent prompts + LVLM encoder set new SOTA on CIR tasks","LVLM encoder with hybrid prompts beats cross-attention in CIR","Qwen-VL fine-tuned as bi-encoder achieves SOTA composed retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3545,"prompt_tokens":973,"completion_tokens":2572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2469}},"tokens_in":589,"tokens_out":2572,"duration_ms":15852,"temperature":1.0,"reasoning_tokens":2469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:17:57.237418+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A same-scale control experiment would settle the claim: run the same architecture with a comparable-size pure text encoder or VLPM of roughly 7B parameters, and also run the LVLM without the Connector and prompt pool. If recall parity persists, the reasoning claim fails; if the LVLM only wins when the full instruction module is present, the mechanism is the prompts rather than the backbone.","supporting_citations":[],"review_version":1}