{"id":"73048153-8ac2-456a-bba2-ff21a853c313","arxiv_id":"2504.14467","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LGD generates attribute and surrounding descriptions from a multi-modal LLM to improve CLIP-based mask selection, setting new state-of-the-art results on RefCOCO, RefCOCO+, and RefCOCOg.","lead":"This paper introduces LGD, a zero-shot method that uses a multimodal language model to generate attribute and surrounding descriptions of objects, then matches CLIP features between these descriptions and image masks to segment the object named by a referring expression. The authors report large gains over prior zero-shot referring segmentation systems on RefCOCO, RefCOCO+, and RefCOCOg.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unmeasured oracle ceiling on proposal sets: reported oIoU/mIoU are bounded by SAM/DINO recall, but no recall or oracle analysis is reported, so the central SOTA claim is not yet fully supported.","rationale":"The reader's weakest assumption identifies proposal-set recall as the critical dependency of Eq. 11, and my read agrees: the method is a re-ranking procedure over a fixed proposal set, so the reported numbers cannot exceed the oracle ceiling of the proposal generator. The paper contains no oracle experiment, no recall curve, and no analysis of failure cases where the referent is absent from M, even though that is the exact regime where the proposed score combination cannot help. Because this concern is already reflected in the reader's CONDITIONAL verdict, my stress-test does not change the verdict. I would add one concrete condition to the revision: report the oracle ceiling and failure decomposition, or the SOTA claim remains qualified by an unverified upper bound. I do not see an internally inconsistent step or a fraudulent claim; the concern is about missing support for the mechanism's contribution, not about the validity of the equations as written.","tokens_in":14725,"tokens_out":4019,"duration_ms":37966,"concrete_test":"Using the same SAM (and DINO+SAM) proposal generators and CLIP/ViT-B/32, on RefCOCO val compute: (1) proposal recall at IoU ≥ 0.5 for the GT referent mask; (2) oracle mIoU/oIoU if the argmax were replaced by the proposal with highest GT IoU; (3) LGD mIoU on only the subset of images where at least one proposal has IoU ≥ 0.5 with GT. If LGD is close to the oracle ceiling, the unmeasured recall bound is confirmed as the bottleneck; if LGD falls well below oracle on covered images, then the scoring/description mechanism is the limiting factor and the claim can be evaluated on its own terms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 restricts the output to an off-the-shelf proposal set M ('a set of segmentation proposals m ∈ M'), and Eq. 11 is an argmax over that set: final mask M = arg max_m S(m). Therefore every reported oIoU/mIoU is bounded above by the oracle best-proposal performance of SAM (or DINO+SAM). The paper reports large gains over TAS/BMS without ever measuring proposal recall, the oracle ceiling, or the fraction of failures caused by the referent being absent from M. This matters because the three CLIP scores and MLLM descriptions can only re-rank proposals; they cannot recover a missed referent. If proposal recall is high, the argument that generative descriptions drive the gain is plausible; if recall is low on the failure cases, the headline numbers are dominated by proposal-generator quality, not by the claimed mechanism. The absence of this analysis leaves the central SOTA claim under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LGD, a training-free zero-shot referring image segmentation framework. Given an image and a referring expression, LGD uses an MLLM (LLaVA-v1.6-Vicuna-7B) with two hand-designed prompts to generate an attribute description of the referent and a surrounding description of contextual objects. CLIP text embeddings of these descriptions and of the original expression are compared with CLIP image embeddings of mask proposals produced by SAM (optionally with Grounding DINO). Three cosine-similarity scores, S_att, S_sur, and S_van, are combined linearly as S = S_van + αS_att + βS_sur, and the proposal with the highest score is output. Experiments on RefCOCO, RefCOCO+, and RefCOCOg report oIoU and mIoU improvements over prior zero-shot RIS methods, including TAS and BMS, with the largest gains when Grounding DINO is added.","tokens_in":15000,"tokens_out":4499,"duration_ms":39585,"significance":"If the reported results are robust and properly validated, the paper makes a useful contribution: it demonstrates that MLLM-generated attribute and surrounding descriptions can improve CLIP-based region-text matching for zero-shot referring image segmentation without any training. The ablation study shows that each of the three scores contributes, and the use of surrounding-object noun phrases as negative samples is a sensible extension of the TAS and BMS lines of work. The main limitations are that the headline numbers rely on per-dataset hyperparameter tuning, the method is bounded by the recall of the off-the-shelf proposal generator, and no statistical validation is provided; these issues currently weaken the strength of the state-of-the-art claim.","major_comments":[{"comment":"The hyperparameters α and β are tuned per benchmark: β=1 for all datasets, α=0.5 for RefCOCO and RefCOCO+, and α=0.3 for RefCOCOg, selected via the sensitivity analyses in Figs. 3–5. Because the reported SOTA numbers use these per-dataset values, the evaluation is not fully a-priori and may give LGD an advantage over baselines that did not receive the same tuning. Please state explicitly whether α was chosen using only validation splits, provide results with a single fixed α across all datasets, or otherwise justify the per-dataset tuning protocol.","section":"Section 4.1, Eq. (10)"},{"comment":"The final mask is always selected by argmax over the proposal set M produced by SAM or DINO+SAM, so the reported oIoU and mIoU are upper-bounded by the recall of that proposal generator. The paper does not report proposal recall, the oracle ceiling (best IoU achievable among proposals), or any analysis of failure cases where the referent is absent from M. Without this information, the claimed improvements over TAS and BMS cannot be fully attributed to the proposed scoring mechanism, because they may partly reflect the quality of the proposal set. Please add a recall/oracle analysis and discuss failure cases.","section":"Section 3.2, Eq. (11)"},{"comment":"The procedure for selecting negative noun phrases from the surrounding description is underspecified: the paper says CLIP is used to identify phrases 'semantically irrelevant to the referent object' but gives no threshold, criterion, or constraint on the number of negative samples. This is a free parameter that directly affects S_sur and the final result. Please specify the selection rule, report sensitivity to it, and state how the case |Tsur|=0 is handled, since Eq. (8) divides by |Tsur|.","section":"Section 3.2, Eq. (8)"},{"comment":"No error bars, confidence intervals, or multiple-run statistics are reported. Several claimed improvements are small (e.g., 0.69% mIoU on RefCOCOg versus TAS), and the MLLM generation process may be stochastic, so single-run numbers are not sufficient to establish SOTA-level claims. Please report means and standard deviations over at least a few runs, or otherwise clarify that the MLLM inference is deterministic and the reported numbers are reproducible.","section":"Table 1"}],"minor_comments":[{"comment":"The heading 'Sensitive toward α and β' should be 'Sensitivity to α and β'.","section":"Section 4.3"},{"comment":"There is a typo in 'α andβ represent adaptive hyberparameters' — 'hyberparameters' should be 'hyperparameters'.","section":"Section 3.4"},{"comment":"The comparison of different MLLMs is presented only as a figure; please also provide the numerical values in a table or in the caption for reproducibility.","section":"Figure 6"},{"comment":"In the prompt templates, placeholders like '<input>', '<object>', '(attribute)', and '(entities)' should be clearly marked as placeholders rather than literal strings.","section":"Section 3.1"},{"comment":"The notation Tsur is used both for the full surrounding description and, after extraction, for the set of negative noun phrases; please use distinct notation to avoid confusion.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest but potentially useful extension of TAS and BMS. The main risks are that the headline numbers depend on per-dataset tuning of α and on the recall of the proposal generator, and that the evaluation lacks statistical validation. These issues are fixable with additional experiments and reporting, so I recommend major revision rather than rejection. The manuscript is within the scope of Pattern Recognition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LGD is a sensible, incremental combination of known pieces: have an MLLM write two descriptions of the referent (attributes, surrounding objects), then re-rank SAM or DINO+SAM mask proposals with three CLIP scores. The two-prompt decomposition and the negative-noun-phrase score are genuinely new relative to TAS and BMS, and the ablations show each score adds something. The qualitative examples are also convincing in the easy cases. Credit where due: the pipeline is coherent and the paper is easy to follow.\n\nThe soft spots are real but not fatal. First, alpha is tuned on each benchmark (0.5 for RefCOCO/RefCOCO+, 0.3 for RefCOCOg) and the sensitivity analysis is run on the same test splits. That makes the reported improvements partly a product of per-dataset calibration. Beta is fixed at 1 everywhere, which is good, but alpha still matters. Second, the negative-sample selection rule is underspecified: the paper says Spacy extracts noun phrases and CLIP identifies irrelevant ones, but never states the relevance threshold or how many negatives survive. Third, there are no error bars, and the MLLM is stochastic, so I cannot tell whether a 1-2 point RefCOCOg gain is noise. No code is released either.\n\nThe stress-test concern about the oracle ceiling is legitimate. Every reported number is an argmax over a fixed proposal set, so the method cannot recover a referent that SAM or DINO+SAM never proposes. The paper never measures proposal recall or the oracle-best-mask upper bound. That matters: if the proposal set already contains the right mask most of the time, the three scores are just re-ranking; if it misses the referent often, the headline gains depend more on the proposal generator than on the generative descriptions. I would not call this a load-bearing flaw, but it is exactly the analysis a referee should demand before accepting the SOTA claim.\n\nWho is this for? Researchers working on zero-shot referring segmentation and CLIP-based region-text matching. They will find the two-prompt idea worth testing even if the exact numbers do not survive reimplementation.\n\nMy recommendation: send it to peer review. The core approach is coherent and the ablations are informative, but the revision should be asked to report oracle/proposal-recall numbers, fix or cross-validate alpha, disclose the negative-sample selection rule, and ideally release code. With those additions, the SOTA claim would be much easier to evaluate.","headline":"A clean incremental pipeline that likely works, but the SOTA claim is under-supported because the two fusion weights are tuned per benchmark and no one measures the proposal-recall ceiling.","tokens_in":15465,"tokens_out":1180,"would_cite":false,"duration_ms":13027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot referring image segmentation improves when a multimodal language model rewrites the referring expression into attribute and surrounding descriptions, and CLIP similarity scores are linearly combined to pick the mask.","keywords":["zero-shot referring image segmentation","referring expression","multi-modal large language model","vision-language model","CLIP","SAM","visual-text matching","prompt design"],"falsifier":"On a random sample of RefCOCO/RefCOCO+/RefCOCOg images, run SAM alone and Grounding DINO + SAM and record whether any proposal overlaps the ground-truth mask by more than 0.5 IoU, then compare that set with LGD's failures. If the failures coincide with missing proposals, the score combination is not the limiting factor; if LGD fails on images where a good proposal exists, the fusion itself is the bottleneck.","tokens_in":14494,"feed_emoji":"🖼️","tokens_out":6727,"duration_ms":53397,"temperature":0.7,"pith_summary":"The paper tries to show that zero-shot referring image segmentation (locating and masking an object from a free-form phrase without any training) can be made more accurate by enriching the phrase before matching. Instead of matching the raw referring expression directly to candidate masks, LGD asks a multi-modal large language model to generate two extra texts: an attribute description of the referent and a surrounding description naming nearby distractors. These three texts are encoded with CLIP and turned into three cosine-similarity scores, and the mask with the highest weighted sum wins. If the claim holds, the method reports gains on RefCOCO, RefCOCO+, and RefCOCOg over prior zero-shot techniques with no training and only two tuned weights.","feed_headline":"MLLM rewrites referring expressions to lift zero-shot segmentation","feed_subtitle":"Adding attribute and surrounding descriptions to CLIP scoring pushes RefCOCO gains up to 9.97% oIoU and 11.29% mIoU.","key_machinery":"The load-bearing machinery is the pair of hand-designed prompts: the attribute prompt asks the MLLM to output a caption of the form “A photo of <object> (attribute)”, and the surrounding prompt asks for “A photo of <object> surrounded by (entities)”. Noun phrases from the surrounding description are extracted with a dependency parser and treated as negative samples, making $S_{\\mathrm{sur}}$ a penalty for looking like a distractor. The three CLIP scores are fused with fixed weights, $\\alpha=0.5$ on RefCOCO and RefCOCO+ and $0.3$ on RefCOCOg, with $\\beta=1$ everywhere, and the maximum of the resulting sum selects the mask. Every component except the two weights is frozen, so the method is a pure inference-time combination of an MLLM, CLIP, and a mask proposal network.","core_discovery":"LGD's central claim is that a linear score combination $S = S_{\\mathrm{van}} + \\alpha S_{\\mathrm{att}} + \\beta S_{\\mathrm{sur}}$ followed by $M = \\arg\\max_m S(m)$ is enough to beat existing zero-shot referring image segmentation methods on all three standard benchmarks. $S_{\\mathrm{van}}$ is the CLIP cosine similarity between the instance-level visual feature and the vanilla referring expression, $S_{\\mathrm{att}}$ is the similarity to a generated attribute description, and $S_{\\mathrm{sur}}$ is the negative mean similarity to noun phrases of surrounding objects. The paper reports maximum improvements of 9.97% in oIoU and 11.29% in mIoU over the previous BMS baseline, with consistent improvements over TAS as well. The ablation shows that the full three-score combination performs best, while adding only $S_{\\mathrm{att}}$ can hurt on RefCOCOg because its referring expressions already carry the key attributes.","pith_inferences":["My inference: the paper's reported ceiling is set by proposal recall, and LGD inherits every failure of SAM or Grounding DINO + SAM; a recall-vs-failure analysis would reveal how much headroom remains.","My inference: the dataset-dependent optimal $\\alpha$ suggests an adaptive scheme where the model estimates how informative the raw expression already is and down-weights generated descriptions on expressions that are already detailed.","My inference: the same description-enrichment step should transfer to CLIP-based referring expression comprehension (boxes instead of masks) and to other frozen vision-language encoders, which the paper does not test."],"forward_implications":["Reported gains appear on every split of all three datasets, with the largest margins on RefCOCO and RefCOCO+, where referring expressions are shortest and least informative.","The ablation implies both generated terms are needed: $S_{\\mathrm{sur}}$ helps on every dataset, while $S_{\\mathrm{att}}$ alone lowers RefCOCOg performance.","Using an open-set detector before SAM changes the proposal set and produces the largest improvements over BMS, so proposal quality and score quality compound.","Because nothing is trained, the same pipeline can be pointed at any new image and phrase at inference time, limited only by the frozen components."],"supporting_citations":[{"why":"CLIP supplies the frozen visual and text encoders used for all three matching scores.","marker":"[20]"},{"why":"SAM supplies the off-the-shelf mask proposals from which the final mask is selected.","marker":"[21]"},{"why":"Global-Local CLIP is the first zero-shot RIS baseline and contributes the M&C masked-image strategy that LGD builds on.","marker":"[23]"},{"why":"TAS is the main non-DINO comparison and contributes the M&B mask strategy and the use of surrounding noun phrases as negatives.","marker":"[24]"},{"why":"BMS is the strongest comparison that LGD+DINO improves on and contributes the image feature extraction pipeline.","marker":"[25]"},{"why":"LLaVA is the default multi-modal LLM that generates the attribute and surrounding descriptions.","marker":"[31]"},{"why":"Grounding DINO is the open-set detector used in the LGD+DINO variant to produce box prompts for SAM.","marker":"[52]"},{"why":"SpaCy is the dependency parser used to extract noun phrases from the surrounding description as negative samples.","marker":"[44]"},{"why":"LLaVA-v1.6-Vicuna-7B is the specific MLLM weights reported in the main experiments.","marker":"[53]"}],"fun_headline_variants":["MLLM-generated descriptions lift zero-shot segmentation","Attribute and surrounding prompts refine region-text matching","Generative descriptions add up to 9.97% oIoU in zero-shot segmentation","LGD uses dual descriptions to improve segmentation matching","Two description types sharpen zero-shot referring segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method presupposes that a good mask for the referent object is already present in the candidate set produced by the mask generator; if SAM or Grounding DINO + SAM never proposes a mask covering the target, no combination of scores can select it, so LGD's accuracy is capped by the recall of that proposal generator.","fun_headline_variants_meta":{"raw":{"variants":["MLLM-generated descriptions lift zero-shot segmentation","Attribute and surrounding prompts refine region-text matching","Generative descriptions add up to 9.97% oIoU in zero-shot segmentation","LGD uses dual descriptions to improve segmentation matching","Two description types sharpen zero-shot referring segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001646,"raw_usage":{"total_tokens":6568,"prompt_tokens":1005,"completion_tokens":5563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":5495}},"tokens_in":621,"tokens_out":5563,"duration_ms":36469,"temperature":1.0,"reasoning_tokens":5495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:47:34.488791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a random sample of RefCOCO/RefCOCO+/RefCOCOg images, run SAM alone and Grounding DINO + SAM and record whether any proposal overlaps the ground-truth mask by more than 0.5 IoU, then compare that set with LGD's failures. If the failures coincide with missing proposals, the score combination is not the limiting factor; if LGD fails on images where a good proposal exists, the fusion itself is the bottleneck.","supporting_citations":[{"cited_title":"Kirillov, E","cited_arxiv_id":null,"evidence_quote":"SAM supplies the off-the-shelf mask proposals from which the final mask is selected."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Global-Local CLIP is the first zero-shot RIS baseline and contributes the M&C masked-image strategy that LGD builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TAS is the main non-DINO comparison and contributes the M&B mask strategy and the use of surrounding noun phrases as negatives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BMS is the strongest comparison that LGD+DINO improves on and contributes the image feature extraction pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA is the default multi-modal LLM that generates the attribute and surrounding descriptions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounding DINO is the open-set detector used in the LGD+DINO variant to produce box prompts for SAM."},{"cited_title":"Honnibal, M","cited_arxiv_id":null,"evidence_quote":"SpaCy is the dependency parser used to extract noun phrases from the surrounding description as negative samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA-v1.6-Vicuna-7B is the specific MLLM weights reported in the main experiments."}],"review_version":1}