{"id":"4ffb7db9-25c3-4f57-96ff-293a4c1f3ac3","arxiv_id":"2506.01795","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"R2SM provides the first benchmark pairing modal and amodal text prompts with matching masks, letting models learn when to segment only visible parts versus complete occluded shapes.","lead":"This paper introduces R2SM, a benchmark that asks vision-language models to choose between a modal mask (visible part only) and an amodal mask (complete shape including hidden parts) based on a natural-language prompt. The authors built it by generating text queries for three existing amodal datasets and show that fine-tuning five models on it improves their measured segmentation scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's mask-type labels are confounded with explicit lexical cues (Section 4.2), so the 'intent reasoning' claim is not yet supported; the dataset may only test phrase recognition.","rationale":"The reader's weakest_assumption identified the same issue: the benchmark's validity rests on whether the text queries capture genuine user intent, but the generation pipeline makes amodal queries lexically marked by explicit occlusion cues. Reading Sections 1.3, 3.3, 4.2, and Figures 9-14 confirms this is the single most load-bearing concern. The task's novelty is the 'selective mask' decision; if that decision can be solved by keyword detection, the benchmark does not measure intent reasoning. The empirical results remain valid as showing that fine-tuning improves in-distribution performance on this dataset, but they do not validate the conceptual claim. The proposed classifier test is decisive because it isolates the mask-type decision variable without using visual information. No issue with the source code or dataset release detracts from reproducibility; the concern is about construct validity. Hence the CONDITIONAL verdict should remain unchanged.","tokens_in":11265,"tokens_out":4502,"duration_ms":49131,"concrete_test":"Train a text-only bag-of-words logistic regression on the released R2SM validation splits to predict modal vs amodal label from query text alone, without any image or mask input. If this text-only classifier exceeds roughly 95% accuracy on held-out queries and its top-weighted features are the instruction-mandated words (e.g., 'occluded', 'hidden', 'complete', 'whole', 'behind'), then the mask-type decision is solvable without visual reasoning, confirming the concern. A complementary check: take fine-tuned models' correct amodal predictions and paraphrase the prompts to remove those cue words; if mask-type predictions flip to modal, the models are cue-following rather than intent-aware.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 describes the query pipeline: for every instance marked 'is_occluded', Claude is instructed to generate an amodal query that 'explicitly mentioning that the instance is occluded and requesting the complete or inferred outline of the hidden parts' (Figures 9-14). Modal queries are generated without this instruction. The label 'amodal' therefore does not measure user intent; it is a deterministic consequence of prompt construction. The paper's own examples in Section 3.3 list the cue phrases: 'the complete form of the cat' and 'even the part hidden behind the gray cat'. Under this design, a model can pick the mask type by detecting a small lexicon ('occluded', 'hidden', 'complete', 'whole', 'behind') and ignore the image entirely. The large fine-tuning gains in Table 2 (e.g., ReLA from 30.65 to 71.78 gIoU on MUVA All) are consistent with learning this surface mapping rather than reasoning about occlusion or intent. The resource may still be useful for studying lexical conditioning and amodal completion, but the central claim that R2SM evaluates 'inferring user intent' and 'reasoning' is not established by the paper's experiments. This is a validity concern, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces R2SM (Referring and Reasoning for Selective Masks), a task and benchmark that extends text-guided segmentation by requiring a model to choose between a modal mask (visible region) and an amodal mask (complete region, including occluded parts) based solely on a natural language prompt. The dataset is built by augmenting three existing amodal segmentation datasets (COCOA-cls, D2SA, MUVA) with Claude-generated text queries, roughly half modal and half amodal, each paired with the corresponding ground-truth mask. The authors fine-tune five representative vision-language models (LISA, PSALM, ReLA, GLaMM, PixelLM) on each split and report consistent gIoU/cIoU improvements over the un-fine-tuned baselines. They also analyze a metric mismatch between fused gIoU/cIoU and instance-level AP/AR, and provide qualitative evidence of remaining challenges in modal/amodal boundary selection and occlusion reasoning.","tokens_in":11505,"tokens_out":4937,"duration_ms":52462,"significance":"If the central claim were fully established, R2SM would be a useful new benchmark for intent-aware, occlusion-sensitive segmentation, and the observation that gIoU/cIoU can mask degraded per-instance predictions is a valuable methodological point. The dataset release, the breadth of models benchmarked, and the careful reporting of fine-tuning setups are concrete strengths. However, the paper's core claim that the benchmark measures 'inferring user intent' or 'reasoning' is not yet supported by the presented evidence. The query-generation pipeline in Section 4.2 introduces a strong lexical confound: amodal queries are explicitly instructed to mention occlusion and request the complete outline, while modal queries are not. This makes mask-type selection solvable by keyword spotting, and the reported fine-tuning gains are consistent with learning that surface mapping. In addition, the absence of a control fine-tuning condition on class-labeled amodal data means the improvements in Table 2 could stem from additional segmentation fine-tuning data rather than text-conditioned mask-type selection.","major_comments":[{"comment":"The query-generation pipeline deterministically ties mask type to explicit lexical cues: for every instance marked is_occluded, Claude is instructed to generate an amodal query that 'explicitly mentioning that the instance is occluded and requesting the complete or inferred outline of the hidden parts,' while modal queries are generated without that instruction. This means the label 'amodal' is a consequence of prompt construction, not of intrinsic user intent. A model could select the mask type by detecting words such as 'occluded,' 'hidden,' 'complete,' 'whole,' or 'behind' without performing any visual or commonsense reasoning about occlusion, so the task as currently formulated does not establish the 'inferring user intent' claim made in Sections 1.3 and 3.3. I would need to see a deconfounded version of the benchmark: amodal queries without explicit occlusion wording, modal queries that mention occlusion but ask for the visible part only, and an analysis of model accuracy on the subset of prompts that lack the cue lexicon. A human-validated intent split would also strengthen the claim.","section":"Section 4.2, Figures 9-14"},{"comment":"The headline result that 'fine-tuning on R2SM invariably improves performance across all models and dataset splits' is missing a critical control condition. There is no experiment that fine-tunes the same models on an equal number of text queries generated from the same underlying amodal annotations but without the modal/amodal text conditioning, for example using class-label prompts or generic template prompts. Without such a control, the large gIoU gains in Table 2 (e.g., ReLA from 30.65 to 71.78 on MUVA All) could reflect the benefit of additional segmentation fine-tuning data rather than learning to condition mask type on language. Please add control experiments, or explicitly weaken the claim to state that fine-tuning with additional paired text-mask data improves segmentation quality on these splits.","section":"Section 5.3, Table 2"},{"comment":"The text states that 'although gIoU and cIoU reflect improved performance after fine-tuning when the evaluation is restricted to amodal queries, AP and AR exhibit a decline under the same condition.' This is not what Table 3 shows: the table compares the All and Amodal rows for the R2SM-fine-tuned PSALM only, and no baseline AP/AR values are reported. The supported claim is that gIoU/cIoU and AP/AR rank these two evaluation subsets differently, not that fine-tuning causes an AP/AR decline. Additionally, the AP/AR protocol matches predictions to ground-truth masks by query ID, but the handling of queries that refer to multiple instances is not described; please clarify the matching procedure and, if feasible, report baseline AP/AR values for a proper before/after comparison.","section":"Section 6.3, Table 3"},{"comment":"The training and evaluation queries are both generated by the same Claude pipeline with the same instruction templates, so the benchmark may measure a model's fit to a particular LLM prompt style rather than general user intent. No human evaluation, distribution-shift test, or cross-generation generalization experiment is reported. I would like to see either a small human study that validates the naturalness and intent clarity of the generated queries, or an out-of-distribution evaluation using prompts written by humans or generated with a different instruction set, to support the claim that the benchmark measures intent understanding rather than style overfitting.","section":"Section 4.2 and Section 5.3"}],"minor_comments":[{"comment":"The abstract says 'whether to generate amodal(visible) or amodal(complete) segmentation mask'; the first occurrence should be 'modal (visible)'.","section":"Abstract"},{"comment":"The D2SA split has 3,600 validation images but only 2,000 training images; please clarify whether this follows the original dataset's split or a deliberate choice, since an inverted train/val ratio is unusual.","section":"Section 4.3, Table 1"},{"comment":"The caption refers to the 'COCOA split' while the paper consistently uses 'COCOA-cls'; please unify the nomenclature.","section":"Figure 6 caption"},{"comment":"The '-' entries for APs and APm should be explained (for example, 'no sufficiently small objects exist in this split'), and the object-size thresholds should be defined in a footnote.","section":"Table 3"},{"comment":"The training setup says 'one epoch corresponds to a full pass through all queries in the respective split,' but the Appendix also mentions that PSALM's best checkpoint is epoch 10 for COCOA-cls and epoch 9 for D2SA/MUVA; please clarify whether the epoch count is measured over queries or over images, and whether hyperparameters were tuned per model or fixed from the official repositories.","section":"Section 5.2"},{"comment":"The text says each image is annotated with 'up to 10 diverse text queries,' but no distribution of queries per image is reported; please add this statistic.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful resource and a clear evaluation effort, but the central 'intent reasoning' claim is currently not supported by the experimental design because of the lexical confound in query generation and the missing control fine-tuning condition. I do not recommend rejection, as these issues are addressable within the manuscript's scope: the authors could add deconfounding experiments, control conditions, or a more careful restatement of the claims. Given the current evidence, I would not accept the paper in its present form."},"author_rebuttal":null,"desk_editor":null,"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces R2SM, a benchmark that pairs modal and amodal text queries with masks, and claims that fine-tuning on it lets vision-language models learn to choose the mask type a prompt asks for.","keywords":["referring segmentation","amodal segmentation","reasoning segmentation","vision-language models","user intent","occlusion reasoning","benchmark dataset","mask type selection"],"falsifier":"An experiment would be to evaluate a fine-tuned model on paraphrased amodal queries that ask for the complete object without using any explicit occlusion keyword; if performance drops to the level of the unfine-tuned baseline, the dataset has taught keyword detection rather than intent-based mask selection.","tokens_in":11072,"feed_emoji":"🖼️","tokens_out":6446,"duration_ms":60659,"temperature":0.7,"pith_summary":"This paper introduces a new task, Referring and Reasoning for Selective Masks (R2SM), in which a vision-language model reads a natural-language prompt and must decide whether to output a modal mask (only the visible part of an object) or an amodal mask (the full shape including occluded regions). To support the task, the authors build the R2SM dataset by attaching language-model-generated text queries to images from COCOA-cls, D2SA, and MUVA, pairing roughly half of the prompts with modal masks and half with amodal masks. The paper's central empirical claim is that fine-tuning five representative segmentation models on R2SM improves performance on every model and dataset split, with the largest gain being ReLA rising from 30.65 to 83.18 gIoU on D2SA. A sympathetic reader would care because the task separates prompt-driven intent from fixed output behavior, making occlusion reasoning explicitly testable. The paper also shows that fused-mask metrics can hide weak per-instance predictions, recommending AP/AR for multi-instance queries.","feed_headline":"Fine-tuning on R2SM lifts gIoU on every model and split","feed_subtitle":"Five vision-language models improve at choosing visible vs complete masks; the benchmark makes intent-aware segmentation testable.","key_machinery":"The central object is the R2SM dataset, composed of roughly 74,000 amodal and 73,000 modal text queries paired with masks across three source datasets. The mechanism that carries the argument is the query-generation pipeline: structured instance annotations, including an occlusion flag, are fed to a large language model that is instructed to produce diverse referring prompts and, for each occluded instance, an amodal variant that explicitly mentions the occlusion and asks for the complete or inferred shape. That explicit pairing of cue to mask type is what makes it possible to fine-tune a model on the mask-selection decision.","core_discovery":"On the paper's own terms, the discovery is that a dataset pairing modal and amodal text queries with matching ground-truth masks lets existing vision-language models learn to select mask type from language, and that this learning transfers across real and synthetic occlusion scenes. The strongest evidence is Table 2, where fine-tuning on R2SM improves gIoU and cIoU for LISA, PSALM, ReLA, GLaMM, and PixelLM on every split; for example, ReLA goes from 30.65 to 83.18 gIoU on D2SA and from 24.88 to 71.78 on MUVA. The authors argue no prior benchmark offered this paired supervision, so models were limited to fixed mask types. They also report that although fused gIoU/cIoU rise, per-instance mask quality can decline under amodal fine-tuning, which they interpret as evidence that instance-level metrics are needed.","pith_inferences":["My editorial inference: the generation pipeline's instruction to mention occlusion explicitly means the amodal queries may be recognizable by keywords such as 'occluded' or 'hidden behind'; testing with paraphrased prompts that express the same intent without those words would show whether models learn true intent or phrase matching.","My editorial inference: the same text-query generation recipe could be applied to other amodal datasets such as KINS, WALT, or MP3D-Amodal, which would test whether the reported gains generalize beyond the three chosen source datasets.","My editorial inference: the observed trade-off between sharper modal boundaries and coarser amodal predictions suggests a promising next step in which a model predicts both masks and lets the prompt, or an uncertainty signal, arbitrate between them."],"forward_implications":["Fine-tuning on R2SM raises both gIoU and cIoU for every benchmarked model and every split, so mask-type selection appears learnable from paired supervision rather than requiring a new architecture.","The benchmark's All and Amodal evaluation settings allow the mask-selection decision and the amodal-completion skill to be measured separately.","Larger gains on D2SA and MUVA than on COCOA-cls suggest that the semi-synthetic and synthetic splits offer denser or more consistent occlusion annotations for learning amodal shape completion.","Because cIoU and gIoU diverge across splits, conclusions about model quality depend on which metric is used; reporting both is necessary.","The AP/AR analysis indicates that future work on this task should evaluate per-instance predictions, not only fused masks."],"supporting_citations":[{"why":"Provides the COCOA-cls and D2SA images and instance-level modal/amodal masks that form two of the three R2SM splits.","marker":"[2]"},{"why":"Provides the MUVA multi-view shopping-scene images and amodal annotations that form the third split.","marker":"[3]"},{"why":"Supplies the text-query generation methodology that R2SM adapts for producing diverse referring prompts.","marker":"[11]"},{"why":"LISA is one of the five baselines and represents the embedding-as-mask reasoning-segmentation approach.","marker":"[5]"},{"why":"PSALM is a pixelwise multi-modal baseline whose AP/AR decline under amodal fine-tuning motivates the instance-level metric discussion.","marker":"[6]"},{"why":"ReLA/GRES is the best-performing model on D2SA and MUVA after fine-tuning; its region-based probability estimates anchor the main quantitative gains.","marker":"[10]"},{"why":"GLaMM is the best-performing baseline on COCOA-cls after fine-tuning, providing the grounded-conversation reference point.","marker":"[13]"},{"why":"PixelLM is one of the five baselines and supplies the multi-target pixel-reasoning comparison.","marker":"[12]"},{"why":"AURA defines the prior state of amodal reasoning segmentation and motivates why models need mask-type selection rather than a fixed amodal output.","marker":"[1]"}],"fun_headline_variants":["Fine-tuning on R2SM lifts gIoU across all models","R2SM task: pick visible or complete mask from language","New dataset R2SM pairs prompts with modal and amodal masks","R2SM: five VLMs learn intent-aware segmentation from text","R2SM fine-tuning: big gIoU gains, but per-instance risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each generated text query's intended mask type is faithfully captured by the prompt template, so the benchmark measures real user intent rather than a model's ability to spot formulaic phrases such as 'occluded' or 'hidden behind'.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning on R2SM lifts gIoU across all models","R2SM task: pick visible or complete mask from language","New dataset R2SM pairs prompts with modal and amodal masks","R2SM: five VLMs learn intent-aware segmentation from text","R2SM fine-tuning: big gIoU gains, but per-instance risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2942,"prompt_tokens":960,"completion_tokens":1982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1888}},"tokens_in":576,"tokens_out":1982,"duration_ms":14781,"temperature":1.0,"reasoning_tokens":1888,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:33:06.033760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment would be to evaluate a fine-tuned model on paraphrased amodal queries that ask for the complete object without using any explicit occlusion keyword; if performance drops to the level of the unfine-tuned baseline, the dataset has taught keyword detection rather than intent-based mask selection.","supporting_citations":[{"cited_title":"Learning to see the invisible: End-to-end trainable amodal instance segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the COCOA-cls and D2SA images and instance-level modal/amodal masks that form two of the three R2SM splits."},{"cited_title":"MUV A: A new large-scale benchmark for multi-view amodal instance segmentation in the shopping scenario","cited_arxiv_id":null,"evidence_quote":"Provides the MUVA multi-view shopping-scene images and amodal annotations that form the third split."},{"cited_title":"LISA: reasoning segmentation via large language model","cited_arxiv_id":null,"evidence_quote":"LISA is one of the five baselines and represents the embedding-as-mask reasoning-segmentation approach."},{"cited_title":"PSALM: pixelwise segmentation with large multi-modal model","cited_arxiv_id":null,"evidence_quote":"PSALM is a pixelwise multi-modal baseline whose AP/AR decline under amodal fine-tuning motivates the instance-level metric discussion."},{"cited_title":"GRES: generalized referring expression segmen- tation","cited_arxiv_id":null,"evidence_quote":"ReLA/GRES is the best-performing model on D2SA and MUVA after fine-tuning; its region-based probability estimates anchor the main quantitative gains."},{"cited_title":"Shaker, Salman H","cited_arxiv_id":null,"evidence_quote":"GLaMM is the best-performing baseline on COCOA-cls after fine-tuning, providing the grounded-conversation reference point."},{"cited_title":"Pixellm: Pixel reasoning with large multimodal model","cited_arxiv_id":null,"evidence_quote":"PixelLM is one of the five baselines and supplies the multi-target pixel-reasoning comparison."},{"cited_title":"Unveiling the Invisible: Reasoning Complex Occlusions Amodally with AURA","cited_arxiv_id":"2503.10225","evidence_quote":"AURA defines the prior state of amodal reasoning segmentation and motivates why models need mask-type selection rather than a fixed amodal output."}],"review_version":1}