{"id":"dae23969-dc8b-402c-af50-bc4933d97f80","arxiv_id":"2412.14006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single 3B-parameter end-to-end model with object-aware video perceiving and multi-granularity text fusion reports SOTA results across four instructed visual segmentation tasks.","lead":"InstructSeg describes a single multimodal model that performs referring and reasoning segmentation on both images and videos, unifying four tasks under one end-to-end pipeline. The authors report state-of-the-art scores on several benchmarks using a compact 3B-parameter backbone, which could simplify deployment of language-guided segmentation systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA attribution is confounded by base MLLM choice: no same-backbone comparison isolates OVP/VMTF, so the large gains over LISA/VISA/PSALM may reflect Mipha-3B rather than the proposed architecture.","rationale":"Agreeing with the reader, I see the base-MLLM confound as the single load-bearing issue for the contribution claim. The empirical result may well be correct, and the paper's internal ablations (Tables 6-9) show the modules and multi-task training help under a fixed base. But those ablations are within InstructSeg; they do not establish that a 3B Mipha-Phi-2 backbone is not the dominant cause of the large margins over prior 7B/13B MLLM baselines. This matters because the paper frames the SOTA as evidence for its architectural designs ('demonstrating the effectiveness of our simplified pipeline'). The same-backbone control is standard and inexpensive once code is released; until then, a conditional verdict is appropriate. I would not reject: the results are internally consistent, the modules have plausible mechanisms, and the data-recipe ablations partially support the story. I would not accept unconditionally because the central attribution is unproven and no error bars or exact splits are given.","tokens_in":15817,"tokens_out":10798,"duration_ms":98247,"concrete_test":"Release the training recipe and run InstructSeg with LLaVA-1.5-7B and Chat-UniVi-7B as the base LLM, keeping OVP, VMTF, decoder, data splits, and optimization identical, then re-evaluate on RefCOCO+ val and ReVOS reasoning. If the margin over LISA/VISA collapses to the ablation-scale ~2 points, the architecture is not the source of the headline SOTA; if it persists, the base-LLM confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim (Section 4.1: 'InstructSeg surpasses all the previous works...') concerns the complete system, so it is not false merely because the base LLM differs. But the accompanying claim that this 'demonstrates the effectiveness of our simplified pipeline' is underdetermined. InstructSeg uses Mipha-3B (Phi-2.7B; Section 3.1), while the strongest prior MLLM baselines in Tables 2-4 use LLaVA-7B, LLaVA-13B, or Chat-UniVi-7B/13B. Mipha is not an implementation detail: it is an instruction-tuned MLLM with substantially stronger small-model VQA/reasoning performance, so it can account for much of the +7.2 cIoU on RefCOCO+, +9.0 gIoU on ReasonSeg, and +11.0 J&F on ReVOS reasoning. The only same-backbone evidence is Table 6, where OVP+VMTF add 2.5 cIoU on RefCOCO val and 1.6 J&F on ReVOS overall, much smaller than the headline margins. Without swapping Mipha into LISA/VISA (or LLaVA/Chat-UniVi into InstructSeg), the SOTA gains cannot be attributed to the proposed modules.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper defines Instructed Visual Segmentation (IVS) as the union of referring and reasoning segmentation at both image and video levels, and proposes InstructSeg, an end-to-end segmentation pipeline built on a lightweight MLLM (Mipha-3B). The method combines an Object-aware Video Perceiver (OVP) for temporal and object information, a Vision-guided Multi-granularity Text Fusion (VMTF) module, and a Mask2Former-style decoder. The model is trained jointly on RefCOCO/+/g, ReasonSeg, Ref-YouTube-VOS, and ReVOS, and reports state-of-the-art results on these benchmarks, along with competitive VQA results. The paper also includes ablations of the proposed components, reference-frame count, text fusion strategy, and training-data recipes.","tokens_in":16164,"tokens_out":6808,"duration_ms":57312,"significance":"If the reported results are reliable, InstructSeg provides a strong simplified alternative to multi-specialist pipelines such as VISA: a single 3B-scale MLLM with frozen encoders, LoRA fine-tuning, and a single decoder handles four text-guided segmentation tasks with state-of-the-art performance. The release of code is a strength for reproducibility. However, the significance is tempered by two main issues: the SOTA claims are confounded by the choice of base MLLM (Mipha-3B/Phi-2 versus LLaVA/Vicuna/Chat-UniVi in the baselines), and the proposed modules contribute only modest gains in the ablations compared with the large headline margins. The absence of error bars further weakens the support for the claimed improvements. With additional same-backbone comparisons and uncertainty quantification, the contribution could become solid.","major_comments":[{"comment":"The central SOTA claim is confounded by the base MLLM choice. InstructSeg uses Mipha-3B (Phi-2, Section 3.1) while the strongest MLLM-based baselines (LISA, VISA, PSALM, TrackGPT) use LLaVA-7B/13B, Vicuna, or Chat-UniVi. The ablation in Table 6 shows that OVP+VMTF add only +2.5 cIoU on RefCOCO val and +1.6 J&F on ReVOS overall, whereas the headline margins over prior methods are much larger (e.g., +7.2 cIoU on RefCOCO+ val and +7.0 J&F on ReVOS overall). Without a same-backbone comparison (e.g., training the baselines with Mipha-3B, or removing OVP/VMTF from InstructSeg while keeping the same base), the gains cannot be attributed to the proposed architecture. Please provide such experiments or, if they are impossible, explicitly discuss the base-model contribution and temper the attribution claim.","section":"Section 4.1, Tables 2-4, and Table 6"},{"comment":"The final mask is derived by applying a threshold to mask scores, but the threshold value and selection procedure are not reported. If the threshold is tuned per benchmark or on the test set, the comparison with baselines could be unfair. Please specify the threshold, state whether it is fixed across all datasets, and describe how it was chosen; ideally, show sensitivity to threshold variation.","section":"Section 3.4 and evaluation protocol"},{"comment":"No error bars or multiple-seed results are reported. Several improvements are modest (e.g., +0.7 J&F on Ref-DAVIS over VISA-13B in Table 4; +1.6 J&F overall in the Table 6 ablation). Without estimates of variance, it is unclear whether these differences are significant or within run-to-run noise. Please report means and standard deviations (or comparable uncertainty measures) for the main tables and ablations.","section":"Tables 2-6"},{"comment":"The paper trains on ReVOS and evaluates on ReVOS, but does not state the train/test split. Please clarify whether the evaluation set is a held-out portion disjoint from training; if training and evaluation use the same ReVOS set, the ReVOS results would be circular. This is load-bearing because ReVOS is a cited SOTA benchmark.","section":"Section 4 (Datasets) and Table 3"}],"minor_comments":[{"comment":"There is a typo: \"Maks2Former\" should be \"Mask2Former\" (appears also in Supplementary Material A).","section":"Section 4 (Implementation details)"},{"comment":"The notation E is used both for input text tokens and for the output embeddings of the LLM in Eq. (2); please disambiguate these two uses.","section":"Section 3.2, Eqs. (1)-(2)"},{"comment":"Table 1 shows that VISA already covers all four tasks of IVS. The novelty of this work should be positioned as the simplified end-to-end pipeline rather than the task union itself, otherwise the contribution statement overclaims.","section":"Section 1 and Table 1"},{"comment":"The sentence \"surpasses all the previous works\" is made per benchmark, but the set of baselines differs across Tables 2-4. Please qualify the SOTA claim accordingly (e.g., \"state-of-the-art on each benchmark\" rather than an unqualified global claim).","section":"Section 4.1"},{"comment":"Several cited methods are arXiv preprints at the time of writing (e.g., [41], [42], [47]). Please add official versions or venues if they have been published, as this helps the reader assess maturity of the baselines.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The self-citations to LaSagnA and HyperSeg are the authors' own prior work and are used as baselines; this is acceptable but should be disclosed in the submission process. The stress-test concern about base-MLLM confounding is valid and is the main reason for the major-revision recommendation. If the authors can supply same-backbone comparisons (even for a subset of tables) or clearly delimit the claim, the paper could become publishable. The code release is a positive sign but cannot be verified from the manuscript alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is that one end-to-end model with a 3B Phi-2 backbone handles four instructed segmentation tasks without an external tracker, and the internal ablations hold together. The OVP and VMTF modules each contribute measurable gains, and Table 9 shows sensible cross-task generalization. The writing is clear and the experimental setup is standard. That part earns credit.\n\nThe soft spot is exactly what the stress-test flags. Every prior MLLM baseline in Tables 2-4 uses LLaVA, Chat-UniVi, or Vicuna; InstructSeg uses Mipha-3B, which is known to be much stronger than LLaVA-7B on small-model VQA. The paper claims the SOTA margins 'demonstrate the effectiveness of our simplified pipeline,' but Table 6 shows the proposed modules add only 2.5 cIoU on RefCOCO and 1.6 J&F on ReVOS. The remaining 5-10 point gaps are unaccounted for. No same-backbone comparison, no error bars. That is a load-bearing under-determination of the central claim, though not a fatal flaw: the system itself works and the task unification is useful.\n\nMinor issues: the abstract promises code at a URL, the supplementary says it will be released 'in the near future' - one of those is stale. Also, the reference list contains a stray '1' in several entries [7], [53], [27], and the citation for GroundHog in Table 2 points to a spectrum-guided R-VOS paper [31], which looks wrong.\n\nThe stress-test concern holds up. It does not make the paper unserious - the authors just need to swap the base LLM into one prior baseline, or tone down the component-attribution language. I would send it to review; a good referee can demand that experiment or a rewritten claim. This is a solid systems paper that people building MLLM-based segmentation models will want to read.","headline":"The unified end-to-end system is real and the ablations hold together, but the headline SOTA margins are mostly unexplained because the base MLLM differs from every prior MLLM baseline.","tokens_in":16637,"tokens_out":1678,"would_cite":true,"duration_ms":15730,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstructSeg unifies referring and reasoning segmentation in images and videos into one MLLM pipeline and reports state-of-the-art results on every benchmark it tests.","keywords":["instructed visual segmentation","referring expression segmentation","reasoning segmentation","referring video object segmentation","multi-modal large language models","object-aware video perceiver","vision-guided multi-granularity text fusion","end-to-end training"],"falsifier":"Run InstructSeg's training recipe with its two new modules removed but the same Mipha-3B backbone, and run LISA or VISA with the same backbone under the same data. If the +7.2 cIoU on RefCOCO+ and +11.0 J&F on ReVOS reasoning largely persist with a shared backbone, the claim that the architecture is responsible would be falsified; if they shrink or vanish, the backbone is the driver.","tokens_in":15658,"feed_emoji":"✂️","tokens_out":6984,"duration_ms":55158,"temperature":0.7,"pith_summary":"This paper tries to show that four text-guided segmentation tasks—referring and reasoning segmentation in both images and videos—can be handled by a single end-to-end model rather than by separate specialists or by bolting together a video-LLM, a tracker, and a segmentation module. It introduces InstructSeg, a 3-billion-parameter multi-modal large language model pipeline with frozen vision encoders and LoRA fine-tuning, and reports state-of-the-art numbers on RefCOCO/+/g, ReasonSeg, Ref-YouTube-VOS, Ref-DAVIS, and ReVOS. On the reasoning-heavy ReVOS benchmark it claims a +11.0 J&F gain over the previous best method while using fewer parameters. The point of the unification is that image and video segmentation share the same instruction-following core, so a single training run should transfer understanding across both domains. The practical payoff would be one model that a user can point at either a photo or a video, ask in plain language for an object or something requiring world-knowledge reasoning, and get a mask.","feed_headline":"One 3B model tops all four instructed-segmentation benchmarks","feed_subtitle":"It unifies referring and reasoning segmentation across images and videos, beating larger specialists and 13B MLLMs.","key_machinery":"The two modules that carry the argument are the Object-aware Video Perceiver (OVP) and the Vision-guided Multi-granularity Text Fusion (VMTF). OVP uses learnable queries in cross-attention layers to condense a short sequence of reference frames plus the language instruction into fixed-length tokens that the LLM consumes, giving video tasks temporal and object information without a separate tracker. VMTF takes the LLM's per-token text embeddings, pools a global text embedding, and refines both global and detailed embeddings through cross-attention with fine-grained visual features, producing multi-granularity text embeddings that act as the mask classifier. The surrounding design—Mask2Former-style mask decoding, LoRA fine-tuning of the LLM, and frozen CLIP and visual encoders—makes the pipeline end-to-end trainable with a single loss composed of text, classification, and mask terms.","core_discovery":"On the paper's own terms, the central claim is that instructed visual segmentation (IVS)—referring expression segmentation, reasoning segmentation, referring video object segmentation, and reasoning video object segmentation—is a single task family, and that one end-to-end MLLM can master all of them. InstructSeg feeds the image or key frame, text tokens, and compressed reference-frame tokens from the object-aware video perceiver into one large language model, which emits mask tokens and detailed text embeddings; the vision-guided multi-granularity text fusion module combines global and per-token text information under visual guidance, and a Mask2Former-style decoder produces masks and scores. Trained jointly on RefCOCO/+/g, ReasonSeg, Ref-YouTube-VOS, ReVOS, and LLaVA-150k, the model reports higher scores than segmentation specialists and prior MLLM-based generalists on all reported benchmarks, including image-level referring, image-level reasoning, video referring, and video reasoning splits.","pith_inferences":["If multi-granularity text fusion is the right mechanism, similar fusion of global and token-level embeddings could improve other MLLM grounding outputs, such as referring comprehension or grounded captioning, not just masks.","The object-aware video perceiver's token compression points toward scaling to longer videos: the paper already reports gains when the reference frame count rises from 4 to 8, suggesting a longer-video benchmark could test where further gains saturate.","The IVS task union suggests a natural next step: a combined benchmark where a model receives a mix of image and video instructions and is scored on all four task types at once, making cross-domain transfer an explicit evaluation rather than a by-product."],"forward_implications":["A single 3B model replaces the two-stage pipelines that combine a frame sampler, a video-LLM, and a tracker for video segmentation.","Training jointly on image and video tasks transfers across them: the paper reports that an image-only model shows zero-shot ability on video reasoning, and a video-only model already performs well on RefCOCO.","Reasoning-style training alone transfers to referring tasks, supporting the view that reasoning and referring share one instruction-following core.","The reported gains on RefCOCO/+/g and ReasonSeg indicate that an MLLM with frozen encoders and LoRA can beat dedicated segmentation specialists, not just other generalists."],"supporting_citations":[{"why":"Defines the reasoning segmentation task and ReasonSeg benchmark, and supplies the segment-token paradigm InstructSeg extends.","marker":"[18]"},{"why":"Closest prior unified image/video MLLM segmentation method; supplies ReVOS benchmark and the main comparison numbers InstructSeg claims to beat.","marker":"[47]"},{"why":"Strong image-domain MLLM segmentation baseline whose RefCOCO/+/g and ReasonSeg results InstructSeg uses as the bar to surpass.","marker":"[53]"},{"why":"Provides the transformer and pixel decoder structure plus the mask and class loss formulation used to turn LLM embeddings into masks and scores.","marker":"[7]"},{"why":"The pre-trained lightweight MLLM (with Phi-2) chosen as the base model, whose language abilities are part of what the paper's results build on.","marker":"[55]"},{"why":"Provides the Ref-YouTube-VOS benchmark and the referring video segmentation formulation used in training and evaluation.","marker":"[37]"},{"why":"Supplies the LLaVA-150k vision-language instruction data used alongside the segmentation datasets in joint training.","marker":"[25]"}],"fun_headline_variants":["InstructSeg: one model masters all instructed segmentation","3B MLLM unifies image and video segmentation tasks","Joint training powers InstructSeg past all specialists","Single model tops every IVS benchmark in the paper","InstructSeg: referring and reasoning in one pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported margins come from the proposed architecture: all comparisons use different base language models (LLaVA/Vicuna, Chat-UniVi, Phi-2), and no same-backbone control is reported, so part of the gain could be the stronger base model rather than the new modules.","fun_headline_variants_meta":{"raw":{"variants":["InstructSeg: one model masters all instructed segmentation","3B MLLM unifies image and video segmentation tasks","Joint training powers InstructSeg past all specialists","Single model tops every IVS benchmark in the paper","InstructSeg: referring and reasoning in one pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1347,"prompt_tokens":935,"completion_tokens":412,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":551,"tokens_out":412,"duration_ms":4179,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:33:47.694735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run InstructSeg's training recipe with its two new modules removed but the same Mipha-3B backbone, and run LISA or VISA with the same backbone under the same data. If the +7.2 cIoU on RefCOCO+ and +11.0 J&F on ReVOS reasoning largely persist with a shared backbone, the claim that the architecture is responsible would be falsified; if they shrink or vanish, the backbone is the driver.","supporting_citations":[{"cited_title":"Masked-attention mask trans- former for universal image segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the transformer and pixel decoder structure plus the mask and class loss formulation used to turn LLM embeddings into masks and scores."},{"cited_title":"Urvos: Unified referring video object segmentation network with a large-scale benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the Ref-YouTube-VOS benchmark and the referring video segmentation formulation used in training and evaluation."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-150k vision-language instruction data used alongside the segmentation datasets in joint training."}],"review_version":1}