{"id":"90859ae3-9b12-4a0f-be28-5fda3bf594c9","arxiv_id":"2412.01136","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SOLA aligns SAM2 object tokens with language to select video mask tracks, achieving state-of-the-art J&F 48.6 on MeViS with 32.9M trainable parameters.","lead":"A new method called SOLA selects the correct object track from a frozen video segmentation model by matching its internal object tokens to a sentence, and it reaches the top score on the MeViS benchmark. The key trick is freezing the heavy vision model and training only a small language-selection module, so training runs on a single GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim depends on unquantified proposal coverage: candidate tracks from SAM2 + Grounding DINO may omit referred objects, and selection cannot recover them; an oracle/recall check would settle whether the 48.6 J&F is a true token-alignment result.","rationale":"The reader identified proposal coverage as the weakest assumption, and I agree that this is the load-bearing point. My read adds a concrete mechanism: Grounding DINO is language-conditioned and is used at inference to generate box prompts, so the proposal stage already does part of the referring work. That does not make the method wrong, but it means the ablation in Table 4 does not cleanly attribute the gain to token-language alignment. The paper has real independent strengths: fully frozen SAM2, precomputed tokens enabling single-GPU training, a lightweight 32.9M-parameter selection module, and ablations showing the value of the selection module, the contrastive alignment loss, and the two attention types. These support the conditional acceptance posture. The proposed oracle/recall test is cheap and would settle whether the missing-object concern actually limits the reported numbers. Since the reader already issued a conditional verdict, my stress-test does not move that verdict; it sharpens the condition: the authors should report candidate coverage statistics or release code enabling such a check.","tokens_in":13390,"tokens_out":3846,"duration_ms":38642,"concrete_test":"On the MeViS validation set, for every expression and its ground-truth mask track, compute the max mean IoU across all SOLA candidate tracks, and report (i) the fraction of expressions with at least one candidate track at mIoU ≥ 0.5 (proposal recall) and (ii) an oracle J&F obtained by selecting the best candidate for each expression. Compare the oracle to SOLA's reported 48.6 J&F. If oracle ≈ SOLA, proposal coverage is not the bottleneck and the central claim survives; if oracle is substantially higher or recall is low, the SOTA result is ceiling-limited by the SAM2/GDINO proposal stage rather than by the language-aligned selection module.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold—that frozen SAM2 object tokens are a sufficient video-level representation for language-guided RVOS—the pool of candidate tracks generated in Sec. 3.3 must contain the referred object for every test expression. This is not guaranteed: candidates come from grid points plus Grounding DINO boxes sampled every fourth frame, and any object missed by this prompt set has no object token O_i, so Eq. (5) can only assign scores over an incomplete set and cannot recover the missed object. The IoU pseudo-labels in Sec. 3.6 inherit the same ceiling: if no candidate track reaches mIoU ≥ 0.5 with the ground-truth track, that expression contributes no positive training example, and the module is never taught to identify objects the proposer missed. The paper does not report candidate recall or an oracle upper bound, so the 48.6 J&F could be limited by the proposer rather than by token-language alignment. Moreover, inference-time Grounding DINO boxes are language-conditioned, so some referring capability is embedded in the proposal stage; the 'w/o selection module' baseline (36.9 J&F, Table 4) does not isolate this, since it still uses Grounding DINO detections. The assumption is not internally inconsistent, but it is load-bearing and currently unquantified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SOLA, an RVOS framework that treats SAM2 object tokens as compact video-level object representations. Candidate mask tracks are first generated by prompting a frozen SAM2 with a grid of points and with Grounding DINO bounding boxes sampled every fourth frame; the corresponding SAM2 object pointers are then aligned with RoBERTa text features by a lightweight track selection module. A binary alignment score per track is produced, and tracks above a threshold are merged. Training uses pseudo-labels obtained by thresholding the mIoU between candidate tracks and the ground-truth track, with a BCE loss plus a contrastive alignment loss based on learnable negative anchors. The paper reports state-of-the-art results on MeViS (J&F 48.6 vs 46.4 for DsHmp), zero-shot generalization to Ref-YouTube-VOS and Ref-DA VIS, combined-dataset training, ablations, and analyses of SAM2 object tokens.","tokens_in":13680,"tokens_out":4053,"duration_ms":37431,"significance":"If the central claim holds, the paper makes a useful contribution: it shows that frozen SAM2 object pointers carry enough spatio-temporal object information to support language-guided track selection, and it does so with a very lightweight trainable module (32.9M parameters) that can be trained on a single GPU using precomputed tokens. The design is clearly specified, the ablations are informative, and the zero-shot and combined-dataset results strengthen the evidence that the representation generalizes. The main significance hinges on whether the reported MeViS numbers are a genuine selection-module result rather than an artifact of candidate-track coverage, and on whether the comparison set and evaluation split are complete and clearly identified.","major_comments":[{"comment":"The proposal-coverage assumption is load-bearing and currently unquantified. Candidate tracks are generated only from grid points and Grounding DINO boxes sampled every fourth frame; any referred object missed by this prompt set has no object token O_i, so Eq. (5) can only assign scores over an incomplete set. The pseudo-labeling in Section 3.6 inherits the same ceiling: if no candidate track reaches mIoU >= tau with the ground-truth track, that expression contributes no positive training example. The paper does not report candidate recall or an oracle upper bound (e.g., selecting the candidate with the highest mIoU). I request an oracle-selection curve and per-expression recall on MeViS so that the reader can determine how much of the 48.6 J&F is attributable to the language-aligned selection module rather than to the proposal stage, and how much is lost on expressions whose referent is absent from the candidates.","section":"3.3, 3.6, Table 1"},{"comment":"The evaluation split for the main MeViS comparison is not identified. The dataset section states that MeViS has a validation set of 140 videos and a test set of 154 videos, but Table 1 only says “MeViS”. If the reported 48.6 J&F is on the validation set, this must be stated; if it is on the test set, the paper must explain how the test set was obtained and what protocol was used. In either case, the main results should include multiple-seed standard deviations, since the margin over DsHmp is 2.2 J&F and the paper currently reports single-run scores without any indication of variance.","section":"4.1, Table 1"},{"comment":"The related-work section discusses SAM/SAM2-based RVOS methods (Ref-SAM and AL-RefSAM2) and criticizes their prompt-level use of SAM2, but Table 1 does not compare against them. For the “state-of-the-art on MeViS” claim to be evaluable, the paper should either include the published MeViS results of these methods or explicitly state the reason for their exclusion (for example, if they do not report on MeViS or use additional training data).","section":"2, Table 1"}],"minor_comments":[{"comment":"The citation label for ReferFormer is inconsistent: Tables 2, 3, and A.1 cite ReferFormer as [12], but reference [12] is the CLIP paper (Jia et al.), while ReferFormer is reference [28] (Wu et al.).","section":"Tables 2, 3, A.1"},{"comment":"The project page URL differs between the abstract (https://cvlab-kaist.github.io/SOLA) and the introduction (https://github.com/cvlab-kaist/SOLA); please unify.","section":"Abstract, Section 1"},{"comment":"There is a typo in “Nneg = 32for number of negative anchors” (missing space), and the sentence “relies solely exclusively on object tokens” in Section 4.3 is redundant; one of the two adverbs should be removed.","section":"4.2"},{"comment":"The token-similarity analysis would be easier to interpret if the caption stated how the curve was computed: the number of object-token pairs, the binning of mIoU, and whether the pairs are from the same video or across videos.","section":"4.5, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for a computer-vision venue. The central idea is reasonable and the efficiency argument is attractive, but the SOTA claim needs the oracle/recall analysis to address the proposal-coverage ceiling and a clear statement of the evaluation split. I would be willing to accept after these points are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first RVOS paper I know of that uses SAM2's internal object tokens as video-level object representations rather than treating SAM2 as a mask generator. That is a genuinely new angle, and the lightweight selection module (32.9M trainable params) with IoU-based pseudo-labeling is clean, effective design. Second, the SOTA claim on MeViS (48.6 J&F) is plausible but hinges on an unquantified assumption about proposal coverage.\n\nThe new pieces hold up well. Taking the object tokens (memory pointers) from a frozen SAM2 and aligning them with RoBERTa text features in a small transformer is a sensible way to reuse a video segmentation foundation model. The IoU pseudo-labeling is simple: label a candidate track positive if its mask overlaps the ground truth track well, negative otherwise, and train a binary classifier with a contrastive loss. Ablations support the choices: the selection module adds 11.7 J&F over a baseline, both attention types matter, and the alignment loss adds 4.1. The efficiency (single 3090, 7 hours) is a real practical plus.\n\nThe main soft spot is exactly what the stress-test flags. Candidate tracks come from grid points plus Grounding DINO boxes sampled every fourth frame. If the referred object is not in that pool, the selection module can only rank what exists; it cannot recover a missed object. The paper never reports candidate recall or an oracle upper bound, so we do not know whether 48.6 reflects the language-alignment module or just the track proposer. That is load-bearing, not a nitpick. The 'w/o selection module' baseline does not fully isolate the effect either, because it still uses Grounding DINO detections at inference; the 36.9-to-48.6 gap mixes temporal selection with frame-level detection quality.\n\nSmaller issues: no error bars in the main table, the MeViS split (validation vs test) is not labeled in Table 1, and the SAM-based related work (Ref-SAM, AL-RefSAM2) is discussed but not compared. No code appears to be released, which limits reproducibility.\n\nThe central claim — that frozen SAM2 object tokens are alignable with language and work well for RVOS — holds up as a proof of concept. The paper deserves serious refereeing. I would ask for a candidate-recall/oracle analysis before acceptance; without it, the headline number is hard to interpret.","headline":"Genuinely new way to reuse SAM2 tokens for RVOS, with a credible SOTA on MeViS—but the proposal-coverage assumption needs an oracle check before I'd fully trust the headline number.","tokens_in":14260,"tokens_out":3312,"would_cite":true,"duration_ms":28517,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen SAM2 tokens pick the right track, topping MeViS with 48.6 J&F.","keywords":["referring video object segmentation","SAM2","object tokens","language alignment","track selection","pseudo-labeling","video segmentation","zero-shot generalization"],"falsifier":"A concrete test: on a held-out set of videos, restrict the prompt set to a subset of object categories, then ask whether SOLA can still select the referred track for a held-out category. If the selection accuracy drops to zero whenever the candidate set excludes the referent, the claim that selection rather than proposal carries the performance is falsified. More quantitatively, compute the recall of candidate tracks with mIoU above 0.5 against ground-truth tracks on MeViS; if that recall is substantially below the reported J&F, then proposal-stage failures, not language alignment, dominate the error.","tokens_in":13194,"feed_emoji":"🎯","tokens_out":4657,"duration_ms":39655,"temperature":0.7,"pith_summary":"SOLA proposes that referring video object segmentation can be reframed as selecting, among candidate object tracks produced by a frozen SAM2, the one that best matches a natural language expression. The paper claims that SAM2's object tokens—the per-frame object pointers the model already outputs—are compact video-level representations that encode both motion and identity, so a lightweight language-aligned selection module can align them with text features. On MeViS this selection approach reaches J&F 48.6, surpassing the previous best result with only 32.9M trainable parameters, and it transfers zero-shot to Ref-YouTube-VOS and Ref-DA VIS. The core message is that a frozen video segmentation model's internal tokens are a sufficient basis for language-conditioned video object selection, provided the candidate tracks cover the objects a sentence might refer to.","feed_headline":"Frozen SAM2 tokens pick the right track: 48.6 J&F on MeViS","feed_subtitle":"SOLA shows that SAM2's object tokens alone carry enough motion and identity to align with natural language.","key_machinery":"The load-bearing object is the SAM2 object token: the object pointer produced by SAM2's mask decoder at each frame, concatenated over time to form a video-level token per candidate track. A short-term motion encoder (a 1D convolution along the temporal dimension) refines the token, and a stack of attention layers—inter-object attention across tracks in a frame, motion attention along each track's time axis, and object-to-language cross-attention with text tokens—aligns the visual and linguistic representations. The alignment score is computed by aggregating per-token text similarity, and training uses binary cross-entropy on IoU-based pseudo-labels plus a contrastive alignment loss that pulls matched tokens toward the text anchor and pushes mismatched tokens toward learnable negative anchors.","core_discovery":"The central claim is that SAM2's object token representations, extracted along with mask tracks, contain enough spatio-temporal objectness that a lightweight module can align them with referring expressions and select the correct track. SOLA operationalizes this by precomputing candidate tracks with fully frozen SAM2, representing each track as a sequence of object pointers over time, and training a small selection network with IoU-based pseudo-labels derived from ground-truth mask tracks. The selection module outputs an alignment score per track, and tracks scoring above a threshold are merged into the final segmentation. The result is a new state of the art on MeViS (J&F 48.6) and strong zero-shot transfer to Ref-YouTube-VOS (J&F 47.9) and Ref-DA VIS (J&F 45.4), validating that SAM2's frozen object tokens are language-alignable with light supervision.","pith_inferences":["The framework's performance upper bound is set by the proposal stage: if neither grid points nor Grounding DINO boxes prompt the referred object, no candidate track exists and selection necessarily fails. A testable extension is to measure candidate recall against ground-truth objects and evaluate SOLA on only the covered subset to isolate selection accuracy.","The same frozen-token representation could be reused for other language-conditioned video tasks, such as referring attribute classification or instruction-guided video editing, with the same selection head, since the alignment objective is task-agnostic.","The paper's corruption experiments hint that SAM2 tokens are robust to input perturbations, but the mechanism is not isolated; a natural follow-up is to test whether the selection module itself inherits robustness or whether the robustness comes entirely from the frozen backbone."],"forward_implications":["Because the selector consumes only precomputed tokens, training the full RVOS pipeline takes roughly 7 hours on a single GPU, making such systems feasible without large-scale end-to-end video training.","The zero-shot results on Ref-YouTube-VOS and Ref-DA VIS suggest that language-aligned track selection trained on one dataset can transfer to others, positioning it as a reusable building block for language-conditioned video object tasks.","The IoU-based pseudo-labeling cleanly separates track quality from selection quality: the selector can be retrained or swapped without regenerating SAM2 tracks, which simplifies future improvements.","Since SAM2 is frozen, any future improvement in the underlying segmentation model's object tokens could directly raise SOLA's performance without retraining the selection module.","The lightweight design (32.9M trainable parameters) and the use of generic prompts mean the approach can be applied to arbitrary video datasets that provide language annotations, not just the evaluated benchmarks."],"supporting_citations":[{"why":"Supplies the frozen SAM2 model whose object tokens and mask tracks form the entire visual representation; the method treats it as a fixed feature extractor.","marker":"[24]"},{"why":"The MeViS benchmark is the primary dataset on which state-of-the-art J&F 48.6 is claimed.","marker":"[4]"},{"why":"Grounding DINO provides the bounding-box prompts used alongside grid points to generate candidate mask tracks.","marker":"[19]"},{"why":"RoBERTa is the frozen text encoder whose language features are aligned with object tokens.","marker":"[20]"},{"why":"FuseMix inspires the precomputing strategy that makes single-GPU training feasible by avoiding on-the-fly SAM2 inference.","marker":"[27]"},{"why":"DsHmp is the prior state-of-the-art baseline on MeViS that SOLA is compared against and surpasses.","marker":"[9]"}],"fun_headline_variants":["SOLA aligns frozen SAM2 tokens to language for video track picking","Language-aligned track selection with SAM2 tokens hits 48.6 J&F","Frozen SAM2 tokens + IoU pseudo-labels = SOTA RVOS on MeViS","Zero-shot RVOS: SOLA selects tracks via language-aligned tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline depends on the candidate-generating stage—grid points plus Grounding DINO boxes—actually producing a mask track for every object a sentence might refer to; if the referred object is never prompted, the selection module has nothing to choose, and the method cannot recover.","fun_headline_variants_meta":{"raw":{"variants":["SOLA aligns frozen SAM2 tokens to language for video track picking","Language-aligned track selection with SAM2 tokens hits 48.6 J&F","Frozen SAM2 tokens + IoU pseudo-labels = SOTA RVOS on MeViS","Zero-shot RVOS: SOLA selects tracks via language-aligned tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001048,"raw_usage":{"total_tokens":4389,"prompt_tokens":917,"completion_tokens":3472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":3385}},"tokens_in":533,"tokens_out":3472,"duration_ms":22901,"temperature":1.0,"reasoning_tokens":3385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:39:09.173858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on a held-out set of videos, restrict the prompt set to a subset of object categories, then ask whether SOLA can still select the referred track for a held-out category. If the selection accuracy drops to zero whenever the candidate set excludes the referent, the claim that selection rather than proposal carries the performance is falsified. More quantitatively, compute the recall of candidate tracks with mIoU above 0.5 against ground-truth tracks on MeViS; if that recall is substantially below the reported J&F, then proposal-stage failures, not language alignment, dominate the error.","supporting_citations":[{"cited_title":"Data-efficient mul- timodal fusion on a single gpu","cited_arxiv_id":null,"evidence_quote":"FuseMix inspires the precomputing strategy that makes single-GPU training feasible by avoiding on-the-fly SAM2 inference."},{"cited_title":"Mevis: A large-scale benchmark for video segmentation with motion expressions","cited_arxiv_id":null,"evidence_quote":"The MeViS benchmark is the primary dataset on which state-of-the-art J&F 48.6 is claimed."},{"cited_title":"Decoupling static and hier- archical motion perception for referring video segmentation","cited_arxiv_id":null,"evidence_quote":"DsHmp is the prior state-of-the-art baseline on MeViS that SOLA is compared against and surpasses."}],"review_version":1}