{"id":"b9f3e861-0fcf-4b9d-9843-fe230b5414eb","arxiv_id":"2608.02284","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"By removing SAM 3's text prompts and pooling SigLIP features with decoder attention maps, EOVSAM performs open-vocabulary segmentation in one pass, beating vanilla SAM 3 in accuracy while running up to 338× faster.","lead":"EOVSAM adapts SAM 3, a segmentation model that normally scans one text category at a time, into a single-pass system that segments and classifies all categories in one forward pass. It reports up to 338× faster inference than vanilla SAM 3 with small accuracy gains on open-vocabulary benchmarks, making large-vocabulary segmentation practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attentional Aggregation's point-to-point alignment between decoder attention maps and SigLIP features is asserted, not demonstrated; Eq. (3) may pool spatially misaligned features, jeopardizing the classification head that drives the reported gains.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: the spatial correspondence between A and F_siglip in Eq. (3) is asserted rather than measured. This is the linchpin of the method's novelty and of the ablation gain. My concrete test would directly quantify alignment using the released model. I do not think this rises to REJECT because the end-to-end benchmark improvements are real (if self-reported) and could still hold even if the formal justification is incomplete. The paper is a systems contribution; the concern is about missing validation of a central mechanistic claim, not about internal inconsistency. Therefore the appropriate verdict remains CONDITIONAL, matching the reader's. I agree with the reader's identification of this weakness and with their overall assessment that the central claim is plausible but not fully verified.","tokens_in":12930,"tokens_out":4329,"duration_ms":39666,"concrete_test":"Download the released checkpoint and run EOVSAM on 100 ADE20K val images. For each predicted mask M_i and its corresponding attention map A_i, compute (1) IoU between a thresholded argmax of A_i and the final mask M_i; (2) cosine similarity between the attention-pooled embedding o_i and a mask-pooled embedding computed by applying M_i to F_siglip. Average over all queries with non-empty masks. If the mean IoU is low (<0.5) or the embedding similarity is not significantly above a random-patch baseline, then A and F_siglip are not aligned, and the claimed benefit of Attentional Aggregation is not explained by the mechanism described.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim rests on Eq. (3): object embeddings o_i are obtained by pooling SigLIP features F_siglip with attention maps A produced by the detector decoder from the SAM-3-path features F_enh. The paper asserts that A 'inherently maintains a precise point-to-point spatial correspondence' with F_siglip because the object queries interact with F_enh. But C-RADIOv4's two adapter heads are independently distilled from SAM 3 and SigLIP 2; nothing guarantees that their spatial feature grids share identical patch centers, strides, or receptive fields. The only supervision on A is indirect: classification loss on the pooled embedding plus auxiliary box losses. No direct measurement of alignment is provided. If the grids are offset by even a few pixels or have different effective resolutions, the attention weights will mix features from neighboring regions into the object embedding, degrading open-vocabulary classification. The ablation w/o Attn. Agg. collapses to 22.6 mIoU on A-150 (vs. 39.0 full), so the +16.4 mIoU gain—the paper's main evidence for its core novelty—depends entirely on this unverified premise. This is not a fatal flaw; the benchmark results themselves are indirect evidence that some alignment exists. But the premise is load-bearing and currently unsupported by any direct analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EOVSAM adapts SAM 3 for single-pass open-vocabulary segmentation. It replaces the SAM 3 image encoder with C-RADIOv4, which provides both SAM-3-like features (F_sam) and SigLIP 2 features (F_siglip); removes text cross-attention from the fusion encoder and detector decoder; and trains a Mask2Former-style decoder with 200 object queries to produce masks and attention maps. The attention maps pool SigLIP features into object embeddings (Eq. 3), which are classified with a SigLIP text classifier; a frozen SAM classifier provides objectness and supplementary classification. Training is on COCO Panoptic. The paper reports mIoU of 39.0/16.6/60.9/20.4/97.0 on A-150/A-847/PC-59/PC-459/PAS-20 and ADE20K PQ of 30.9, with large speedups over vanilla SAM 3. Ablations attribute substantial gains to Attentional Aggregation and to inheriting pretrained SAM 3 weights.","tokens_in":13171,"tokens_out":8899,"duration_ms":72330,"significance":"If the claims hold, EOVSAM is a valuable contribution to efficient open-vocabulary segmentation: a single COCO-Panoptic-trained model sets a new state of the art on ADE20K panoptic segmentation among open-vocabulary methods and avoids the vocabulary-length-dependent inference of SAM 3. The paper is well structured, with explicit equations, a useful set of component ablations (Table 4), a direct comparison of aggregation strategies (Table 5), and sufficient training details to enable reproduction. The main caveat is that the core Attentional Aggregation mechanism rests on a spatial-alignment assumption that is asserted rather than directly demonstrated; the empirical gains should be accepted with caution until that link is verified.","major_comments":[{"comment":"Eq. (3) pools SigLIP features F_siglip with attention maps A produced by the detector decoder from the SAM-path features F_enh. The paper asserts that A 'inherently maintains a precise point-to-point spatial correspondence' with F_siglip, but no direct evidence is given. C-RADIOv4's SAM-3 and SigLIP-2 adapter heads are distilled from different teachers; identical grid geometry and semantic alignment do not follow. Table 4 shows that removing Attentional Aggregation drops A-150 mIoU from 39.0 to 22.6, so the +16.4 mIoU gain—the main evidence for the proposed novelty—rests on this unverified premise. Please add direct alignment analysis (e.g., attention-to-mask IoU in SigLIP feature space, grid-offset/receptive-field statistics, or explicit spatial resampling/alignment) or otherwise demonstrate that Eq. (3) does not pool spatially misaligned features.","section":"Efficient Open-Vocabulary Recognition via Attentional Aggregation, Eq. (3)"},{"comment":"The final prediction procedure is incompletely specified. Eq. (4) defines only the SigLIP-classifier probability P_i,k; the paper also introduces a SAM classifier, objectness scores, and a geometric ensemble with weights alpha and beta, but no equation describes how these signals are combined. It is also unclear how the background token is used and how the N=200 masks are converted into the semantic mIoU and panoptic PQ numbers in Tables 1-2. Additionally, alpha=0.7, beta=1.0 in Table 6 is selected by sweeping on the same test benchmarks, so the reported final numbers include test-set selection pressure; the effect is small relative to the alpha=beta=1.0 row, but the protocol should be disclosed and, ideally, the coefficients chosen on a validation split. Please provide the ensemble formula, the inference/post-processing protocol, and clarification of the selection procedure.","section":"Objective / Additional Experiments, Table 6"},{"comment":"The 'up to 338x' speedup claim is not fully supported. Table 3 reports FPS but does not state the exact measurement protocol (e.g., batch size, warm-up, whether text encoding is included, whether timings are end-to-end or model-only). Since text features are precomputed, the comparison should specify what is timed. Also, EOVSAM uses C-RADIOv4-H and a different resolution than SAM 3's official setting, so the speed advantage may be dominated by backbone/resolution rather than the single-pass design. Please provide a latency breakdown (image encoder / fusion encoder / decoder / post-processing) and the exact setting that yields 338x.","section":"Experiments / Table 3"}],"minor_comments":[{"comment":"The term 'prompt-free' should be qualified: category text embeddings are still used for classification via the SigLIP/SAM classifiers; only the mask-generation path removes per-category prompts.","section":"Adapting SAM 3 for Single-Pass Full-Image Segmentation"},{"comment":"The learning-rate decay milestones are not specified, and 'C-RADIOv4-H' is not defined in the manuscript. Please provide the exact schedule and model configuration.","section":"Experimental Setup / Implementation Details"},{"comment":"Add a note identifying which row/dataset corresponds to the 'up to 338x' speedup, since the reported FPS values vary by dataset and resolution.","section":"Experiments / Table 3"},{"comment":"Mark which components are frozen and which are trainable. This would clarify the 'end-to-end' claim, because the visual backbone and both text encoders are frozen.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"I found no indication of deliberate misreporting. The most important issue is the unverified spatial-alignment assumption behind Eq. (3); if the authors can supply direct evidence (or an explicit alignment mechanism), the paper is close to acceptable. The alpha/beta tuning and speed-measurement details are secondary but should be disclosed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EOVSAM is a competent, genuinely useful systems paper. It does what it says: turns SAM 3's per-category forward passes into one pass by dropping text cross-attention, and uses decoder attention maps to pool SigLIP features for open-vocabulary classification. The reported gains over their reproduced SAM 3 are consistent across datasets, and the speedups are in the right ballpark. The architecture is simple, the equations check out, and the ablations are informative. The +16.4 mIoU gain from Attentional Aggregation is strong internal evidence that the mechanism carries the open-vocabulary generalization.\n\nThe soft spots are things a good reviewer would ask for, not deal-breakers. The geometric ensemble coefficients α=0.7, β=1.0 come from a sweep over test benchmarks (Table 6); the authors call the choice 'unified' but there's no held-out selection. A quick fix is to report a validation-based selection or to show the model is robust across the whole Table 6 range, which it mostly is. The speedup headline 'up to 338×' divides by a reproduced SAM 3 running at 0.01–0.03 FPS; the ratio is inherently fragile, and the absolute latency numbers in Table 3 are the more honest comparison. There are no error bars anywhere, so the SOTA claims on ADE20K panoptic rest on single runs.\n\nThe more substantive concern is the alignment premise behind Eq. (3). The paper asserts that decoder attention maps have point-to-point correspondence with the SigLIP features because C-RADIOv4 produces them in parallel. That's plausible — it's one ViT with two adapter heads, so the spatial grids likely coincide — but it's not measured. A simple check (e.g., comparing attention maps against SigLIP feature similarity or mask-pooled embeddings) would make the central mechanism much more credible. The benchmark results are indirect evidence that the alignment is adequate, so I don't view this as fatal. The reader's stress-test is appropriately calibrated.\n\nWho is this for? Anyone working on open-vocabulary segmentation or efficient SAM adaptation. It's a solid baseline and the method is reproducible enough to build on. The authors should release code with a commit hash and disclose the coefficient selection. I'd send this to peer review.","headline":"Solid single-pass SAM 3 adaptation with real speed and accuracy gains; the attentional-pooling mechanism is plausible but the alignment premise deserves a direct check.","tokens_in":13867,"tokens_out":2328,"would_cite":true,"duration_ms":20402,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EOVSAM removes SAM 3's prompt conditioning to turn it into a single-pass open-vocabulary segmenter, pooling SigLIP features with decoder attention maps to classify masks in the same forward pass.","keywords":["open-vocabulary segmentation","SAM 3","single-pass inference","attentional aggregation","panoptic segmentation","zero-shot recognition","vision-language models","efficient inference"],"falsifier":"Compute the overlap between each attention map and its final mask (e.g., attention-to-mask IoU) in SigLIP feature space; if overlap is low yet classification still works, the aggregation mechanism is not doing what the paper claims. Alternatively, shift the SigLIP grid by a few pixels at inference and measure the drop in mIoU: a sharp drop would confirm dependence on exact alignment, while little change would imply another mechanism.","tokens_in":12659,"feed_emoji":"⚡","tokens_out":5368,"duration_ms":47396,"temperature":0.7,"pith_summary":"EOVSAM claims that SAM 3—which currently runs one forward pass per searched noun phrase—can be rewritten as a single-pass open-vocabulary segmenter without losing localization strength. It removes text cross-attention, making the model prompt-free, and classifies the resulting masks by pooling SigLIP 2 image features through the decoder's own attention maps. Trained only on COCO Panoptic, the model reports mIoU of 39.0/16.6/60.9/20.4/97.0 on A-150/A-847/PC-59/PC-459/PAS-20, a panoptic PQ of 30.9 on ADE20K (state of the art among open-vocabulary panoptic models), and inference up to 338× faster than vanilla SAM 3. The practical stake is that open-vocabulary segmentation need not choose between accuracy and latency: one frozen backbone with a light trainable decoder can do both.","feed_headline":"SAM 3 segments any vocabulary in one pass, 338x faster","feed_subtitle":"A prompt-free SAM 3 with attention-pooled SigLIP features beats the original on accuracy at a fraction of the cost.","key_machinery":"Attentional Aggregation is the load-bearing mechanism: across the six decoder layers, a learned set of weights pools the multi-head attention maps into one soft attention map per object query, and that map averages the SigLIP 2 feature grid into an object embedding (Eq. 3). The embedding is classified by cosine similarity to SigLIP text embeddings, with a learnable temperature. This differs from prior mask-cropping/mask-pooling schemes because the attention map is never binarized, so the gradient from the open-vocabulary classification loss reaches the mask decoder. The other half of the machinery is the prompt-free reuse of SAM 3: removing text cross-attention and keeping the pretrained wei","core_discovery":"The central claim is that the multi-pass, per-category loop of SAM 3 is an architectural accident, not a necessity. EOVSAM turns SAM 3's detector decoder into a prompt-free mask proposal engine: it drops text cross-attention from the fusion encoder and decoder, keeps the pretrained SAM 3 weights, and uses a vision backbone that emits SAM-oriented and SigLIP-oriented features on the same grid. The decoder's learned attention maps then aggregate SigLIP 2 features into per-object embeddings, which are scored against a SigLIP text classifier. Because the aggregation is a soft, differentiable average rather than binarized mask pooling, the recognition loss flows into mask localization, and the pa","pith_inferences":["A direct test of the alignment premise—measuring how well each attention map overlaps its corresponding mask in SigLIP feature space—would separate the benefit of the aggregation rule from the benefit of the end-to-end training signal.","The geometric ensemble with the SAM classifier is nearly unnecessary at α=β=1.0, so a simpler single-classifier variant could be deployed; the ensemble mainly adds robustness for seen categories.","The method suggests a recipe for other DETR-like segmenters: freeze a strong mask proposer, attach a dual-feature backbone, and let attention maps pool a vision-language feature map. One could test whether this transfers to SAM 2 or other prompt-based segmenters.","Since the model uses 200 queries regardless of scene content, adaptive query selection or iterative refinement could be explored for scenes with many objects."],"forward_implications":["Inference cost stops growing with vocabulary size; the only added cost for a larger vocabulary is encoding the text labels once.","A single checkpoint trained on COCO Panoptic covers both semantic and panoptic tasks, and one trained model serves arbitrary unseen category lists without fine-tuning.","Resolution can be dropped (e.g., 512 vs 1152) with only small mIoU loss, giving a latency-performance knob for deployment.","The fixed 200 object queries bound the number of masks per image; scenes with more than 200 objects would exceed the proposal budget.","The approach avoids multi-stage pipelines and post-processing heuristics used by existing methods, since masks and labels come from the same forward pass."],"fun_headline_variants":["One pass, 338x faster: SAM 3 without the category loop","EOVSAM: single-pass open-vocab segmentation, 338x speedup","SAM 3 de-looped: 338x faster open-vocab segmentation","No more vocabulary traversal: SAM 3 in one pass"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire classification branch assumes the decoder's attention maps and the SigLIP feature map are aligned pixel-to-pixel spatially; the paper does not report a direct measurement of that alignment.","fun_headline_variants_meta":{"raw":{"variants":["One pass, 338x faster: SAM 3 without the category loop","EOVSAM: single-pass open-vocab segmentation, 338x speedup","SAM 3 de-looped: 338x faster open-vocab segmentation","No more vocabulary traversal: SAM 3 in one pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000431,"raw_usage":{"total_tokens":2046,"prompt_tokens":764,"completion_tokens":1282,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1200}},"tokens_in":508,"tokens_out":1282,"duration_ms":11343,"temperature":1.0,"reasoning_tokens":1200,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T09:58:22.793264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between each attention map and its final mask (e.g., attention-to-mask IoU) in SigLIP feature space; if overlap is low yet classification still works, the aggregation mechanism is not doing what the paper claims. Alternatively, shift the SigLIP grid by a few pixels at inference and measure the drop in mIoU: a sharp drop would confirm dependence on exact alignment, while little change would imply another mechanism.","supporting_citations":[],"review_version":1}