{"id":"dc558ffb-e8ca-4541-b47c-979633df5969","arxiv_id":"2505.04410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A decoupled attention fine-tuning method that uses self-distillation and VFM correlation distillation to improve CLIP's local feature quality for open-vocabulary dense prediction.","lead":"DeCLIP is a new training recipe that reworks CLIP's attention into two separate streams, one for recognizing objects and one for keeping spatial relationships clean, then tunes each with a different teacher. It reports consistent gains over prior open-vocabulary detection and segmentation methods on several standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Q-Q attention switch is not isolated from the decoupled distillation, so Table 1's decoupling gain may largely be the known SCLIP/ClearCLIP effect.","rationale":"The paper's central empirical claim is that DeCLIP significantly improves open-vocabulary dense prediction, and that claim is supported by extensive tables, including sanity checks against EVA-CLIP baselines (Tables 9 and 10). The load-bearing weakness is not the numerical results but the attribution of those results to the decoupled distillation. Table 1's comparison conflates the Q-Q attention switch with the decoupled training, and prior work demonstrates that the Q-Q switch alone is a strong training-free dense-feature improver. Without an ablation isolating these factors, the paper's novelty claim is underdetermined. This is exactly the reader's weakest assumption, so my read does not change the verdict. The proposed test is a single, cheap ablation that would directly resolve whether the context distillation branch is necessary or whether the gains reduce to a known trick plus self-distillation.","tokens_in":26803,"tokens_out":8081,"duration_ms":78128,"concrete_test":"Run the missing ablation: fine-tune DeCLIP on COCO with the decoupled Q-Q attention and only L_content (set λ=0 in Eq. 10), using the same hyperparameters as Table 1. Measure COCO Thing mAcc and Context59 mIoU and compare with row 3 of Table 1. If the drop is less than about 1 point, the VFM context distillation is not the source of the decoupling gains; if the drop is large, the decoupled distillation is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest empirical evidence for DeCLIP is Table 1, but the row labeled 'Self+VFM+Decouple' differs from 'Self+VFM Distillation' in two simultaneous ways: it switches attention from Q-K^T to Q-Q^T (Eq. 6 vs Eq. 2) and it splits the feature into content/context with separate losses. Because SCLIP [63] and ClearCLIP [38] already show that the Q-Q switch alone improves dense CLIP features, the +5.5 mAcc / +6.7 mIoU gain attributed to 'decoupling' could be substantially due to the known attention modification rather than the decoupled distillation itself. The paper also motivates the method with the 'proxy token' phenomenon observed in OpenAI CLIP (Fig. 3a), but DeCLIP is trained on EVA-CLIP; the supplementary sanity checks (Tables 9-10) show EVA-CLIP raw features behave differently, and no attention-map evidence is provided for EVA-CLIP. A single missing ablation would settle this: keep the decoupled Q-Q attention but drop the VFM context distillation (L_context=0), and compare with full DeCLIP on the Table 1 metrics. If the gap is negligible, the contribution of the novel context-distillation branch is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DeCLIP is an unsupervised pre-fine-tuning method that modifies the final self-attention block of a CLIP image encoder to produce two features: a context feature X_context, formed by Q-Q self-attention, and a content feature X_content, formed by applying that attention to V. The content feature is distilled toward CLIP's own crop-level [CLS] representations, while the context feature is distilled toward the correlation volumes of a frozen vision foundation model such as DINOv2, SAM, or DINO. The paper reports consistent improvements over strong baselines in open-vocabulary detection with F-ViT and OV-DQUO, open-vocabulary semantic segmentation with CAT-Seg, training-free VLM-feature segmentation, and region classification. It also contributes an attention-visualization analysis of a 'proxy token' phenomenon in CLIP's image tokens.","tokens_in":27043,"tokens_out":7457,"duration_ms":75239,"significance":"If the decoupling mechanism is responsible for the gains, the method is a valuable general recipe: it requires only unlabeled images, transfers across detection and segmentation frameworks, and is supported by a broad benchmark suite, a code release, and sanity checks against EVA-CLIP and ClearCLIP. The empirical scope is the paper's main strength: Tables 2-5 cover multiple detectors, segmenters, resolutions, and datasets, and the gains are consistent even when modest. The central caveat is that Table 1, the key evidence for the decoupling conclusion, changes two variables at once (attention formulation and loss structure), so the mechanism-level claim is not yet isolated. I did not find a circularity problem: the content branch uses the frozen CLIP itself as teacher and the context branch uses an external VFM; no evaluation labels are used in training.","major_comments":[{"comment":"The row 'Self+VFM Distillation' and the row 'Self+VFM+Decouple' differ in two simultaneous ways: the attention is changed from Q-K^T to Q-Q^T and residual connections are removed, and the single feature is split into content and context branches with separate losses. Since SCLIP and ClearCLIP already demonstrate that the Q-Q attention change alone improves dense CLIP features, the reported gains of +5.5 mAcc and +6.7 mIoU cannot be attributed to the decoupled distillation without an additional control. Please add an ablation that keeps the decoupled Q-Q attention fixed and removes only the VFM context loss (L_context=0), and ideally a second variant that keeps the combined loss on a single Q-K feature. This is the minimal experiment needed to support the paper's central claim.","section":"Table 1; Sec. 3.1, Eqs. (2) and (6)"},{"comment":"The 'proxy token' phenomenon is visualized for OpenAI CLIP, but DeCLIP is trained from EVA-CLIP. The supplementary sanity checks show that EVA-CLIP is not a drop-in equivalent: vanilla EVA-CLIP already outperforms OpenAI CLIP on VLM-feature segmentation (18.9 vs 14.1 average mIoU in Table 10), and EVA-ClearCLIP underperforms ClearCLIP. Please provide the same attention-map analysis for EVA-CLIP, or restrict the causal narrative to the backbone actually used in the experiments.","section":"Sec. 2.2, Fig. 3(a); Supp. Tables 9-10"},{"comment":"The paper does not explicitly state which feature is used as the backbone representation at inference. Equation (5) defines X_content as a function of Attn_context and V, and X_context is also available, but the downstream detectors and segmenters are said to use 'DeCLIP' as the image encoder without specifying whether they consume X_content, X_context, or a combination. Please state the inference path explicitly, including how the final projection and residual/FFN operations are treated after the modified last attention block.","section":"Sec. 3.1, Fig. 5"},{"comment":"The claim that content and context features are optimized 'without interference' is not exact: X_content = Proj(Attn_context * V), so the content loss propagates gradients into X_context through the attention weights, and both losses share parameters. The decoupling is at the loss level rather than a full separation of the computational graph. Please clarify this point and discuss whether the optimization-conflict story is consistent with the actual gradient structure.","section":"Sec. 3.1, Eqs. (4)-(6)"}],"minor_comments":[{"comment":"The row labeled 'Self+VFM Distillation [36]' cites reference [36], which is Segment Anything; this citation does not correspond to a distillation scheme and should be corrected or replaced with an explicit statement that the row was implemented by the authors.","section":"Table 1, row 2"},{"comment":"The paragraph begins with 'we employ an image patching method' using a lowercase 'we'; this should be 'We employ' and the sentence should be reworded for clarity.","section":"Sec. 3.2, Eq. (7)"},{"comment":"There are minor typos in the supplementary material: 'Sentitivity Analysis' should be 'Sensitivity Analysis', and 'OVSS based on VLM feautures' should be 'features'.","section":"Supp. Table 11 and Sec. 7.2"},{"comment":"The number of sub-regions k in the content distillation is randomly sampled from [1,6], but no sensitivity study is reported for this choice; a short ablation would improve reproducibility and confidence in the robustness of the content loss.","section":"Sec. 4.3 and Supp. Sec. 9.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is empirically strong and the method is simple and plausible. The central issue is a missing control experiment that isolates the Q-Q attention change from the decoupled distillation; this is easily fixable and would substantially strengthen the paper. I would be comfortable with acceptance after that ablation and after the inference-path clarification are provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful paper in the CLIP pre-fine-tuning line. The decoupling of content and context features with two separate teachers—self-distillation for content, VFM correlation distillation for context—is genuinely new relative to CLIPSelf, ClearCLIP, SCLIP, and CLIP-DINOiser. It is a clean idea, and the experiments are extensive: OV-COCO/OV-LVIS detection, several segmentation benchmarks, and region classification. The gains are consistent and often large, including on the CAT-Seg backbone, which makes the empirical claim credible. The supplementary also does the right sanity checks to show the improvement is not simply an EVA-CLIP artifact—EVA-CLIP alone is worse than OpenAI CLIP in their CAT-Seg comparison, and DeCLIP still wins.\n\nThe main soft spot is exactly the one the stress-test flags. Table 1's 'Self+VFM+Decouple' row differs from 'Self+VFM Distillation' in two ways at once: it switches the attention to Q-Q^T and it decouples the features into content/context with separate losses. Because SCLIP and ClearCLIP already show the Q-Q switch alone helps dense features, the +5.5/+6.7 gains attributed to decoupling are partly confounded. This is not a fatal flaw, but it is a real identification problem for the paper's central claim. The fix is cheap: run the same decoupled Q-Q attention with the context loss set to zero and compare. The paper doesn't report that, and it should. The proxy-token motivation is also shown for OpenAI CLIP while training uses EVA-CLIP; the supplementary Tables 9-10 show EVA-CLIP raw features behave differently, but no EVA-CLIP attention maps are given. This weakens the causal story, though not the empirical recipe.\n\nMinor quibbles: the region-classification numbers in Table 1 are at different resolutions than the ablation tables, which makes cross-table reading slightly annoying. The F-ViT/OV-DQUO temperature changes are explained, so no issue there.\n\nBottom line: this is a paper for people working on open-vocabulary dense prediction. It deserves serious peer review; a competent reviewer should push for the missing ablation and EVA-CLIP attention analysis. With those, the decoupling claim would be solid. I'd cite it and use it as a baseline.","headline":"DeCLIP's decoupled-distillation recipe is worth taking seriously, and the benchmark results are consistently strong; the main caveat is that the key Table 1 comparison doesn't isolate Q-Q attention from decoupling.","tokens_in":27617,"tokens_out":2071,"would_cite":true,"duration_ms":19370,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeCLIP shows that decoupling CLIP's self-attention into content and context streams fixes its dense-prediction weakness, giving state-of-the-art open-vocabulary detection and segmentation.","keywords":["open-vocabulary detection","open-vocabulary semantic segmentation","CLIP fine-tuning","decoupled attention","self-distillation","vision foundation models","dense prediction","proxy token phenomenon"],"falsifier":"Run DeCLIP's two distillation losses on EVA-CLIP while keeping the original Q-K attention in the last block; if most of the reported mIoU and mAP gains disappear, the Q-Q attention replacement rather than the decoupled distillation is the active ingredient. Also check whether EVA-CLIP's attention maps show the proxy-token pattern before fine-tuning; if they do not, the causal story needs revision.","tokens_in":1953,"feed_emoji":"👁️","tokens_out":2159,"duration_ms":98406,"temperature":0.7,"pith_summary":"This paper argues that CLIP's weak performance on dense visual tasks has a specific cause: in deep layers, CLIP's image tokens stop attending to their semantic neighbors and cluster onto a few 'proxy' background tokens, so the resulting features lack local discriminability and spatial consistency. To fix this, DeCLIP splits the last self-attention block into two parallel feature streams: 'content' features, trained by aligning pooled region features with image-crop [CLS] representations, and 'context' features, trained to reproduce the token-correlation structure of a vision foundation model such as DINOv2. The decoupling is key: doing both distillations on the same features hurts region classification, while separating them improves both tasks. The paper shows large gains on open-vocabulary object detection and semantic segmentation benchmarks, including new state-of-the-art results when DeCLIP is used as the backbone of CAT-Seg. The result is a general unsupervised pre-fine-tuning recipe: upgrading CLIP's local features for dense perception without any dense labels.","feed_headline":"Decoupling CLIP's attention beats prior open-vocabulary models","feed_subtitle":"Splitting features into content and context improves detection and segmentation without dense labels.","key_machinery":"The load-bearing object is the decoupled attention module at CLIP's last block, which splits the representation into context features X_context = Proj_q(X) and content features X_content = Proj(Attn_context V), where Attn_context = SoftMax(X_context X_context^T / $\\sqrt$(d)). This separates the query side (which tokens relate to which) from the value side (what semantic content each token carries), so two teachers can act without conflict: a cosine-similarity loss pulls pooled content-region features toward CLIP's own crop-level [CLS] embeddings, and an L2 loss pulls the pairwise cosine correlation matrix of the context tokens toward that of a vision foundation model such as DINOv2. The total objective is L_total = L_content + $\\lambda$ * L_context with $\\lambda$ = 0.25.","core_discovery":"On the paper's own terms, the central discovery is that CLIP's dense features are corrupted by a 'proxy token' phenomenon: starting around layer 7, both the [CLS] token and ordinary image tokens concentrate their attention on a few background tokens, no matter where the query token lies, so tokens from the same object fail to correlate. The paper claims this is why CLIP transfers poorly to detection and segmentation, and that the fix is to decouple the final self-attention into a context stream, which defines spatial and semantic relations, and a content stream, which carries visual-language semantics. The content stream is disciplined by self-distillation, pulling pooled region features toward CLIP's own crop-level [CLS] embeddings, while the context stream is disciplined by matching its pairwise token correlation volume to that of a vision foundation model such as DINOv2. Trained this way on COCO images alone, DeCLIP improves region classification at every resolution tested and lifts open-vocabulary detection and segmentation across multiple benchmarks.","pith_inferences":["The paper does not isolate the Q-Q attention replacement from the decoupled distillation; because the training-free Q-Q trick is known to improve dense CLIP features on its own, part of DeCLIP's gain may come from that trick rather than from the decoupled losses. A control experiment with standard Q-K attention and the same two losses would settle this.","The proxy-token diagnosis is demonstrated on one CLIP variant, while DeCLIP is fine-tuned on EVA-CLIP; whether EVA-CLIP shows the same proxy-token pattern is not shown, so the causal story may be incomplete.","If the decoupling is the key, the recipe should transfer to other CLIP-style models and other context teachers; using a boundary-sensitive teacher such as SAM could sharpen instance edges in segmentation, a natural extension the paper does not test.","The content loss relies on CLIP's own crop-level [CLS] as teacher, so DeCLIP's ceiling is partly set by CLIP's image-level accuracy; extending content distillation to a stronger text-grounded teacher might push region classification further."],"forward_implications":["On OV-COCO, DeCLIP raises novel-class AP of F-ViT from 37.6 to 41.1 with ViT-B/16 and of OV-DQUO from 39.2 to 46.1, with similar gains on OV-LVIS rare classes.","As a drop-in backbone for CAT-Seg, DeCLIP with ViT-B/16 nearly matches prior state-of-the-art results that use much larger encoders such as ConvNeXt-L, and DeCLIP with ViT-L/14 sets new state-of-the-art numbers on open-vocabulary semantic segmentation.","On training-free VLM-feature segmentation, DeCLIP averages 41.9 mIoU across eight benchmarks, outperforming existing training-free methods and improving on every individual benchmark.","Decoupling the distillation matters: combining self-distillation and VFM distillation without decoupling hurts region classification (COCO Thing mAcc drops 3.9), while the decoupled version improves both region classification and segmentation.","The recipe is teacher-agnostic: DINO, SAM, and DINOv2 all work as context teachers, with DINOv2 giving the best balance between region classification and segmentation."],"supporting_citations":[{"why":"Supplies CLIP, the vision-language model whose dense features DeCLIP diagnoses and fine-tunes, including the [CLS] tokens used as the content teacher.","marker":"[52]"},{"why":"Provides CLIPSelf, the self-distillation pre-fine-tuning baseline and the F-ViT detection baseline that DeCLIP improves upon.","marker":"[68]"},{"why":"Provides DINOv2, the default vision foundation model whose token correlations supervise the context stream.","marker":"[51]"},{"why":"Shows that Q-Q attention improves CLIP's dense features in a training-free way, motivating the decoupled attention design.","marker":"[63]"},{"why":"Provides ClearCLIP, another training-free Q-Q attention analysis that supports the choice to decouple self-attention.","marker":"[38]"},{"why":"Supplies CAT-Seg, the segmentation model used to evaluate DeCLIP as a backbone, yielding the state-of-the-art segmentation results.","marker":"[14]"},{"why":"Supplies OV-DQUO, the DETR-based open-vocabulary detector that DeCLIP boosts as a backbone.","marker":"[65]"},{"why":"Supplies EVA-CLIP, the CLIP variant actually used as the DeCLIP backbone, chosen for its masked-image-modeling-initialized visual encoder.","marker":"[61]"},{"why":"Cited to support the proxy-token explanation of CLIP's attention behavior.","marker":"[16]"}],"fun_headline_variants":["DeCLIP fixes CLIP's proxy-token blindness for dense tasks","Decoupling CLIP attention into content and context boosts detection","Splitting CLIP's self-attention cures dense feature confusion","DeCLIP: separate content and context to see open-vocabulary objects","Curing CLIP's proxy tokens: DeCLIP improves dense prediction"],"cache_read_input_tokens":29696,"weakest_assumption_plain":"The load-bearing premise is that the proxy-token attention pattern is the root cause of CLIP's poor dense prediction, and that the decoupled Q-Q attention plus distillation repairs that cause; the paper does not isolate the Q-Q change from the known training-free trick, and the attention analysis was run on a different CLIP model than the one it fine-tunes.","fun_headline_variants_meta":{"raw":{"variants":["DeCLIP fixes CLIP's proxy-token blindness for dense tasks","Decoupling CLIP attention into content and context boosts detection","Splitting CLIP's self-attention cures dense feature confusion","DeCLIP: separate content and context to see open-vocabulary objects","Curing CLIP's proxy tokens: DeCLIP improves dense prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3419,"prompt_tokens":960,"completion_tokens":2459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2375}},"tokens_in":576,"tokens_out":2459,"duration_ms":18460,"temperature":1.0,"reasoning_tokens":2375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:29:29.076911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DeCLIP's two distillation losses on EVA-CLIP while keeping the original Q-K attention in the last block; if most of the reported mIoU and mAP gains disappear, the Q-Q attention replacement rather than the decoupled distillation is the active ingredient. Also check whether EVA-CLIP's attention maps show the proxy-token pattern before fine-tuning; if they do not, the causal story needs revision.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies CLIP, the vision-language model whose dense features DeCLIP diagnoses and fine-tunes, including the [CLS] tokens used as the content teacher."},{"cited_title":"CLIPSelf: Vision transformer distills itself for open-vocabulary dense predic- tion","cited_arxiv_id":null,"evidence_quote":"Provides CLIPSelf, the self-distillation pre-fine-tuning baseline and the F-ViT detection baseline that DeCLIP improves upon."},{"cited_title":"OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision","cited_arxiv_id":"2405.17913","evidence_quote":"Supplies OV-DQUO, the DETR-based open-vocabulary detector that DeCLIP boosts as a backbone."}],"review_version":1}