{"id":"08387a02-65d8-40bf-85a6-ff28c353533c","arxiv_id":"2507.06848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A weakly supervised segmentation method uses class-specific CLS tokens and random token masking to turn ViT attention maps into pseudo-masks.","lead":"This paper trains a vision transformer with one summary token per class and randomly hides some of those tokens during training so each token's attention map highlights only its own class. The attention maps are used directly as pseudo-segmentation masks, and the method reports top results on several weakly supervised segmentation benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The masking mechanism is applied to final [CLS] embeddings after attention, not to attention itself, and only during training; nothing in the paper establishes that the resulting attention maps are class-specific, so the pseudo-mask gains in Table 4 are not yet attributable to the claimed…","rationale":"The paper's central claim is that multiple [CLS] tokens combined with random masking and head pruning produce class-specific attention maps, and that these maps yield high-quality pseudo-masks without CAMs or refinement. For that claim to hold, the random masking must actually constrain each [CLS] token's attention to its assigned class. As written in Section 3.3, the masking operates only on the final output embedding after all attention layers and only for a random subset of absent-class tokens during training. Present-class tokens are never masked, and at inference nothing is masked. The mechanism therefore does not directly enforce the required property in the attention maps themselves, and the train/inference discrepancy provides a possible shortcut for the classifier that would not be available at inference. The Table 4 ablation is useful evidence that masking helps, but it reports only end-to-end pseudo-mask mIoU; it does not isolate whether the improvement comes from class-specific token attention or from other effects such as better classification of which classes are present. A direct per-token attention-to-ground-truth IoU measurement on the released checkpoints would settle this question. Because the reader already issued a CONDITIONAL verdict centered on exactly this uncertainty, and because code is promised and the ablation does provide some empirical support, I do not recommend moving the verdict; I recommend keeping it CONDITIONAL until the diagnostic evidence is provided.","tokens_in":11051,"tokens_out":9712,"duration_ms":118425,"concrete_test":"Use the released checkpoint to compute, on VOC val, the per-class IoU between each [CLS]_c attention map and the ground-truth mask for class c, separately for images where c is present and where c is absent, for the with-masking and without-masking checkpoints. If random masking does not raise present-class per-token attention IoU substantially over the no-masking baseline, or if absent-class tokens have comparable foreground overlap, the pseudo-mask improvements in Table 4 cannot be attributed to class-specific token assignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 masks only the final output embedding z[CLS]^i_L = z[CLS]^i_L · (1 − m(i)) for a random 50% of absent-class tokens. This happens after all self-attention layers, and no masking is applied at inference. Consequently, during training a masked absent-class token still participates in every attention computation; the only supervised signal is that its final embedding is zero before the classification loss. For classes that are actually present in an image, the token is never masked, so the training signal never forces the token's attention maps to localize that class. The mechanism therefore does not, as claimed, promote [CLS] token-class assignment in the attention maps that are later used to build pseudo-masks. It also creates a train/inference discrepancy: a zeroed absent-token embedding is a reliable absent cue during training, but at inference no token is zeroed, so absent tokens produce nonzero embeddings the model was not consistently trained to interpret. The paper presents no analysis showing the assignment survives, and the Table 4 ablation reports only final pseudo-mask mIoU, not per-token attention alignment with ground-truth classes. If the classifier or the [REG] token supplies the class information and the attention maps are not class-specific, the headline result would not validate the proposed mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end weakly supervised semantic segmentation (WSSS) method that uses a Vision Transformer with multiple [CLS] tokens, one per class. A random masking strategy zeroes the final output embeddings of a subset of absent-class [CLS] tokens during training, a register token is appended, and attention heads are pruned during training via a hard-concrete relaxation. At inference, the self-attention maps of the [CLS] tokens corresponding to predicted classes are thresholded and combined to form pseudo-masks, which are then used to train a UNet. Experiments on VOC, MS COCO, DFC2020, EndoTect, and ADE20K report strong pseudo-mask mIoU (73.7% on VOC val) and competitive or state-of-the-art results among the compared WSSS baselines.","tokens_in":11351,"tokens_out":6488,"duration_ms":72255,"significance":"If the central mechanism were validated, the paper would make a useful contribution: it would show that a single-stage ViT without CAMs or iterative refinement can generate high-quality pseudo-masks directly from attention maps, with interpretable class-token assignments. The paper covers multiple domain-specific datasets, provides ablations for each component, includes a sensitivity analysis of the masking ratio, and makes code available. However, the claimed causal mechanism for random masking is not directly supported by the evidence, and the fully-supervised parity claim holds for only one of the three specialized datasets. The reported improvements are also not accompanied by variance or significance measures, which matters for the small margins over prior work.","major_comments":[{"comment":"The masking is applied as z[CLS]_i^L = z[CLS]_i^L · (1 − m(i)), i.e., to the final output embedding after all self-attention layers. During training, a masked absent-class token still participates in every attention computation, and at inference no token is masked at all. Therefore, the mechanism as described cannot directly enforce the claimed hard assignment between a [CLS] token and its class in the attention maps that are later thresholded into pseudo-masks. It also introduces a train/inference discrepancy: zeroed absent tokens are a reliable absent cue during training, but no such cue is present at inference. Table 4 reports only final pseudo-mask mIoU and does not measure per-token attention alignment with ground-truth classes, so the ablation does not establish the proposed causal mechanism. Please provide quantitative attention-class alignment evidence (for example, per-class IoU of the [CLS] attention maps before binarization) or an experiment that isolates the effect of masking on the attention maps themselves.","section":"Section 3.3"},{"comment":"The claim that the method 'achieves results comparable to fully-supervised models' is contradicted by two of the three specialized datasets in Table 3. On ADE20K the method reaches 38.2 mIoU versus 55.0 for the supervised UNet, and on EndoTect it reaches 69.8 versus 73.0. Only on DFC2020 does it surpass the supervised UNet (67.2 vs 53.1). The claim should be restricted to the datasets where it actually holds, and the gap on ADE20K and EndoTect should be analyzed rather than presented as general parity.","section":"Abstract and Section 5.4, Table 3"},{"comment":"All reported results are single runs without error bars or significance tests. The headline VOC validation comparison is 73.7% vs 73.5% for DuPL, a margin of 0.2 percentage points that could easily arise from training stochasticity. In addition, the 50% masking ratio is selected from the sensitivity analysis in Section 5.6 on the same datasets where the final numbers are reported. Please provide multiple seeds with mean and standard deviation, and ideally perform hyperparameter selection on a validation split that is distinct from the split used for the reported final results.","section":"Tables 1–3 and Section 5.6"},{"comment":"The caption of Table 4 states that random masking gives 'considerable improvement on all datasets,' but the table only lists MS COCO and VOC; the ADE20K evidence is qualitative (Figure 5). Quantitative ablations on the specialized datasets are needed to support the generalization claim, especially because the masking mechanism is the paper's main novelty and the mechanism's effect on attention maps is not measured.","section":"Section 5.5, Table 4"}],"minor_comments":[{"comment":"The symbol z0 is used both for the sequence of image patch embeddings and for the full sequence after prepending the [CLS] token; please use distinct notation to avoid confusion.","section":"Section 3.1"},{"comment":"References [35] and [36] are the same paper (Zhou et al., 'Learning Deep Features for Discriminative Localization'); please merge them or cite a single version.","section":"References"},{"comment":"The table reports a 'train' column for pseudo-mask evaluation on the VOC train set; please clarify what train-time evaluation means in this context, since pseudo-masks are generated without training on segmentation labels.","section":"Section 5.2, Table 1"},{"comment":"The sensitivity analysis is described as covering masking ratios 0%, 20%, 50%, 80%, and 100%, but the text does not explain how the curves in Figure 6 map to datasets and metrics; please make the figure self-explanatory.","section":"Section 5.6, Figure 6"},{"comment":"The dataset name is written inconsistently as 'Endotect' and 'EndoTect'; please standardize it.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper builds directly on the authors' prior work [9], and the novelty relative to that work should be stated more crisply. The central issue is that the masking mechanism, as described in Section 3.3, operates on final output embeddings rather than on attention maps, so the paper's main explanatory claim is not yet supported. I recommend major revision: the experimental pipeline could be salvageable with additional attention-alignment analysis and more rigorous reporting, but if the authors cannot provide such evidence, the contribution should be reframed as an empirical regularizer rather than as a mechanism that guarantees class-specific attention maps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know up front. The method is a genuine single-stage WSSS contribution: multiple [CLS] tokens (one per class), random masking of absent-class output embeddings, a register token, and attention head pruning. It reports 73.7 mIoU on Pascal VOC pseudo-masks and beats the plain supervised UNet baseline on DFC2020. That is worth a look.\n\nWhat is genuinely new is the combination. The masking is applied only to the final token embedding, after all attention layers. So the paper's claim that this enforces class-specific attention maps is not directly supported. The ablation shows masking helps end-to-end pseudo-mask mIoU, but that could be because classification improves and therefore the right tokens are selected at inference, not because each token's attention map became class-specific. There is no per-token attention alignment analysis. The train/inference discrepancy is real but probably not fatal, since at inference only tokens for predicted classes are used, and those are never masked when the class is present. Still, I'd want to see attention maps before and after masking aligned to ground-truth masks to believe the mechanistic story.\n\nThe fully-supervised parity claim is overbroad. On EndoTect the method is below the supervised UNet, and on ADE20K it is well below it. It only surpasses supervised baselines on DFC2020, and those baselines are weak (a plain UNet). So \"comparable to fully-supervised\" should read \"competitive with weak fully-supervised baselines on some domains.\"\n\nOther soft spots: no error bars anywhere; the masking ratio is tuned on the same benchmarks where the final numbers are reported; the threshold for binarizing attention maps and the neighbor-filling rule are not specified in enough detail for reproduction. The code is public, which helps.\n\nOverall, the method is plausible, the experiments are broad, and the paper is clearly written. I would send it to a serious referee, but I would ask for a rigorous analysis of the attention maps and a careful rewrite of the claims.\n\nMy bottom line: engage with it, but make them prove the mechanism.","headline":"A real single-stage WSSS idea with strong scores on VOC and DFC2020, but the masking mechanism's role in class-specific attention is unproven and the fully-supervised parity claim overreaches.","tokens_in":11838,"tokens_out":3406,"would_cite":false,"duration_ms":36934,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Vision Transformer with one [CLS] token per class, trained with random token masking and attention-head pruning, can turn image-level labels into segmentation pseudo-masks that rival fully supervised models.","keywords":["weakly supervised semantic segmentation","vision transformer","class tokens","attention maps","pseudo-mask generation","token masking","attention head pruning","image-level labels"],"falsifier":"Take the trained model, run inference without any masking, and for every [CLS] token record which ground-truth class's pixels its binarized attention map overlaps most. If the same token index does not predominantly correspond to the same class across images, the claimed hard token-class assignment is not actually learned, and the pseudo-masks could be right for the wrong reason.","tokens_in":10829,"feed_emoji":"🎯","tokens_out":8961,"duration_ms":97693,"temperature":0.7,"pith_summary":"This paper is trying to establish that a Vision Transformer can do weakly supervised semantic segmentation directly from its self-attention maps, without class activation maps, external refinement, or iterative multi-stage pipelines. The recipe is to give the model one [CLS] token per semantic class, randomly mask half of the tokens whose classes are absent from the training image, and prune redundant attention heads with a stochastic relaxation of $L_0$ regularization. At inference, the attention maps of the tokens corresponding to predicted labels are thresholded and stitched into pseudo-masks, which then train a standard UNet. The paper reports that these pseudo-masks reach 73.7% mIoU on Pascal VOC validation and that the final segmentation model reaches 73.5% mIoU on VOC test, with strong results on COCO and three domain-specific datasets. If true, this would mean that fine-grained pixel labels are largely unnecessary for good segmentation, since image-level labels plus attention structure can get close to fully supervised performance.","feed_headline":"Class-specific ViT tokens beat prior weakly supervised pipelines","feed_subtitle":"Masking absent-class tokens turns attention maps into 73.7% mIoU pseudo-masks on VOC val.","key_machinery":"The load-bearing mechanism is the class-specific [CLS] token combined with random output-embedding masking. In a Vision Transformer, each [CLS] token's self-attention over image patches defines a spatial distribution; with C tokens, attention head h for token c gives a class-specific map, and random masking of absent-class tokens during training removes the gradient signal that would let a token hedge across multiple classes. The auxiliary machinery is a register token, which absorbs generic background context that would otherwise pollute the class tokens, and attention-head pruning via Hard Concrete gates, which drives roughly two-thirds of heads to zero so the surviving heads produce cleaner, less fragmented maps. At inference, the maps are thresholded and fused in order of decreasing logit, with unassigned pixels filled from neighboring values.","core_discovery":"The central claim is that the self-attention maps of a Vision Transformer can be made class-specific by construction, rather than being post-hoc explanations. The authors extend the standard single [CLS] token to C class tokens, add a register token to absorb global context, and during training randomly set to zero the final embeddings of half of the absent-class tokens. This forces each token to specialize on its own class while preventing interference from classes not present in the image. Simultaneously, a Hard Concrete relaxation of $L_0$ regularization prunes roughly two-thirds of attention heads to remove noisy heads and sharpen the remaining maps. At inference, the binarized self-attention maps of the [CLS] tokens for predicted labels are aggregated, and the paper reports pseudo-mask mIoUs of 73.7% on VOC validation and 43.2% on COCO validation, with the resulting UNet reaching 73.5% mIoU on VOC test. On the DFC2020 remote sensing dataset, the weak supervision pipeline even outperforms fully supervised baselines. The paper's message is that attention maps themselves, if trained correctly, are not just interpretable artifacts but usable segmentation signal.","pith_inferences":["A testable extension is a curriculum over masking ratios: start with heavy masking and anneal it down; the sensitivity curve in the paper already shows a peak near 50%, which suggests the amount of masking matters as much as its presence.","If the token-class assignment is truly hard and stable, the same [CLS] tokens could be reused as pseudo-labels for other tasks such as open-vocabulary segmentation, detection, or self-training, since the paper only evaluates them through a UNet.","The random masking may be doing its work by breaking correlations between co-occurring classes; if so, class-balanced sampling or negative-class weighting could achieve the same effect with less compute, an alternative the paper does not test.","The large gap on ADE20K (38.2% mIoU versus 55.0% for supervised UNet) suggests dense scenes with many labels are the stress test; combining these attention pseudo-masks with pixel-affinity refinement, which the paper explicitly avoids, is an obvious next step."],"forward_implications":["Single-stage weakly supervised segmentation can match or beat multi-stage CAM-based pipelines on standard benchmarks, removing the need for CRF post-processing or iterative refinement.","A segmentation model trained on these pseudo-masks comes close to fully supervised accuracy on several datasets, so annotation effort can shift from pixel-level to image-level labels.","The method transfers to remote sensing, endoscopy, and general scene parsing, suggesting the mechanism is not tied to one domain.","Attention-head pruning is not just a computational saving but part of what makes the pseudo-masks accurate, since without it the masks are noisier and mIoU drops.","The parameter count grows with the number of classes because each class gets a token, so the approach has a scalability cost on label sets with hundreds or thousands of classes."],"supporting_citations":[{"why":"Supplies the Vision Transformer backbone whose attention maps are repurposed for segmentation.","marker":"[7]"},{"why":"Defines Class Activation Maps, the standard WSSS localization tool that this method replaces.","marker":"[36]"},{"why":"Establishes the direct use of sparse ViT attention maps for WSSS and the head-sparsity idea this work extends.","marker":"[9]"},{"why":"Introduces multi-class token transformers, the closest architectural precedent for class-specific tokens.","marker":"[33]"},{"why":"Provides the previous best single-stage pseudo-mask baseline that the reported 73.7% mIoU is compared against.","marker":"[32]"},{"why":"Shows that an extra register token removes attention artifacts, motivating the register token used here.","marker":"[6]"},{"why":"Provides the Hard Concrete relaxation used to approximate $L_0$ regularization for attention-head pruning.","marker":"[18]"},{"why":"Defines the UNet architecture trained on the pseudo-masks for final segmentation evaluation.","marker":"[20]"}],"fun_headline_variants":["Class tokens turn ViT attention into segmentation masks","Sparse ViT with class tokens yields 73.7% mIoU pseudo-masks","One CLS token per class: attention maps become segmentation","Masking absent-class tokens sharpens attention for WSSS","End-to-end WSSS via class-specific attention maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that randomly zeroing out half of the class tokens for classes not present in the image during training is enough to make each token lock onto a single class, and that this lock survives at inference when no tokens are masked.","fun_headline_variants_meta":{"raw":{"variants":["Class tokens turn ViT attention into segmentation masks","Sparse ViT with class tokens yields 73.7% mIoU pseudo-masks","One CLS token per class: attention maps become segmentation","Masking absent-class tokens sharpens attention for WSSS","End-to-end WSSS via class-specific attention maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1676,"prompt_tokens":977,"completion_tokens":699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":593,"tokens_out":699,"duration_ms":7059,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:52:28.086120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained model, run inference without any masking, and for every [CLS] token record which ground-truth class's pixels its binarized attention map overlaps most. If the same token index does not predominantly correspond to the same class across images, the claimed hard token-class assignment is not actually learned, and the pseudo-masks could be right for the wrong reason.","supporting_citations":[{"cited_title":"Learning deep features for discrim- inative localization","cited_arxiv_id":null,"evidence_quote":"Defines Class Activation Maps, the standard WSSS localization tool that this method replaces."},{"cited_title":"Sparse multimodal vision transformer for weakly supervised seman- tic segmentation","cited_arxiv_id":null,"evidence_quote":"Establishes the direct use of sparse ViT attention maps for WSSS and the head-sparsity idea this work extends."},{"cited_title":"Multi-class token transformer for weakly super- vised semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Introduces multi-class token transformers, the closest architectural precedent for class-specific tokens."},{"cited_title":"Dupl: Dual student with trustworthy progressive learning for robust weakly supervised semantic segmenta- tion","cited_arxiv_id":null,"evidence_quote":"Provides the previous best single-stage pseudo-mask baseline that the reported 73.7% mIoU is compared against."}],"review_version":1}