{"id":"1e851822-a6e5-46be-9253-a9289b25131f","arxiv_id":"2505.12803","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GradMix masks the most attribution-activated image regions during training, pushing the model to learn additional features and improving open set recognition and robustness.","lead":"This paper introduces GradMix, a training-time augmentation that hides the image regions a model most relies on, forcing it to learn other features. It reports gains in open set recognition, out-of-distribution detection, and robustness, though code and error bars are not yet available.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main results omit the no-GradMix baseline under the same detector; the claimed OSR gain rests on a cross-table comparison with different scoring rules.","rationale":"I read the paper as claiming two things: (1) combining SupCon with SimCLR helps OSR, and (2) adding GradMix to that combination further improves OSR by masking learned concepts. Claim (1) is supported by Table 1 and prior work. Claim (2) is the novel contribution, and it is exactly where the evidence is weakest. Table 2 compares GradMix against external OSR methods, not against the no-GradMix backbone under the same evaluation rule. The paper does include an ablation ('None' vs GradMix in Figure 4), but only on two of six protocols, without error bars, and with k set to 10 while the main table uses k=3. The cross-table comparison with Table 1 is invalid because the detection scores differ (Mahalanobis-to-mean vs cosine top-k). Since the OSR score function is part of the method, the reader cannot tell whether GradMix, the new detector, or the hyperparameters produce the gains. This is more load-bearing than the attribution-quality concern: even if LayerCAM perfectly identifies learned regions, the paper still needs a same-protocol baseline to demonstrate the benefit. The proposed check, running with and without GradMix under identical settings across all six protocols, directly settles the incremental claim. The method is plausible and the controlled evidence on two datasets is encouraging, so conditional acceptance is appropriate, but the central claim needs this direct comparison before it can be assessed.","tokens_in":18078,"tokens_out":10867,"duration_ms":108109,"concrete_test":"Run the exact same training pipeline (ResNet18, SupCon+SimCLR, same epochs, batch size, optimizer, learning-rate schedule, same mask-size distribution, and same layer aggregation) with and without GradMix on all six OSR protocols, evaluating both with the same Algorithm 1 detector using the same k, and report mean +/- std over at least five seeds. If the with-GradMix AUROC does not exceed the without-GradMix AUROC by more than the seed noise on a majority of protocols, the central claim is unsupported. As a secondary check, confirm Table 8's gamma and lambda values against Section 3.2 and Appendix F, and report which value of k produced the main-table numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central incremental claim is that GradMix improves OSR over the SupCon+SimCLR backbone. This claim is not tested in the main results. Table 2 contains no 'SupCon+SimCLR without GradMix' row; the only numerical support for the increment is a cross-table comparison between SupCon+SSL in Table 1 (AUROC 88.75 on CIFAR10, 77.48 on TinyImageNet) and GradMix in Table 2 (91.33, 81.92). Those numbers are not directly comparable: Table 1's Anno score is a Mahalanobis-distance-to-nearest-class-mean version of NNO (Appendix A), whereas Table 2 uses the new Algorithm 1 with cosine similarities to the top-k neighbors and a normalized max score (Section 3.3). A change of detection rule alone can shift AUROC by several points, so the observed gap cannot be attributed to GradMix. The only controlled same-detector comparisons are in Figure 4, and they cover just CIFAR10 and TinyImageNet, use k=10 while Table 8 reports k=3 for the main runs, and report no error bars. Hyperparameter reporting is also internally inconsistent: Table 8 lists gamma=1.0 although Section 3.2 defines gamma ~ U(0.1,0.5), and the lambda values for CIFAR10 and TinyImageNet are swapped between Table 8 and Appendix F. These omissions make it impossible to determine whether the reported OSR gains come from the attribution-guided masks, from the new detection rule, or from hyperparameter choices.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes GradMix, a training-time augmentation method for open set recognition. GradMix uses LayerCAM attribution maps computed from the current model and current contrastive loss to locate the most strongly activated image region, masks that region, and replaces it with a patch from another sample in the minibatch; the augmented SimCLR view is weighted by the masked area fraction in the total loss. The method is combined with SupCon and SimCLR and evaluated with a new distance-based OSR score based on cosine similarities to the top-k neighbors of each class. Across six OSR protocols, the paper reports state-of-the-art AUROC on four of them, with a gain of just over 1% on TinyImageNet, and further reports gains in corruption robustness and in SSL linear-probe accuracy across several architectures and datasets.","tokens_in":18444,"tokens_out":6854,"duration_ms":70196,"significance":"If the central increment were established under controlled comparison, GradMix would be a useful and simple augmentation for OSR: it requires no extra supervision, it fits naturally into contrastive training, and the reported improvements in corruption robustness and downstream linear probing suggest a broader generalization benefit. The paper's strengths are the breadth of OSR protocols, the inclusion of several common augmentation baselines, and the evaluation across SimCLR and MoCo v1 with ResNet18, ResNet34, and ResNet50. However, the central claim is not yet demonstrated because the headline OSR comparison mixes a change of detector with the change of augmentation, the only controlled ablation is reported without variance and with a different k than the main protocol, and the hyperparameter documentation is internally inconsistent.","major_comments":[{"comment":"The paper's central incremental claim is that GradMix improves OSR over the SupCon+SSL backbone, but no same-detector no-GradMix baseline appears in the main results. The GradMix numbers in Table 2 use the new Algorithm 1 score (cosine similarities to the top-k neighbors with a normalized maximum), whereas the SupCon+SSL numbers in Table 1 use the Mahalanobis-distance NNO score described in Appendix A. A change of scoring rule alone can shift AUROC by several points, so the observed gaps (e.g., 77.48 vs. 81.92 on TinyImageNet) cannot be attributed to GradMix. Please add the SupCon+SSL model without GradMix evaluated with Algorithm 1 under the same k, gamma, lambda, and training schedule as the GradMix runs.","section":"Table 2 vs. Table 1, Section 3.3"},{"comment":"Figure 4 (left) is the only same-detector controlled comparison in the paper, but it reports no standard deviations even though Section 4.1 states that all results are averages of five trials, and it fixes k=10 whereas Table 8 reports k=3 for the main OSR protocols. Without error bars and with a different k than the headline experiments, the claimed 'over 3%' gain of GradMix over no augmentation cannot be assessed, and the figure is not directly aligned with the protocol used for Table 2. Please report mean and standard deviation for the five trials and run the augmentation ablation with the same k and loss hyperparameters as the main results.","section":"Figure 4, Section 4.1"},{"comment":"The hyperparameter documentation is internally inconsistent and prevents reproduction. Section 3.2 defines gamma as the mask side-length ratio with gamma ~ U(0.1, 0.5) and Eq. (8) weights the GradMix loss by gamma^2, but Table 8 lists gamma=1.0 for every protocol. In addition, the lambda values for CIFAR10 and TinyImageNet are reversed between Table 8 and Appendix F: Table 8 gives CIFAR10 lambda=1.2 and TinyImageNet lambda=1.0, while Appendix F states the opposite. Please provide a single consistent hyperparameter table and clarify whether the gamma column is the mask ratio, the SupCon loss weight theta, or a separate quantity.","section":"Section 3.2, Table 8, Appendix F"},{"comment":"The layer-aggregation choice is selected on the test OSR AUROC of the same protocols that are later reported as final results, and the supporting SM analysis in Section 5 counts high-valued pixels in the very LayerCAM maps used to create the GradMix augmentations. Both practices make the evidence for the proposed mechanism partly self-fulfilling. A hold-out validation split for selecting the layer set, together with an independent measure of learned feature breadth (e.g., segmentation overlap or transfer to an unrelated task), would be needed to support the claim that GradMix encourages broader feature learning.","section":"Section 4.1, Section 5, Figure 4 (right)"}],"minor_comments":[{"comment":"There are several typos: 'coindice' should be 'coincides' in Section 1, and Section 5 refers to 'LayerGAM' instead of 'LayerCAM'.","section":"Section 1, Section 5"},{"comment":"The similarity computation uses sim(ztest, Zc), but the loop variable is zi; this should be sim(zi, Zc) for consistency.","section":"Algorithm 1, line 8"},{"comment":"The sentence 'Except for MNIST and SVHN, we aggregate layer conv3_2, conv4_2, conv5_2' conflicts with Table 8, which lists MNIST and SVHN among the protocols with reported hyperparameters; please clarify whether GradMix is used for MNIST and SVHN and which layers are aggregated for those datasets.","section":"Appendix D.2"},{"comment":"The Attentive CutMix comparison uses an ImageNet-pretrained ResNet18 to compute attention maps, and the text notes this gives an unfair advantage; please state explicitly how the semantic overlap between ImageNet and CIFAR/TinyImageNet was handled or restrict the comparison to baselines that do not use pretrained feature extractors.","section":"Appendix B, Figure 9"},{"comment":"Please define the overbar notation for average accuracy drop consistently and state whether the reported differences, such as 18.08 vs. 15.97 on CIFAR10, are averaged over multiple training trials or a single run.","section":"Section 4.4, Table 5"},{"comment":"The paper states that code will be published upon acceptance, but no code or standard deviations are available in the current version; given the hyperparameter inconsistencies, releasing the code or at least the exact per-protocol settings would be important for reproducibility.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a preliminary arXiv submission with no code and no standard deviations. The main technical concern is that the headline OSR gain is not measured under a controlled same-detector comparison: the cross-table comparison mixes a change of scoring rule with the change of augmentation. If the authors supply the missing no-GradMix baseline under Algorithm 1 and resolve the hyperparameter inconsistencies, the contribution could become publishable; otherwise the central claim remains unsupported. I also note that the layer aggregation choice and the SM analysis are selected on the same test data, which should be addressed with a validation split and an independent breadth measure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou asked about arXiv:2505.12803. The short version: there is a genuinely new augmentation idea here, and some solid evidence it helps robustness and self-supervised learning, but the headline claim—that GradMix improves open set recognition—is weaker than the abstract suggests. The main OSR table compares numbers from other papers using a different detection score than the one used for the GradMix row, so the reported gain over the SupCon+SimCLR backbone is not actually shown.\n\nWhat is new: GradMix masks the most activated region in the image, using online LayerCAM from the current model, and weights the SimCLR loss by the masked area. That is distinct from Attentive CutMix (which patches salient regions from a pretrained model) and is reasonably motivated by the idea of forcing the model to learn features outside its current focus. The controlled experiments in Figure 4, with the same detection rule and same backbone, do show GradMix ahead of no augmentation, Mixup, and CutMix on CIFAR10 and TinyImageNet. The corruption-robustness and SSL linear-probe results also look consistent and are not confounded by the detection-rule issue. Those are real contributions, and the paper is honest that it loses to ConOSR on SVHN and CIFAR10.\n\nThe problems are mostly evaluation hygiene. First, the central incremental claim—that GradMix improves OSR over the plain SupCon+SimCLR backbone—is only supported in the main text by a cross-table comparison: Table 1 uses a Mahalanobis-distance NNO score, while Table 2 uses the new cosine k-NN score from Algorithm 1. A detection-score change alone can move AUROC by several points, so the gap is uninterpretable. The same-detector comparison in Figure 4 only covers two datasets, uses k=10 instead of the main k=3, and reports no error bars. Second, there is no code, and the five-trial averages come without standard deviations. Third, the hyperparameter reporting is inconsistent: Section 3.2 says gamma is sampled from U(0.1, 0.5), but Table 8 lists gamma=1.0 for all protocols; the lambda values for CIFAR10 and TinyImageNet appear swapped between Table 8 and Appendix F. Fourth, the layer-aggregation choice was made on the same protocols used for the final numbers, so there is some selection bias. The analysis showing broader attention (the SM metric) uses the same LayerCAM maps that were used to create the augmentations, so it is partly self-confirming.\n\nNone of this kills the idea. The mechanism is plausible and the non-OSR evidence gives it independent support. But the paper as written overclaims on OSR. I would send it to peer review—the idea deserves referee time—and ask for code, same-detector baselines in the main table, error bars, and fixed hyperparameter reporting. The OSR gain should be re-measured against the actual backbone under the same scoring rule before anyone cites it.\n\nThis is a clear case of serious thinking: the work is coherent and honest, and the flaws are fixable. Not something I would cite yet, but I'd bring it to reading group to discuss evaluation discipline in augmentation papers.","headline":"A well-motivated augmentation idea whose headline OSR gain isn't supported by the main comparison (it mixes detection rules), but the controlled robustness and SSL results give it enough substance for serious review.","tokens_in":18946,"tokens_out":3003,"would_cite":false,"duration_ms":29239,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GradMix masks the regions a model already knows during training, forcing it to learn more diverse features; the paper reports improved open set recognition on most standard protocols.","keywords":["open set recognition","data augmentation","attribution maps","LayerCAM","contrastive learning","feature diversity","out-of-distribution detection","corruption robustness"],"falsifier":"Compare GradMix to a control that masks a random region of identical size and shape, with the same uniform $\\gamma$ distribution for the mask side length; if the random control matches or beats GradMix's AUROC on CIFAR10 and TinyImageNet, the attribution signal is not what carries the gain.","tokens_in":17873,"feed_emoji":"🎯","tokens_out":6188,"duration_ms":60571,"temperature":0.7,"pith_summary":"Open set recognition asks a classifier to flag images from classes it never saw during training. The paper argues that a common failure mode is feature collapse: the model locks onto a few discriminative features of known classes and never learns the broader, less obvious features that would help tell a novel class apart. To counter this, the paper proposes GradMix, a data augmentation that uses the model's own gradient-based attribution map (LayerCAM) to find the image region it has already learned, masks that region, and replaces it with a patch from another training image. It then trains with a supervised-plus-self-supervised contrastive objective, so the model must keep working even when its preferred cues are hidden. The paper reports that this raises open set AUROC on four of six standard protocols, with the largest gain on TinyImageNet, and also improves corruption robustness and self-supervised linear-probe accuracy.","feed_headline":"Masking learned regions helps models spot the unseen","feed_subtitle":"GradMix hides the areas a model already knows, diversifying features and raising open-set detection AUROC.","key_machinery":"The load-bearing object is LayerCAM, a gradient-based attribution method that assigns a per-location weight to each feature map channel, producing finer-grained masks than Grad-CAM; the paper aggregates the attribution maps from several ResNet layers. This map is computed from the model being trained, not a fixed pretrained network, so the mask tracks what the current model finds salient. The masked view is created by patching the most activated area with a resized sample from the same minibatch, and the augmentation loss is scaled by $\\gamma^2$, the fraction of masked area. These masks turn augmentations into a self-adjusting curriculum for feature diversity.","core_discovery":"The central discovery is that a model's own attribution map can be recycled as a training-time curriculum. GradMix computes fine-grained LayerCAM maps from the current loss, selects the single most activated region, and masks it with a patch from another sample in the minibatch; the augmented view is fed to the self-supervised branch of a SupCon+SimCLR objective, weighted by the fraction of the image area masked ($\\gamma^2$). As training proceeds and the model's saliency shifts, the masks move to new regions, so the representation is repeatedly pushed off its current focus. The paper claims this yields more diverse features, better open set detection on most protocols (with over 1% AUROC gain on TinyImageNet), and consistent gains over Mixup, CutMix, and Attentive CutMix baselines.","pith_inferences":["A cleaner causal test, beyond the paper's ablations, would compare GradMix against random masking of the same area and shape; the paper's ablations compare against Mixup and CutMix, not an equal-area random-CutOut control.","The same masking curriculum could plausibly help open-set detection in object detection or segmentation, where the notion of a known class is less crisp; this is an untested extension.","GradMix's gains under corruption and in linear probing suggest attribution-guided masking may serve as a general regularizer for representation learning, independent of the OSR setting.","If attribution noise early in training is a concern, a warm-up schedule that enables GradMix only after the first epochs might improve stability; the paper does not explore schedule variants."],"forward_implications":["GradMix improves open set recognition AUROC on four of the six standard protocols, including a gain of more than 1% over the previous best on TinyImageNet.","GradMix reduces average accuracy drop under common corruptions at almost every severity level on CIFAR10 and TinyImageNet.","GradMix improves linear-probe top-1 accuracy for SimCLR and MoCo pretraining on TinyImageNet and ImageNet100 across ResNet18/34/50.","Because the method uses the model's own gradients, it can in principle be dropped into other contrastive or self-supervised pipelines without changing the architecture."],"supporting_citations":[{"why":"Supplies LayerCAM, the attribution method whose maps select the regions to mask.","marker":"Jiang et al., 2021"},{"why":"Supplies Grad-CAM, the gradient-weighted attribution formulation that LayerCAM refines.","marker":"Selvaraju et al., 2017"},{"why":"Supplies SimCLR, the self-supervised contrastive loss in the combined objective.","marker":"Chen et al., 2020"},{"why":"Supplies SupCon, the supervised contrastive loss paired with SimCLR.","marker":"Khosla et al., 2020"},{"why":"Provides the theoretical and empirical link between feature diversity and OSR performance that motivates GradMix.","marker":"Wang et al., 2024"},{"why":"Provides Nearest Non-Outlier, the distance-based detection score the paper adapts.","marker":"Scheirer et al., 2012"},{"why":"Defines the standard OSR evaluation protocols on which GradMix is tested.","marker":"Vaze et al., 2022"},{"why":"Attentive CutMix, the prior attention-guided mixing baseline the paper compares against and distinguishes itself from.","marker":"Walawalkar et al., 2020"}],"fun_headline_variants":["GradMix: mask learned regions to spot the unseen","Attribution-based masking diversifies features for open-set","Hide what the model knows, improve what it sees","Adaptive masking of saliency lifts open-set recognition","Recycle attribution maps to teach novel classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LayerCAM peak points to the region whose features the model has already learned, so cutting out exactly that region pushes the model toward features it would not otherwise acquire; if that map lands on background or noise, GradMix would have no advantage over random masking.","fun_headline_variants_meta":{"raw":{"variants":["GradMix: mask learned regions to spot the unseen","Attribution-based masking diversifies features for open-set","Hide what the model knows, improve what it sees","Adaptive masking of saliency lifts open-set recognition","Recycle attribution maps to teach novel classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1314,"prompt_tokens":870,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":486,"tokens_out":444,"duration_ms":5271,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:26:25.004696+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare GradMix to a control that masks a random region of identical size and shape, with the same uniform $\\gamma$ distribution for the mask side length; if the random control matches or beats GradMix's AUROC on CIFAR10 and TinyImageNet, the attribution signal is not what carries the gain.","supporting_citations":[],"review_version":1}