{"id":"803513e4-5105-4e16-ac5f-dbec46581cc8","arxiv_id":"2501.04269","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"RSS-MGM is a label-noise training method that unions small-loss and high-confidence sample selection and uses margin functions to split noisy samples into open-set (discarded) and closed-set (pseudo-labeled) groups, with modest accuracy gains on several image benchmarks.","lead":"This paper proposes RSS-MGM, a training method that separates noisy image datasets into clean, closed-set, and open-set samples, and then discards open-set noise while re-labeling closed-set noise. It reports modest accuracy gains over prior methods on several benchmarks, but leaves code unreleased and does not directly measure the sample-separation accuracy it claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that RSS-MGM 'more accurately divides' open- and closed-set noise rests entirely on the unmeasured assumption in Eq.","rationale":"This concern is load-bearing because the abstract and conclusion claim accurate division of open- and closed-set noise as a distinctive contribution, and the margin-guided module is the only mechanism executing that division. The paper provides intuition but no evidence that the behavioral signature in Eq. (11) holds. The empirical tables are consistent with the alternative hypothesis that the union of small-loss and high-confidence selection plus consistency regularization drives the gain, independent of the OOD rule. I therefore agree with the reader that the weakest assumption is the unvalidated OOD/ID separation behavior. The concern does not overturn the reader's verdict: the reported accuracy improvements on several benchmarks are real evidence, and a direct separation experiment could rescue the claim. Conditional acceptance remains appropriate, with the added condition that the authors report ID/OOD precision and recall and ideally release code. I do not elevate the pseudo-label equation to the primary concern because the notation there is ambiguous; the missing separation measurement is the decisive, empirically checkable point.","tokens_in":19516,"tokens_out":17868,"duration_ms":180184,"concrete_test":"On CIFAR80N-O, where OOD membership is known by construction, instrument the training loop and compute, for each noise setting (e.g., sym-20% and sym-50%), the precision, recall, and F1 of the Eq. (11) rule against true OOD labels, together with the fraction of D_rest samples that are actually ID. If OOD recall is low or the ID discard rate is high for any setting, the margin-guided separation assumption fails and the paper's explanation of its gains is not supported. Repeating this check after replacing Eq. (11) with a softmax-entropy or max-probability baseline would also show whether the chosen rule is the source of the gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The margin-guided module is the only component that operationalizes the headline claim of accurately separating open-set from closed-set noise. Its OOD rule, Eq. (11), sets M_v = I[argmax p_w != argmax p_s] and assigns a noisy sample to D_OOD when the two augmented views disagree. The paper justifies this by asserting that OOD predictions are near-uniform, so their argmax is unstable, whereas ID predictions are one-hot and stable. This is an empirical assumption, not a derived property; no experiment in Section 4 measures precision, recall, or F1 of the ID/OOD split. CIFAR80N-O is a particularly risky test: OOD classes are ordinary CIFAR-100 images excluded from the 80-class label space, and DNNs trained on 80 classes often map such images to a nearby known class confidently and consistently across augmentations. For those samples M_v=0, so they are retained as ID and either re-labeled with a wrong pseudo-label or discarded; conversely, low-confidence ID samples can have unstable argmax and be discarded as OOD. The ablations in Table 6 show only final accuracy drops when M_v or M_p is removed; they do not establish that the margin rule separates the two sets. Without a direct separation metric, the claim that RSS-MGM 'more accurately divides open set label noise samples and closed set ones' is unsupported, and the reported test-accuracy gains could be produced by the robust sample-selection union or the consistency loss alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RSS-MGM, a training scheme for image classification under label noise that mixes both closed-set and open-set noise. The method first builds a larger clean set by unioning a small-loss (JS-divergence) selection with a high-confidence selection, then applies a margin-guided module that splits the remaining noisy samples into OOD (discarded) and ID (re-labeled) sets using two augmented views and a margin ranking function. The final loss combines cross-entropy on clean samples, an improved cross-entropy on high-confidence ID samples, and a consistency regularizer. The authors evaluate on CIFAR100N-C, CIFAR80N-O, WebFG-496, and Food101N and report accuracy gains over several prior methods, with ablations on CIFAR80N-O.","tokens_in":19853,"tokens_out":4327,"duration_ms":41713,"significance":"If the claimed ID/OOD separation were convincingly demonstrated, RSS-MGM would be a useful contribution to the open-set label-noise literature, because it attempts to exploit discarded noisy samples rather than simply dropping them. The robust sample-selection idea of combining small-loss and high-confidence criteria is plausible and easy to reproduce, and the paper includes comparisons on both synthetic and real-world benchmarks. However, the paper's headline claim that RSS-MGM 'more accurately divides open set label noise samples and closed set ones' is never directly measured, and on two of the ten synthetic-noise settings the method is outperformed by PNP. The contribution is therefore currently conditional on additional evidence; the overall significance is moderate rather than definitive.","major_comments":[{"comment":"In two of the ten reported synthetic-noise configurations the method does not outperform PNP: on CIFAR100N-C Asymmetric-40% RSS-MGM gives 58.36% versus PNP's 60.25%, and on CIFAR80N-O Asymmetric-40% RSS-MGM gives 59.80% versus PNP's 61.23%. The surrounding text nevertheless states that 'RSS-MGM outperforms the prior PNP approach' and claims general superiority over state-of-the-art methods. Please qualify the claim to report the configurations where the method wins and loses, and add error bars or significance statements for the real-world tables, which currently have none.","section":"§4.2, Tables 1 and 2"},{"comment":"The central claim that RSS-MGM 'more accurately divides open set label noise samples and closed set ones' is not supported by any direct measurement. The OOD decision rule in Eq. (11) assumes that OOD samples produce near-uniform predictions whose argmax flips between weak and strong augmentations, while ID samples produce stable one-hot predictions, but no experiment in Section 4 reports precision, recall, or F1 for the ID/OOD split, even though CIFAR80N-O provides ground-truth OOD membership by construction. This is particularly important because OOD samples in CIFAR80N-O are ordinary CIFAR-100 images from 20 held-out classes, and a classifier trained on 80 classes may map them confidently and consistently to a nearby known class, making M_v=0 and causing them to be retained as ID; the ablations in Table 6 only report final accuracy, which can change through the selection or consistency-loss channels without validating the separation behavior.","section":"§3.2, Eqs. (11)-(13), and §4.4"},{"comment":"The re-labeling mechanism is not consistently defined. Section 3.3 states that samples in D_high receive pseudo-labels computed by sharpening (Eq. (21)) and averaging the two views (Eq. (22)), but the loss in Eq. (24) is written as a cross-entropy against y_i^c, which the text identifies as 'the ground-truth label of sample x_i'. If the original noisy label is used, the re-labeling step is not actually implemented; if a pseudo-label is intended, the notation should be changed to q_i^c and the sum should range over D_high, not D_id as written. This ambiguity is load-bearing because the claimed benefit of 'correcting ID samples' depends on which label the loss uses.","section":"§3.4, Eq. (24), versus §3.3, Eqs. (21)-(22)"},{"comment":"The selection thresholds tau_s=0.75, tau_h=0.9, and tau_p=0.9 are fixed values with no sensitivity analysis or selection criterion. These thresholds directly determine the sizes of the clean, ID, and OOD sets, and the method's behavior could change substantially with their values. Please report the performance as a function of each threshold, or justify the chosen values with a principled rule, since otherwise the reported gains may be threshold-tuned rather than intrinsic to the method.","section":"§4.1, thresholds in Eqs. (6), (8), and (18)"}],"minor_comments":[{"comment":"The dataset name is inconsistent: the abstract and several places in the text say 'WebFG-469', while Section 4 and the Data Availability statement say 'WebFG-496'; please correct this.","section":"Abstract and §4"},{"comment":"The caption reads 'RSS-MGS Algorithm'; this should be 'RSS-MGM'.","section":"Algorithm 1 caption"},{"comment":"The set-builder notation in Eq. (12), D_OOD = {x_i | M_i^v · D_n, x_i in D_n}, is malformed; it should be written as D_OOD = {x_i in D_n | M_i^v = 1}.","section":"§3.2, Eq. (12)"},{"comment":"The heading 'Food01N' is a typo for 'Food101N'.","section":"§4.3"},{"comment":"The sentence 'The selection coefficients λ1 and λ2 are set to 0.05 each are both set to 0.05' is duplicated and should be rewritten.","section":"§4.1, experimental settings"},{"comment":"The sentence 'The results of existing approaches are sourced from Jo-SRC (Wei et al., 2020)' appears to cite the wrong paper: Jo-SRC is by Yao et al. (2021), while Wei et al. (2020) is JoCoR; please check this citation.","section":"§4.2, citation"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible incremental contribution over the authors' own JSMix (Wen et al., 2023), and the relation between JSMix's JS-divergence selection and the proposed RSSM is not clearly delineated. The main experimental weakness is the absence of any direct ID/OOD separation metric; I would ask the authors to add such measurements on CIFAR80N-O. There is also a recurring citation error (Jo-SRC attributed to Wei et al.) and dataset-name inconsistency that should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a workmanlike training recipe for open-set label noise: it takes the known small-loss selection, adds a high-confidence selection branch, and then splits the noisy set by a margin rule (argmax disagreement between weak and strong views) into discarded OOD and re-labeled ID samples, with pseudo-labeling and consistency loss on the latter. As a pipeline, that union is new; each ingredient comes from prior work, including the authors' own JSMix for JS-divergence selection. The ablations in Tables 5 and 6 show each module buys a couple of accuracy points, and the real-world numbers on WebFG and Food101N are competitive, beating PNP on Food101N by 1.2% and edging it on Web-Aircraft/Bird.\n\nThe soft spots are real and concentrated around the headline claim. The abstract and conclusion say RSS-MGM \"more accurately divides\" open-set from closed-set noise, but no experiment measures precision, recall, or F1 of that split. The only evidence is that removing the margin functions lowers final test accuracy, which is indirect and could come from the consistency loss or sample selection alone. The mechanism in Eq. (11) assumes OOD samples have near-uniform predictions whose argmax flips across augmentations, while ID samples are one-hot and stable. That assumption is empirically shaky, especially for CIFAR80N-O, where the OOD classes are ordinary CIFAR-100 images: a model trained on 80 classes can confidently map those to a nearby known class, making M_v=0 and causing them to be kept and mislabeled, while low-confidence ID samples can flip and get discarded. The authors should measure the split quality directly, or soften the claim.\n\nThe experimental comparison also has gaps. PNP beats RSS-MGM on CIFAR100N-C Asym-40 (60.25 vs 58.36) and on CIFAR80N-O Asym-40 (61.23 vs 59.80), which undercuts the \"outperforms many SOTA\" framing. The real-world tables (Tables 3 and 4) have no error bars, several relevant baselines are missing, and no code is released. The thresholds are hand-tuned, but that is standard in this subfield, not a load-bearing flaw.\n\nWho gets value: anyone actively working on open-set label noise who wants a concrete, plausible recipe to compare against. It deserves a serious referee, but the review should demand direct separation metrics, error bars, and a fairer baseline table before acceptance. If you are in this area, it is worth reading; otherwise you are not missing much.","headline":"A plausible incremental LNL recipe that overclaims its core innovation: the margin-guided ID/OOD split is never directly measured, and PNP beats it on two key settings.","tokens_in":20390,"tokens_out":1829,"would_cite":false,"duration_ms":19000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining a two-stage robust sample selection with margin-based separation of out-of-distribution noise yields higher test accuracy than prior label-noise learning methods on both synthetic and real-world image…","keywords":["open set label noise","label noise learning","robust sample selection","margin-guided module","out-of-distribution detection","semi-supervised relabeling","noisy label image classification","CIFAR-100N"],"falsifier":"On the CIFAR80N-O benchmark, record which known open-set samples are placed in $\\mathcal{D}_{\\mathrm{OOD}}$, $\\mathcal{D}_{\\mathrm{high}}$, and $\\mathcal{D}_{\\mathrm{rest}}$; if a substantial fraction of true OOD samples are kept as in-distribution and re-labeled, or a substantial fraction of clean or closed-set samples are discarded, the margin-guided separation is not working as claimed and the accuracy gains must be attributed to other components.","tokens_in":19323,"feed_emoji":"🖼️","tokens_out":7072,"duration_ms":62390,"temperature":0.7,"pith_summary":"This paper addresses a realistic failure mode in image classification: training labels contain both closed-set errors, where the true class is in the known label list, and open-set errors, where the image belongs to an unknown class. The authors propose RSS-MGM, which first selects a larger set of clean samples by taking the union of small-loss selection and high-confidence selection, then divides the remaining noisy samples by margin rules. Open-set noise is discarded, while confident in-distribution noise is re-labeled through sharpened pseudo-labels and trained with a consistency loss. The claim is that this multi-level reuse of noisy data outperforms existing label-noise learning methods on CIFAR100N-C, CIFAR80N-O, WebFG-496, and Food101N, and separates open-set from closed-set noise more accurately.","feed_headline":"Separating open- and closed-set noise lifts accuracy on four benchmarks","feed_subtitle":"It discards true out-of-distribution noise and relabels confident in-distribution errors, beating prior methods on four benchmarks.","key_machinery":"The load-bearing mechanism is the two-module partition of the training set. The robust sample selection module computes $d_i = D_{JS}(p_i \\| y_i)$ and a confidence score $s_i$, taking $\\mathcal{D}_{\\mathrm{clean}} = \\{x_i : 1-d_i > \\tau_s\\} \\cup \\{x_i : s_i > \\tau_h\\}$, which widens the clean set beyond small-loss filtering. The margin-guided module then works on the noise set, using the weak/strong augmentation disagreement margin $M_v = \\mathbb{I}[\\arg\\max_c p^w \\neq \\arg\\max_c p^s]$ to flag out-of-distribution samples, and the average top-one-minus-top-two logit margin $M_p = \\frac{1}{2}[(z_y(v^w)-\\max_{k\\neq y} z_k(v^w)) + (z_y(v^s)-\\max_{k\\neq y} z_k(v^s))]$ to select confident in-distribution noise for pseudo-label relabeling. These two margins carry the argument: they decide which samples are discarded, which are re-labeled, and which are trained as clean.","core_discovery":"The central claim is that label noise should not be treated as all-or-nothing: instead of deleting everything that fails a clean-sample test, a network can profitably re-use part of the noise. RSS-MGM keeps the union of samples with low Jensen-Shannon divergence between prediction and label and samples whose predicted probability on the annotated class is high, giving a larger clean set than small-loss selection alone. The remaining noise set is split by two margin functions: a sample is treated as out-of-distribution if the argmax of the weakly augmented and strongly augmented predictions disagree, and as in-distribution otherwise; among in-distribution noise, samples with a large margin between the top and second logit are re-labeled via temperature-sharpened pseudo-labels and trained with a consistency-regularized loss, while the rest are dropped. On the reported benchmarks this pipeline beats or matches the strongest published methods, with the largest absolute gains at high noise rates.","pith_inferences":["The paper does not report precision or recall for the OOD-versus-ID partition; a direct test would be to count true open-set samples that survive into the relabeled set, and if that fraction is high, the accuracy gains may come more from loss robustness than from accurate margin-guided separation.","The margin rule assumes OOD predictions are near-uniform, so the method's advantage should shrink when open-set classes resemble in-distribution classes and produce confident but wrong predictions.","The same triage idea could be applied to other data-quality problems such as webly supervised fine-grained recognition, where margin disagreement is used to identify mislabeled images before pseudo-labeling; this is an extension rather than part of the paper's experiments.","A testable consequence of the paper's claim is that replacing the weak/strong disagreement rule with any uncertainty-based OOD detector should yield similar or better gains, and the gap should be measurable on WebFG-496."],"forward_implications":["At 80% symmetric noise on CIFAR100N-C, RSS-MGM reports 41.88% test accuracy versus 31.32% for PNP, so the high-confidence selection and relabeling are most valuable when few clean labels remain.","On CIFAR80N-O, the reported accuracies of 67.32%, 61.59%, and 39.86% at 20%, 50%, and 80% noise are higher than all compared methods, indicating the OOD-discarding rule helps in open-set settings even at high noise.","On Web-Aircraft, Web-Bird, and Web-Car, RSS-MGM reaches 85.82%, 82.34%, and 90.15%, beating the strongest published baseline PNP on all three real-world web datasets.","Ablations show that removing both RSS and MGM drops accuracy from 67.32% to 62.47% on CIFAR80N-O with 20% symmetric noise, and removing the margin functions $\\mathcal{M}_v$ and $\\mathcal{M}_p$ separately drops accuracy to 64.75% and 65.38%."],"supporting_citations":[{"why":"Supplies the CIFAR-100 images used to build the CIFAR100N-C and CIFAR80N-O synthetic noise benchmarks.","marker":"Krizhevsky et al., 2009"},{"why":"Introduces the open-set label noise problem and an iterative learning framework that this paper directly extends.","marker":"Wang et al., 2018"},{"why":"Co-teaching, the small-loss sample selection baseline that the robust sample selection module is designed to improve.","marker":"Han et al., 2018"},{"why":"Provides the pseudo-labeling and consistency-regularization ideas used for high-confidence selection and semi-supervised relabeling.","marker":"Sohn et al., 2020"},{"why":"Jo-SRC is the consistency-based contrastive method whose experimental settings are followed and whose results are the main comparison on the synthetic benchmarks.","marker":"Yao et al., 2021"},{"why":"PNP is the strongest prior baseline that RSS-MGM must beat and consistently outperforms on all four benchmarks.","marker":"Sun et al., 2022"},{"why":"InsCorr, a prior method that reuses discarded noisy data, providing context for the paper's decision to relabel confident in-distribution noise.","marker":"Xia et al., 2021"},{"why":"EvidentialMix, a prior method that handles combined open-set and closed-set noise, framing the setup this paper targets.","marker":"Sachdeva et al., 2021"}],"fun_headline_variants":["Split noise into open/closed, re-label confident errors","Don't throw away noise: re-label the useful part","RSS-MGM: distinguish and reuse label noise, top benchmarks","Re-classify in-distribution noise, beat SOTA on 4 datasets","Margin-guided noise re-labeling wins on four noisy datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline rests on the behavioral assumption that open-set noisy images produce near-uniform prediction probabilities whose argmax flips between weak and strong augmentation, while in-distribution images produce confident, stable one-hot predictions.","fun_headline_variants_meta":{"raw":{"variants":["Split noise into open/closed, re-label confident errors","Don't throw away noise: re-label the useful part","RSS-MGM: distinguish and reuse label noise, top benchmarks","Re-classify in-distribution noise, beat SOTA on 4 datasets","Margin-guided noise re-labeling wins on four noisy datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00095,"raw_usage":{"total_tokens":4088,"prompt_tokens":1012,"completion_tokens":3076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2989}},"tokens_in":628,"tokens_out":3076,"duration_ms":23564,"temperature":1.0,"reasoning_tokens":2989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:37:01.244123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the CIFAR80N-O benchmark, record which known open-set samples are placed in $\\mathcal{D}_{\\mathrm{OOD}}$, $\\mathcal{D}_{\\mathrm{high}}$, and $\\mathcal{D}_{\\mathrm{rest}}$; if a substantial fraction of true OOD samples are kept as in-distribution and re-labeled, or a substantial fraction of clean or closed-set samples are discarded, the margin-guided separation is not working as claimed and the accuracy gains must be attributed to other components.","supporting_citations":[],"review_version":1}