{"id":"b54ee5cb-832a-4baf-a15f-ea3f88a78502","arxiv_id":"2608.03432","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A decoupled two-source reliability score for label correction and sample reweighting improves refurbishment-based noisy-label learners across synthetic and real-world benchmarks.","lead":"When computers learn from examples that contain wrong answers, a common trick is to replace a suspicious label with the computer's own guess. This paper shows that the guess can be wrong in the same way as the original label, and proposes checking the original label and the guess separately.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-target 'independent' gate in Eq. (13) is still the same noise-absorbed model's confidence; high-confidence wrong targets (6.16% in Tab. 7, 30.93% pseudo accuracy on CIFAR-100N noisy samples in Tab. 18) mean the replacement-noise risk is attenuated, not removed.","rationale":"The paper identifies a real and well-motivated coupling in refurbishment rules, and its empirical package is broad and internally consistent. The difficulty is in the core design claim: TRACE separates observed-label and pseudo-target reliability, but the pseudo-target score in Eq. (13) is model confidence from the same networks whose noisy bias motivated the separation. The reader's weakest assumption names exactly this fragility, and the paper's own diagnostics confirm it is not fully resolved: high-confidence wrong targets persist at 6.16% in the headline CIFAR-100 setting, and the hardest real-noise case, CIFAR-100N, shows pseudo targets following noisy labels 51.72% of the time on observed-noisy samples. This does not make the paper wrong; it makes the central claim conditional on a proxy whose limits are only partially mapped. The proposed test would settle whether a genuinely independent pseudo-target score changes the conclusion, or whether the current confidence gate is already near the achievable bound. Because the concern is an addressable weakness rather than a demonstrated contradiction, I keep the reader's CONDITIONAL verdict rather than moving to REJECT or ACCEPT.","tokens_in":28577,"tokens_out":4728,"duration_ms":48623,"concrete_test":"Run CIFAR-100 with 40% instance-dependent noise and CIFAR-100N; after training RoLR and TRACE with seeds 0/42/1027, restrict to low-clean+noisy samples and compute, within each s_pseudo decile, pseudo-target accuracy, follow-noisy rate, and HC Wrong for Eq. (13). Then re-run TRACE with s_pseudo replaced by (a) confidence from a clean-trained reference model and (b) a non-confidence signal such as dual-network disagreement or relation drift. If (a) substantially improves pseudo accuracy and lowers HC Wrong relative to Eq. (13), the model-confidence gate is not providing independent pseudo-target evidence; if (a) and (b) do not help, the current gate already captures most available signal. Compare all variants with standard errors across the three seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (13) defines pseudo-target reliability as s_pseudo_i = (max_c q_i,c)^rho, where q_i is the sharpened average of two network predictions (Eq. 5) trained on the same corrupted labels. The 'independent' pseudo score is therefore not evidence-independent: the confident wrong targets it passes are produced by the same noise-absorbed model whose observed-label reliability is in doubt. The paper is candid that this is a filter, not a certificate, and Tab. 7 still reports 6.16% of high-confidence pseudo targets are wrong in the headline setting; Tab. 18 shows on CIFAR-100N observed-noisy samples, pseudo-target accuracy is only 30.93% while follow-noisy is 51.72%. Under stronger or instance-dependent noise, high confidence and correctness can decouple further, so Eq. (8)'s b_i = (1 - s_obs_i) * s_pseudo_i can still inject the noisy label exactly when correction is needed. The theoretical SNR argument in App. B (Eqs. 16-17) assumes a meaningful r_pseudo estimate, but model confidence is only a proxy for it. The paper demonstrates attenuation of the 'replacing noise with noise' failure, not its removal; whether the separation is sufficient as a general interface remains unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies label refurbishment in noisy-label learning, where a corrected target is formed by interpolating the observed noisy label and a model-derived pseudo target via a single sample-wise cleanliness score. The authors argue that this one-scalar controller couples two distinct decisions—whether the observed label is trustworthy and whether the proposed pseudo target is trustworthy—and can therefore replace one unreliable signal with another. They present representation diagnostics showing that noisy supervision perturbs deeper layers more strongly while shallower relations remain relatively stable, and they use this to motivate TRACE, a plug-in framework that scores the observed label with a combination of loss confidence, shallow-to-deep relation stability, and dual-network agreement, while scoring the pseudo target separately with a confidence-based gate. The paper evaluates TRACE on CIFAR synthetic noise, CIFAR-N, WebVision, Food-101N, and Clothing1M, reporting accuracy improvements over several refurbishment baselines and additional diagnostics on pseudo-target reliability. The central claim is that source-specific reliability assessment, rather than a single complementary coefficient, is a safer interface for label correction and sample reweighting.","tokens_in":28871,"tokens_out":4728,"duration_ms":48959,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: it identifies a genuine design flaw in a family of popular noisy-label methods and proposes a simple, reasonably general interface for correcting it. The SNR derivation in Appendix B is elementary but correct, and the ablation in Table 5 is well designed because it isolates the decoupling effect by changing only the pseudo-branch weight while holding the pseudo-target generator and training pipeline fixed. The benchmark coverage is broad, spanning synthetic, human-annotation, and large-scale real-world noise. The paper is also candid about the limitations of its pseudo-target confidence gate, explicitly calling it a filter rather than a certificate. However, the overall contribution is incremental rather than foundational: the gains over strong baselines are often small, the reported improvements lack statistical grounding because no variances are given, and the key 'independent' pseudo-target reliability signal is produced by the same noise-trained model whose observed-label reliability is being questioned.","major_comments":[{"comment":"The pseudo-target reliability score s_pseudo_i = (max_c q_i,c)^rho is not an evidence-independent signal: q_i is the sharpened average of two network predictions trained on the same corrupted labels (Eq. 5), so high confidence can be produced by the same noise-absorbed model whose observed-label reliability is in doubt. The paper is honest that this is a filter, not a certificate, but the abstract and Eq. (8) call it an 'independent confidence gate' and 'separate evidence,' which overstates the case. The paper's own Table 7 reports 6.16% high-confidence wrong pseudo targets on CIFAR-100 with 50% symmetric noise, and Table 18 shows that on CIFAR-100N observed-noisy samples pseudo-target accuracy is 30.93% while follow-noisy is 51.72%. The SNR argument in Eqs. (16)-(17) assumes a meaningful r_pseudo; model confidence is only a proxy for it. I recommend either rephrasing the claim as 'separate' rather than 'independent' and adding an explicit analysis of how pseudo-target accuracy varies with s_pseudo under instance-dependent or harder real-world noise, or replacing the confidence gate with a more genuinely external reliability signal such as cross-view agreement or temporal consistency.","section":"Tables 1-3"},{"comment":"The pseudo-target reliability score is model confidence, and the paper's own tables show residual high-confidence errors, so the separation is attenuation rather than elimination.","section":"Tables 1-3"},{"comment":"The ablation in Table 5 is a strength, but the absence of variance information in the main tables prevents the reader from judging whether the reported improvements are meaningful.","section":"Hyperparameter sensitivity, Table 7"},{"comment":"The scope limitation stated in Appendix G is appropriate and should be kept, but it also weakens the generality of the claimed interface: the paper's evidence is strongest for closed-set image classification with explicit pseudo-target generators, and the hardest real-noise setting (CIFAR-100N) shows only a modest +0.70-point gain in pseudo-target accuracy on the low-clean+noisy subset (Table 17). This is not a fatal flaw, but the abstract's phrase 'yields more reliable pseudo supervision' should be qualified by the actual magnitude and by the residual follow-noisy behavior documented in Table 18. I recommend adding a short discussion in the main text of the CIFAR-100N boundary case, rather than leaving it only in the appendix.","section":"Appendix G, Limitations"}],"minor_comments":[{"comment":"The notation Norm(·) is used for two different transformations in Eq. (10): first for delta_i and then for exp(-gamma tilde_delta_i). Please define these separately or use different symbols, e.g., batch min-max normalization vs. a second min-max pass.","section":"Eq. (10)"},{"comment":"The caption contains the phrase 'holds obs' which appears to be a typo for 'holds s_obs fixed'. Please correct this.","section":"Table 5 caption"},{"comment":"The caption says 'the shared legend is shown in the middle plot,' but the legend is difficult to read in the printed figure. Please ensure the legend is legible or move it to the caption.","section":"Figure 2"},{"comment":"Several low-clean pseudo-accuracy values decrease under TRACE (e.g., CIFAR10-Sym20 from 96.05 to 93.20, CIFAR10-Sym50 from 95.93 to 94.51), while low-clean+noisy pseudo-accuracy increases. The text notes small decreases only for global pseudo-accuracy, not for these low-clean decreases; please state this explicitly so readers can interpret the trade-off.","section":"Table 17"},{"comment":"The Clothing1M diagnostic uses GPT-5.6 API annotations as surrogate reference labels. The paper mentions the limitation, but the main-text reference to this experiment should also state that these labels are model-generated and not human-verified, to avoid over-interpretation.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of cs.LG and the core conceptual observation—that a single cleanliness coefficient couples observed-label and pseudo-target trust—is sound and worth publishing after revision. The main risks are empirical: the absence of error bars in the headline tables and the selection of key hyperparameters on a headline test setting. The 'independence' of the pseudo-target confidence gate is also overstated relative to what the paper actually demonstrates, though the authors are transparent about the filter/certificate distinction. I would not reject on these grounds, but the revision should address the statistical grounding of the comparisons and the wording of the independence claim. No code is released; the authors should be encouraged to provide it, as the paired plug-in protocol would otherwise be hard to reproduce across five datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TRACE is worth engaging seriously. The paper identifies a real flaw in refurbishment methods — using one cleanliness score to both discount the observed label and boost the pseudo target — and shows that decoupling the two decisions helps across a wide range of benchmarks. The formal bit (Eqs. 16–17) is correct as far as it goes: the pseudo branch only improves the mixture SNR when its own reliability exceeds the observed branch's. That is a clean way to state the problem.\n\nThe strongest evidence is the ablation in Tab. 5. It isolates the decoupled reliability form (1 - s_obs) * s_pseudo against the coupled null and shows a consistent gain. The representation diagnostics (deeper layers more corrupted, shallow relations more stable) are also a reasonable motivating story, and the paper is careful not to overclaim them as a universal depth law — the appendix explicitly notes MobileNetV2 deviates.\n\nThe soft spots are real but not fatal. The pseudo-target reliability score is max_c q_i,c, confidence from the same noise-absorbed model that produced the pseudo target. That is not independent evidence. The paper is honest that this is a filter, not a certificate, and Tab. 7 still shows 6.16% high-confidence errors on the headline setting. The right reading is that TRACE attenuates the 'replacing noise with noise' failure, not that it removes it. The 'independent' wording in the method section oversells this.\n\nOther issues: Tables 1–3 report no standard deviations despite claiming three seeds, and there's no code or data release. The hyperparameters alpha and rho are tuned on the same benchmarks used for evaluation, though the sensitivity sweeps are smooth enough that this is a minor concern. On the hardest real-noise set (CIFAR-100N) the gain is small, which the paper acknowledges.\n\nNone of these sink the central claim. The decoupling idea is sound, the ablation is clean, and the empirical pattern is consistent. The paper deserves a serious referee, not a desk reject. For publication, the authors should provide error bars, release artifacts, and reword the independence claim to something like 'separate but model-derived.'","headline":"Real contribution with an over-sold 'independent' pseudo gate; deserves peer review, not desk rejection.","tokens_in":29441,"tokens_out":1852,"would_cite":true,"duration_ms":17128,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that refurbishment-based noisy-label learning can replace one noise with another unless the observed label and pseudo target are assessed with separate reliability scores.","keywords":["label noise","label refurbishment","pseudo labels","reliability assessment","sample reweighting","deep representation analysis","CIFAR-10N/CIFAR-100N","noisy-label learning"],"falsifier":"Measure, on a noisy validation set, the accuracy of high-confidence pseudo targets versus low-confidence pseudo targets on samples whose observed label is wrong; if the gap disappears or reverses under stronger noise or a different architecture, the pseudo-target confidence gate carries no signal and TRACE's core benefit collapses. The paper already shows the gate is imperfect on CIFAR-100N, where pseudo-target accuracy on observed-noisy samples is only 30.93%.","tokens_in":28336,"feed_emoji":"🎯","tokens_out":8626,"duration_ms":72113,"temperature":0.7,"pith_summary":"The paper targets the common refurbishment recipe for learning with noisy labels, which interpolates between the observed label and a model-generated pseudo target using a single sample-wise cleanliness score. It argues that this one-scalar control carries a hidden complementarity: when the model lowers its trust in the observed label, it automatically raises the influence of the pseudo target, even though the pseudo target is produced by a model trained on the same corrupted labels. The proposed method, TRACE, instead scores the two supervision sources separately—observed-label reliability from loss fit, shallow-to-deep relation stability, and prediction agreement; pseudo-target reliability from its own confidence—and reports consistent accuracy gains on synthetic, human-annotated, and large-scale real-world noisy benchmarks. The point matters because the failure mode is structural, applying to any method that writes label correction as a complementary mix of two signals.","feed_headline":"Two trust scores beat one in noisy-label learning","feed_subtitle":"Separating observed-label trust from pseudo-target trust cut high-confidence errors on CIFAR-100 from 8.93% to 6.16%.","key_machinery":"The load-bearing object is the branch-weight decomposition of the corrected target. Standard methods use $\\tilde y_i = \\lambda_i \\hat y_i + (1-\\lambda_i) q_i$ with $\\lambda_i$ from a loss-based clean posterior; TRACE writes $\\tilde y_i = (a_i \\hat y_i + b_i q_i)/(a_i + b_i + \\epsilon)$ with $a_i = s^{obs}_i$ and $b_i = (1 - s^{obs}_i) s^{pseudo}_i$. The observed-label score $s^{obs}_i$ combines the loss posterior $c^{loss}_i$, a shallow-to-deep relation stability $c^{str}_i$ computed from cosine relation matrices across layers, and a dual-network agreement gate $g^{agr}_i$; the pseudo-target score is the confidence gate $s^{pseudo}_i = (\\max_c q_{i,c})^\\rho$. The appendix supplies a local signal-to-noise ratio argument showing that increasing the pseudo-branch weight improves the mixed target only when the pseudo target's conditional reliability exceeds the observed label's, which is the formal reason the two decisions should not be coupled.","core_discovery":"The paper's central claim is that the refurbishment rule $y^{rec}_i = \\lambda_i \\hat y_i + (1 - \\lambda_i) q_i$—one scalar $\\lambda_i$ controlling both the observed label and the pseudo target—is not a safe interface. Since $q_i$ is generated by a model that was itself trained on the noisy labels, low trust in $\\hat y_i$ does not supply evidence that $q_i$ is correct; the pseudo target can faithfully reproduce the error it was meant to fix. TRACE replaces the complementary weights with source-specific scores $a_i = s^{obs}_i$ and $b_i = (1 - s^{obs}_i) s^{pseudo}_i$, so a corrupted observed label creates a need for correction but cannot activate pseudo supervision unless the pseudo target is independently deemed reliable. The empirical claim is that this decoupling improves accuracy and pseudo-target reliability over representative refurbishment baselines (DivideMix, RoLR, DISC, ANNE) across CIFAR synthetic noise, CIFAR-10N/100N, WebVision, Food-101N, and Clothing1M.","pith_inferences":["Editorial extension: the same two-source decoupling applies wherever one model-generated signal supervises another, including self-training and semi-supervised learning; in all such settings a single confidence score for the teacher may over-trust a biased teacher.","Editorial extension: the appendix's SNR inequality suggests a direct validation check—measure per-sample pseudo-target reliability and observed-label reliability on a held-out noisy set and test whether TRACE's gating tracks the condition $r^{pseudo}_i > r^{obs}_i$.","Editorial extension: treating shallow relation stability as the anchor suggests a cheaper variant using early-layer nearest-neighbour agreement only, which the paper's diagnostics imply should carry much of the observed-label signal.","Editorial extension: on CIFAR-100N, where the reported gains are smallest, the confidence gate alone appears insufficient for human annotation noise; a stronger pseudo-target oracle (e.g., an ensemble of diverse teachers) may be needed before the decoupling pays off there."],"forward_implications":["Any future refurbishment method that writes the corrected target as a complementary convex combination of observed and pseudo signals should treat the one-scalar control as a design flaw, not a default.","When both branches are unreliable, the correct response is to down-weight the sample's supervision strength, not to force a correction; TRACE's weight $w_i = \\max(a_i+b_i, w_{\\min})$ encodes this.","The pseudo-confidence gate is what makes pseudo supervision safer: on CIFAR-100 with 50% symmetric noise, TRACE reduces high-confidence pseudo-target errors from 8.93% to 6.16% and raises pseudo-target accuracy on low-clean+noisy samples from 72.92% to 84.70%.","The plug-in design means the decoupling can be added without changing the pseudo-target generator, so the reported gains are attributed to the reliability interface rather than to a new target construction."],"supporting_citations":[{"why":"Canonical refurbishment method (DivideMix) that defines the one-coefficient mixing rule TRACE criticizes.","marker":"Li, Socher, and Hoi 2020"},{"why":"Introduces label refurbishment as a recipe, the family of methods TRACE targets.","marker":"Song, Kim, and Lee 2019"},{"why":"RoLR, the base learner whose pseudo-target generator and confidence mechanism TRACE preserves as its instantiation.","marker":"Chen et al. 2023"},{"why":"Provides the CIFAR-10N/CIFAR-100N human-annotation benchmarks used to test real noise.","marker":"Wei et al. 2022"},{"why":"Representation-level evidence that noisy supervision distorts deep representations, motivating the shallow-deep contrast.","marker":"Wongso, Ghosh, and Motani 2023"},{"why":"Locates memorization in later network stages, supporting the depth-dependent noisy-bias account.","marker":"Maini et al. 2023"},{"why":"Supplies the instance-dependent label-noise protocol used in synthetic experiments.","marker":"Chen et al. 2020"},{"why":"Source of the Clothing1M large-scale real-world noisy dataset.","marker":"Xiao et al. 2015"}],"fun_headline_variants":["Two-source trust beats one in noisy-label learning","Separate trust scores stop noise replacement in label learning","Don't replace noise with noise: decouple label trust","Independent trust scores improve label-noise correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the maximum probability of the pseudo target, raised to a power, is a reliable gate for whether that pseudo target is correct—even though the same model produced both the confidence and the noisy-label error it is meant to correct.","fun_headline_variants_meta":{"raw":{"variants":["Two-source trust beats one in noisy-label learning","Separate trust scores stop noise replacement in label learning","Don't replace noise with noise: decouple label trust","Independent trust scores improve label-noise correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2766,"prompt_tokens":973,"completion_tokens":1793,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1733}},"tokens_in":589,"tokens_out":1793,"duration_ms":11824,"temperature":1.0,"reasoning_tokens":1733,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:50:05.442639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a noisy validation set, the accuracy of high-confidence pseudo targets versus low-confidence pseudo targets on samples whose observed label is wrong; if the gap disappears or reverses under stronger noise or a different architecture, the pseudo-target confidence gate carries no signal and TRACE's core benefit collapses. The paper already shows the gate is imperfect on CIFAR-100N, where pseudo-target accuracy on observed-noisy samples is only 30.93%.","supporting_citations":[],"review_version":2}