{"id":"215738e9-f05a-4b93-aaf3-75220bc8569a","arxiv_id":"2505.06684","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A unified benchmark of 18 federated learning methods under synthetic, human-annotation, and systematic label noise finds no consistently winning method and shows that an SVD-based decorrelation regularizer improves most baselines.","lead":"FNBench runs eighteen federated learning methods under three label noise patterns and six datasets to see which approaches survive noisy client data. It also shows that adding a representation decorrelation loss improves most methods, and that no single method wins everywhere.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed general gains from the SVD loss are not verifiable: no regularization coefficient is reported and Eq. (1) conflicts with Algorithm 1, so Table IX's '+SVD loss' results cannot be reproduced.","rationale":"The reader's weakest assumption was hyperparameter fairness, which is real but is acknowledged by the authors in Section VII and does not threaten the negative claim that no method dominates. The SVD-loss improvement is a separate advertised positive contribution, and its sole evidence table is not reproducible from the paper: Eq. (1) and Algorithm 1 are inconsistent, and no regularization coefficient is reported. This is exactly the kind of missing support that should be flagged. If the weight was selected favorably, the 'generally improve' claim could be a selection artifact; if the code uses an unreported large lambda, the comparison to base methods is unfair. The proposed weight sweep on two representative table rows would settle whether the effect is real and robust. Since the reader already issued a conditional verdict, I keep that verdict rather than moving it.","tokens_in":32774,"tokens_out":8771,"duration_ms":92375,"concrete_test":"Inspect the released FNBench repository for the actual L_SVD implementation and its coefficient lambda. Then re-run Table IX's FedAvg and Co-teaching rows on CIFAR-10 with Dirichlet beta=1.0 and symmetric noise 0.0-0.4, sweeping lambda over {0, 0.001, 0.01, 0.1, 1.0} and implementing both the Eq. (1) formulation and the Algorithm 1 formulation. If no (formula, lambda) combination reproduces the reported +2.31 and +4.85 gains, or if the gains only appear in a narrow lambda window, the 'generally improves' claim is unsupported; if the gains persist across a wide lambda range under both formulations, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central positive claim, that the proposed representation-aware regularization 'generally improves the robustness' of existing methods, is the least secure part of the paper. Section V-B defines L_SVD(X) = (1/d^2)||K_X||_F^2 in Eq. (1), but Algorithm 1 computes something different: it removes the diagonal of the correlation matrix and returns mean(off-diagonal^2)/B, with no scaling coefficient stated anywhere. Table IX then lists '+SVD loss' gains (FedAvg +2.31, Co-teaching +4.85, etc.) without reporting the coefficient lambda used to add L_SVD to the task loss, nor which of the two formulations was actually implemented. The scale of this term controls whether the regularization has any effect; the factor between Eq. (1) and Algorithm 1 depends on feature dimension d and batch size B and is not a constant offset. A missing hyperparameter here is not a cosmetic gap: the 'generally improve' conclusion is a causal claim about a specific loss term, and without lambda and a consistent formula Table IX cannot be reproduced or compared fairly against the base methods. Since the abstract and introduction advertise this as a contribution of the benchmark, the finding is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FNBench is a benchmark study of federated learning (FL) methods under label noise. It compares eighteen methods from four families (general FL, Byzantine-robust FL, centralized noisy-label learning, and federated noisy-label learning) across five image datasets and one text dataset, covering synthetic label noise, human annotation noise (CIFAR-10/100-N), and systematic noise (Clothing1M). The paper reports per-dataset F1 tables under multiple Non-IID partitionings and noise patterns, and it adds two analyses: (i) observations on memorization and dimensional collapse in FL, and (ii) an SVD-based representation regularizer that is combined with existing methods. The main claims are that no single method consistently dominates across settings and that the proposed SVD loss generally improves robustness.","tokens_in":33040,"tokens_out":3112,"duration_ms":30316,"significance":"The benchmark makes a useful contribution to the federated noisy-label learning community: it is open-source, covers a wider set of methods and noise patterns than prior evaluations, includes real-world annotation noise and a text modality, and its negative result (no method dominates across all Non-IID and noise settings) is a valuable caution for practitioners. The dimensional-collapse observation, if reproducible, provides a plausible diagnosis for why noisy labels impair FL. The representation-aware regularizer is the most novel positive claim, and current evidence for it is promising but incomplete because the exact loss scale and regularization coefficient are not reported. If the reproducibility gap is fixed, the paper would provide a solid comparative baseline and a cheap robustness booster for existing methods.","major_comments":[{"comment":"Eq. (1) defines L_SVD(X) = (1/d^2)||K_X||_F^2, but Algorithm 1 computes the mean of squared off-diagonal entries of the correlation matrix divided by batch size B, with no scaling by 1/d^2 and no coefficient lambda. The ratio between these two formulations depends on feature dimension d and batch size B, and is not a constant offset. Since Table IX reports '+SVD loss' gains without stating lambda or specifying which formulation was used, the claim that this regularizer 'generally improves robustness' is not reproducible from the manuscript. Please report the exact coefficient applied to L_SVD in the combined loss, specify whether Eq. (1) or Algorithm 1 was implemented, and ideally provide the code or a pointer to the exact file.","section":"Section V-B, Eq. (1) and Algorithm 1"},{"comment":"The implementation details state that all experiments are averaged over 3 seeds, but no standard deviation or variance is reported in any table. Several differences that underlie the qualitative conclusions are small (e.g., Table II Dirichlet beta=1.0 symmetric 0.0-0.4: FedAvg 73.38 vs FedExP 73.17; Table IX FedProx +0.03 gain). Without dispersion measures, the claims that 'no method consistently outperforms the others' and that SVD loss 'generally improves' performance cannot be distinguished from seed noise. Please add standard deviations, error bars, or a paired statistical test across the three seeds for at least the main synthetic tables and Table IX.","section":"Section VI-A, Tables II–IX"},{"comment":"The benchmark fixes one hyperparameter configuration per method across all settings (forget rate 0.2, Krum/TrimmedMean kappa 0.3, warm-up 20% of rounds, FedLSR gamma_e=0.3 and gamma=0.4). The limitation (ii) acknowledges that more careful tuning could improve some methods. Since the central negative result is that no method dominates under unified settings, the rankings (for example, FedLSR struggling on CIFAR-100, or FedNoRo and FedELC being robust) could be artifacts of the fixed defaults rather than intrinsic properties. Please add a sensitivity analysis that varies at least the forget rate, kappa, and FedLSR gamma parameters on one representative dataset/noise pair, or report per-method tuning ranges, to demonstrate that the qualitative conclusions are stable to reasonable hyperparameter choices.","section":"Section IV-C and Section VII limitation (ii)"}],"minor_comments":[{"comment":"Table VII reports 'Best Test Accuracy' while all other results are reported as average F1 score; please clarify why this metric differs for Clothing1M and, if possible, also report F1 for consistency.","section":"Table VII"},{"comment":"The header 'FedExp' for the general FL method should be 'FedExP' to match the main text and other tables.","section":"Table V"},{"comment":"DivideMix, FedLSR, and FedRN are listed as '-' without explanation in the caption; please add a note that these methods are excluded because they are tailored for image tasks.","section":"Table IV"},{"comment":"The Median aggregation method is cited as [62] (Ditto), but Ditto is a personalization method, not the median aggregation baseline; please cite the appropriate median-aggregation reference.","section":"Section II-E, reference [62]"},{"comment":"The text discusses singular values of the covariance matrix, while Eq. (1) is stated in terms of the correlation matrix K_X; please clarify explicitly how the covariance and correlation matrices are related in the implementation and why the correlation matrix is used.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The benchmark's breadth and open code are strengths, but the SVD-loss contribution cannot be evaluated without the missing coefficient and formula. The manuscript would also be strengthened by releasing seed-level results or standard deviations, since many claimed differences are small. The fit to IEEE TDSC is acceptable given the framing of label noise as a data-quality and poisoning concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The benchmark part is genuinely useful; the SVD-loss part is not reproducible as written and needs fixing before I'd trust Table IX.\n\nFNBench gives the subfield something it did not have: 18 methods under one codebase, three noise families (synthetic, human-annotation errors, systematic), across CIFAR-10/100, CIFAR-10/100-N, Clothing1M, plus AGNews for text. That last piece is a nice addition; almost no FNLL work touches text. The headline finding—no method wins across all Non-IID/noise combinations—is supported by the large tables. The per-group observations (RFA best among robust-aggregation methods, Co-teaching/FedNoRo/FedELC relatively reliable, FedLSR and RFL collapsing on CIFAR-100) are plausible and useful. Limitations are stated honestly, and the code is public.\n\nSoft spots, in order. First, the stress-test concern is correct. Eq. (1) defines L_SVD as (1/d^2)||K_X||_F^2, but Algorithm 1 standardizes the representation, removes the diagonal, and returns mean(off-diagonal^2)/B. Those are different functions, and the missing 1/d^2 and /B are not constant offsets when d and batch size change. Table IX reports gains without any lambda. So the claim that SVD loss generally improves robustness cannot be checked or reproduced. That is a real gap, not cosmetic. The benchmark's main conclusions do not depend on it, but the abstract advertises the regularizer as a contribution, so the authors need to fix the formula, state the coefficient, and ideally rerun the table.\n\nSecond, fixed hyperparameters. One config per method, \"favorably in line\" with original papers, with forget rate, kappa, and warm-up set mildly. The authors admit that more careful tuning would improve some methods. So individual rankings, especially bottom-ranked FedLSR and RFL on CIFAR-100, should be read as \"under these settings,\" not as absolute verdicts. That is normal for a benchmark, but a sensitivity analysis around those choices would make the ranking claims much sturdier. A noiseless FedAvg column in the synthetic tables would also help calibrate absolute degradation; without it, the tables are harder to interpret.\n\nThird, they average over 3 seeds but report no variance. Minor-to-moderate: many gaps are large, but some conclusions rest on one- or two-point differences. Also, \"first comprehensive benchmark\" sits awkwardly next to the concurrent FedNoisy benchmark, which they do cite; I would soften that claim.\n\nBottom line: for anyone working on noisy labels in federated learning, this is a useful comparative resource and deserves a serious referee. But I would not accept it as is. The SVD loss section needs to be rewritten with the actual loss, the lambda value, and a corrected experiment; the authors should report per-seed spreads; and they should add a short tuning-sensitivity check. That is a major-revision path, not a desk reject.","headline":"A genuinely useful unified benchmark for federated noisy-label learning, but the SVD-loss booster claim is not reproducible as written and needs a serious revision before publication.","tokens_in":33575,"tokens_out":2749,"would_cite":true,"duration_ms":30241,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under unified benchmark settings, no federated noisy-label method consistently outperforms the others, and a decorrelation regularizer generally improves robustness.","keywords":["federated learning","noisy labels","benchmark","label noise robustness","dimensional collapse","representation regularization","non-IID data","SVD loss"],"falsifier":"Re-run the CIFAR-10 symmetric-noise (0.0–0.4) comparison with per-method grid search over forget rate, kappa, and warm-up ratio, and compare rankings to the paper's tables; if, for example, a tuned FedLSR or RFA overtakes FedNoRo and FedELC on several settings, the paper's relative-robustness claims would not survive as stated.","tokens_in":1472,"feed_emoji":"📊","tokens_out":1363,"duration_ms":58499,"temperature":0.7,"pith_summary":"FNBench is the authors' attempt to settle, under one set of experimental conditions, how well existing methods really cope with mislabeled data in federated learning. The paper puts eighteen methods from four research lines—general FL, byzantine-robust aggregation, centralized noisy-label learning, and federated noisy-label learning—through three noise regimes: synthetic flips, real human annotation errors, and systematic web-collection noise. Its central conclusion is that no method consistently wins across non-IID partitions and noise patterns, and that adding a simple representation decorrelation regularizer (SVD loss) generally improves most methods' robustness. A sympathetic reader would take this as evidence that the field needs a shared evaluation protocol before claims of superiority can be trusted.","feed_headline":"No single method wins against noisy labels in federated learning","feed_subtitle":"A simple decorrelation regularizer lifts most methods' F1 scores across noise types.","key_machinery":"The load-bearing mechanism is the SVD loss, a representation decorrelation regularizer: within each local batch, the model subtracts the mean, normalizes feature dimensions, computes the correlation matrix of the representation matrix, and penalizes the mean squared off-diagonal entries. This discourages different representation dimensions from correlating and stops the singular-value spectrum from collapsing toward lower rank, countering the dimensional collapse the paper observes under label noise. The benchmark itself is the other piece of machinery: three label-noise patterns (synthetic, human-annotation, systematic), two image datasets plus a text dataset, and eighteen baselines under shared hyperparameters.","core_discovery":"Under a unified protocol with three non-IID partitioning schemes and four synthetic label-noise patterns on CIFAR-10 and CIFAR-100, plus CIFAR-10/100-N human-annotation noise and Clothing1M systematic noise, the authors find that no evaluated method dominates across all settings. Simpler FL methods (FedAvg, FedProx, FedExP) are comparatively robust at low noise rates, co-teaching-style loss-based sample selection and the FNLL methods FedNoRo and FedELC are among the more robust, byzantine-robust aggregation methods—especially Krum—perform poorly on distributed label noise, and methods like FedLSR and RFL degrade sharply on the 100-class datasets. The paper further shows noisy labels induce dimensional collapse of learned representations, and reports that adding an SVD-based decorrelation loss to local training improves F1 for nearly every method, with only FedELC showing negligible change.","pith_inferences":["Editorial inference: the absence of a single winner implies benchmark reports should present per-setting rankings or average ranks rather than a single best method, since any headline 'best' will be setting-dependent.","Editorial inference: the consistent gains from SVD loss suggest that label-noise robustness in federated learning is substantially a representation-quality problem; a testable extension would be to combine SVD loss with label-correction methods on CIFAR-100-N, where the paper finds both ideas individually effective.","Editorial inference: because per-method hyperparameters were fixed, rankings could shift under tuning; a natural follow-up is a sensitivity analysis reporting each method's performance variation across reasonable hyperparameter ranges."],"forward_implications":["Any future claim that a new federated noisy-label method is state of the art should be measured against this benchmark's settings; at present no method has a universal advantage.","Byzantine-robust aggregation methods, particularly Krum, should not be assumed to handle distributed label noise simply because they tolerate adversarial clients.","Loss-based sample selection (Co-teaching, FedNoRo, FedELC) is a more promising robustness pattern than centroid-based or self-regularization approaches in high-class-count regimes.","Adding SVD loss is a cheap, generally positive modification for most existing methods, suggesting representation regularization should be a standard component in future federated noisy-label designs.","Smaller local batch sizes improved performance under limited communication rounds in the paper's ablations, so batch size deserves explicit reporting in comparisons."],"supporting_citations":[{"why":"Supplies the SVD-based decorrelation regularization (FedDecorr) that the paper adapts into the SVD loss.","marker":"[15]"},{"why":"Provides CIFAR-10/100-N real-world human annotation error data used as the imperfect-annotation noise benchmark.","marker":"[73]"},{"why":"Provides Clothing1M, the large-scale dataset with systematic web-collection label noise.","marker":"[34]"},{"why":"Co-teaching, the loss-based sample-selection method that is among the more robust NLL baselines.","marker":"[21]"},{"why":"FedNoRo, an FNLL baseline that the benchmark finds relatively robust across settings.","marker":"[6]"},{"why":"FedELC, an end-to-end label-correction FNLL method whose robustness the benchmark reports.","marker":"[1]"},{"why":"FedLSR, a local self-regularization FNLL baseline whose earlier IID/noise settings the paper generalizes.","marker":"[4]"},{"why":"RFL, the first FNLL method, providing global classwise centroid supervision as a baseline.","marker":"[18]"}],"fun_headline_variants":["No FL method dominates across all label-noise settings","Benchmark shows FedAvg robust, Krum weak to label noise","Decorrelation loss lifts most FL methods' F1 on noisy labels","FNBench: first unified benchmark for noisy-label FL","Noisy labels trigger dimensional collapse in FL"],"cache_read_input_tokens":35712,"weakest_assumption_plain":"The rankings assume that one fixed hyperparameter setting per method, chosen from original papers or mild defaults, is equally fair to all eighteen methods across every noise pattern and non-IID partition; if some methods are systematically undertuned, the relative robustness conclusions could be artifacts of configuration.","fun_headline_variants_meta":{"raw":{"variants":["No FL method dominates across all label-noise settings","Benchmark shows FedAvg robust, Krum weak to label noise","Decorrelation loss lifts most FL methods' F1 on noisy labels","FNBench: first unified benchmark for noisy-label FL","Noisy labels trigger dimensional collapse in FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1217,"prompt_tokens":921,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":217}},"tokens_in":537,"tokens_out":296,"duration_ms":3392,"temperature":1.0,"reasoning_tokens":217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:35:37.475621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CIFAR-10 symmetric-noise (0.0–0.4) comparison with per-method grid search over forget rate, kappa, and warm-up ratio, and compare rankings to the paper's tables; if, for example, a tuned FedLSR or RFA overtakes FedNoRo and FedELC on several settings, the paper's relative-robustness claims would not survive as stated.","supporting_citations":[],"review_version":1}