{"id":"22891b94-0beb-4f19-a56a-5690c28048ee","arxiv_id":"2501.11360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage federated learning method that progressively trains on low-loss samples first and adds high-loss samples later reduces client drift and improves accuracy under label skew, feature skew, and noisy labels.","lead":"FedBSS is a new federated learning training scheme that ranks each client's local samples by loss, splits them into low-bias and high-bias groups, and gradually adds the high-bias samples during local training to reduce client drift. A generalist reader should care because client drift is a main obstacle to training accurate models on heterogeneous user data, and this sample-level approach is a new angle on an old problem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's core mechanism—loss/uncertainty ranking as drift proxy—is never tested against a random or local-model ranking; without that ablation, Table 1's gains do not establish the drift claim.","rationale":"The paper's practical contribution—a sample-selection curriculum with no extra communication—could still be useful even if the stated drift mechanism is wrong, which is why I do not recommend rejection. But the abstract and introduction claim a causal explanation ('client drift can be viewed as a cumulative manifestation of biases present in all local samples'), and the experiments do not discriminate between that explanation and a generic curriculum effect. The missing control is cheap: randomize the ordering that Eq. 6 relies on, or compute the ranking with the local model instead of the global model. This is the same soft spot the reader identified as the weakest assumption; I agree with that reading. The concern is load-bearing because every subsequent design choice (warmup length, cosine schedule, uncertainty boundary) is justified by the drift story, and if the story is wrong those hyperparameters are unprincipled. However, the concern is empirically testable and does not by itself overturn the reported accuracy numbers, so the conditional verdict stands pending the ablation.","tokens_in":14706,"tokens_out":6533,"duration_ms":69647,"concrete_test":"Run the full FedBSS pipeline on CIFAR-10 Dirichlet(0.1) (ResNet-50, 100 clients, 10% participation, 200 rounds, same hyperparameters) in three variants, each with 5 seeds: (A) loss/uncertainty ranking from Eqs. 3-5; (B) same progressive schedule (Eq. 6) but with a random permutation of the client's samples replacing the loss ranking; (C) ranking by the client's local-model loss after the warmup stage. If (B) matches (A) within 1 point, the improvement is a curriculum effect and the central drift mechanism is unsupported. If (C) matches (A), the global model is not doing the claimed work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that client drift is a cumulative per-sample bias and that the progressive schedule in Eq. (6) mitigates it. This requires two identifications: loss under the current global model equals per-sample drift contribution (Eq. 3), and the maximum-uncertainty sample is a meaningful unbiased/biased boundary (Eq. 5). Neither is validated. The supporting experiment (Fig. 3a) trains separate models on the bottom-50% and top-50% loss subsets, which differ in set size and label composition, so it cannot isolate per-sample drift. Moreover, Fig. 3b shows uncertainty is non-monotone in loss, with a confident-but-wrong high-loss regime; Eq. (5) therefore places the boundary at a point that does not correspond to 'samples that contribute most to drift.' It is a loss cut justified only by the assumed proxy. Under this cut, Eq. (6) eventually trains on exactly the samples the model is most confidently wrong about, which could reinforce error rather than reduce drift. The only selection ablation (Fig. 5b) compares filter/linear/cosine schedules, never the ranking criterion itself. Consequently, the large gains in Tables 1-2 could be produced by any easy-to-hard curriculum, with no evidence that the drift mechanism is what drives them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FedBSS, a two-stage federated learning algorithm. In Stage 1 (warm-up), standard FL runs for T1 rounds. In Stage 2, each client computes per-sample losses under the current global model, ranks samples, splits them at the sample with maximum uncertainty (Eq. 5), and trains on the low-loss 'unbiased' set while progressively adding high-loss 'biased' samples according to the cosine schedule in Eq. (6). The authors argue that client drift is the cumulative effect of per-sample biases and that progressively learning from low-bias to high-bias samples mitigates drift. They report accuracy comparisons on Fashion-MNIST, CIFAR-10, and CIFAR-100 under Shards and Dirichlet label-skew partitions, on noisy-label versions of those datasets, and on DomainNet with feature skew.","tokens_in":15009,"tokens_out":4850,"duration_ms":47959,"significance":"If the mechanism were established, FedBSS would be a useful addition: it is communication-free beyond standard FedAvg, simple to implement, and the tables show consistent gains over several strong baselines, especially under high label skew and high noise. The authors also provide a schedule ablation (Fig. 5b) and a warm-up sensitivity study (Fig. 5a). However, the core conceptual claim—that loss under the global model measures per-sample drift contribution and that the maximum-uncertainty split (Eq. 5) separates unbiased from biased samples—is not directly tested. The empirical gains could arise from a generic curriculum effect rather than from drift mitigation. The DomainNet experiment is under-reported. These gaps limit the paper's significance until the mechanism is isolated.","major_comments":[{"comment":"The experiment supporting the claim that 'loss is a measure of the sample's influence on client drift' trains separate models on the bottom-50%, top-50%, and 100% loss subsets. These conditions differ in both subset size and label composition, so the visualized drift differences do not isolate per-sample bias; the same pattern could appear with any random hard/easy split. Please add a size- and class-balanced control, and quantify drift (e.g., local-update deviation from the global update) rather than showing unlabeled trajectories.","section":"Motivation (Q1), Figure 3(a)"},{"comment":"Figure 3(b) shows that uncertainty is non-monotone in loss: the highest-loss samples are often confidently wrong. Thus the maximum-uncertainty point does not delineate 'samples that contribute most to drift' from those that contribute little; it merely picks the most uncertain sample. The paper provides no evidence that this boundary is the correct unbiased/biased split. Please test the threshold rule against alternatives (e.g., fixed quantile, a loss-based boundary, or a boundary chosen by directly measured drift contribution) and report results separately.","section":"Methodology, Eqs. (4)-(5) and Figure 3(b)"},{"comment":"The only selection ablation varies the addition schedule (filter/linear/cosine), never the ranking criterion. To support the drift-mitigation explanation, hold Eq. (6)'s schedule fixed and compare global-model-loss ranking against (a) random ranking and (b) local-model-loss ranking. Without this ablation, the large gains in Tables 1 and 2 could be produced by any easy-to-hard curriculum that progressively adds samples, with no need for the claimed drift mechanism.","section":"Experiments, Figure 5(b), Tables 1-2"},{"comment":"The warm-up length T1=50 is chosen after an ablation on the benchmark datasets (Fig. 5a), and the text concedes that '50 rounds is not always optimal'. This makes T1 a tuned hyperparameter, yet it is fixed across all datasets and partitions. Please provide a principled selection rule or a per-setting tuning protocol, and state how T1 was chosen in Tables 1-2; otherwise the comparison includes benchmark-specific parameter fitting.","section":"Implementation Details and Ablation (Warmup Stage)"},{"comment":"The DomainNet experiment reports only FedAvg and FedBSS, with no standard deviations, no baseline methods, and no numerical final global accuracy. The claim that FedBSS is effective under feature distribution skew is therefore not supported. Please add results for at least the main baselines (FedProx, Scaffold, FedLC, InCo) with multiple runs, and specify the per-client data composition.","section":"Result with Feature Distribution Skew, Figure 4"}],"minor_comments":[{"comment":"The sentences 'The specific comparison results are in section .' and 'More discussion is demonstrated in section .' have empty cross-references; fill them with the appropriate experiment or ablation subsection.","section":"Methodology"},{"comment":"The aggregation step writes ϑ_t = (1/N) Σ_n ϑ^n_t although only a random subset N_t of clients is selected; clarify whether the average is over the selected clients only.","section":"Algorithm 1, line 12"},{"comment":"The heading 'Noisy Label Radio' should be 'Noisy Label Ratio'; also state how label noise is generated (e.g., symmetric vs. asymmetric flip rates).","section":"Table 2"},{"comment":"The axes in Figure 3(a) are unlabeled and the caption does not define the plotted quantities; Figure 3(d) is captioned 'Variance of uncertainty' but the text refers to abrupt changes in adaptive classification points. Please reconcile the caption with the text.","section":"Figure 3(a) and Figure 3(d)"},{"comment":"Acar et al. 2021a and 2021b are the same arXiv item and should be distinguished or merged; there are also typographical issues such as 'our methodFedBSS' and 'V ollgraf' in the Fashion-MNIST reference.","section":"References and text"},{"comment":"The expression X^n_{t,e} = X^n_no + α X^n_bias is ambiguous: clarify whether this denotes a union of sample sets, a fractional subset of the biased set, or a weighted sampling distribution.","section":"Equation (6)"}],"recommendation":"major_revision","confidential_remarks":"The empirical gains in Tables 1 and 2 are real but the paper's central mechanism-specific evidence is missing. I would want to see the ranking-criterion ablation and a proper DomainNet comparison before considering acceptance; these additions are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you. This is an empirical FL paper with a real new angle: instead of calibrating local updates or fixing aggregation, it treats client drift as a per-sample phenomenon and trains each client on low-loss samples first, then progressively adds high-loss samples. That sample-level progressive selection is the genuinely new piece. It is orthogonal to SCAFFOLD/FedProx-style methods, and the experiments are broad: label skew (Shards, Dirichlet), feature skew (DomainNet), noisy labels, three datasets, two backbones. The gains in Tables 1 and 2 are consistent, not just one lucky setting. The authors also run an ablation on the schedule shape (filter vs linear vs cosine), which is more than many papers do.\n\nThe soft spot, and it is the load-bearing one, is that the drift mechanism is never actually tested. The paper identifies loss under the global model as the per-sample drift contribution and uses the maximum-uncertainty sample as the unbiased/biased boundary, but no experiment compares loss-based ranking against random ranking or a local-model ranking. The motivating experiment (Fig. 3a) is confounded: it trains on bottom-50% vs top-50% loss subsets that differ in size and label composition. Fig. 3b even shows a confident-but-wrong high-loss regime, which undercuts the claim that high uncertainty marks the most drift-contributing samples. So the large gains in Tables 1-2 may simply be an easy-to-hard curriculum effect, a known trick, not evidence about client drift. The warm-up length (50 rounds) is tuned on the same benchmarks, and the DomainNet result has no baseline comparison or error bars. These are fixable but real.\n\nOn the citation side, the references are normal and the related work is fair. The paper overclaims in the abstract relative to what the tables show in some settings, and there is a duplicated reference entry, but neither is central.\n\nBottom line: this is a plausible heuristic worth knowing for anyone working on data heterogeneity in FL, but it is not a proven conceptual result. A serious referee could turn it into a good paper by adding the missing ranking-ablation, multi-seed runs, and a more measured claim. I would send it to review rather than desk-reject, but I would not cite the drift story until the mechanism is actually shown.","headline":"A plausible sample-level curriculum for FL with broad empirical gains, but the drift-mitigation mechanism itself is never actually tested.","tokens_in":15462,"tokens_out":2147,"would_cite":false,"duration_ms":22378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that client drift in federated learning is the accumulated effect of per-sample biases, and that training clients on low-loss samples first and progressively adding high-loss samples reduces drift and improves accuracy…","keywords":["federated learning","client drift","sample-level bias","progressive learning","label distribution skew","feature distribution skew","noisy labels","uncertainty-based selection"],"falsifier":"On a heterogeneous benchmark, compute each sample's actual contribution to client drift (for example, the norm of the per-sample gradient projected against the global-update direction) and compare that ordering to the loss ordering FedBSS uses; if the two orderings disagree substantially on high-loss samples, FedBSS will be seen to add the wrong samples, and its accuracy should drop below FedAvg on a dataset engineered so that high-loss samples are the informative rare class.","tokens_in":14517,"feed_emoji":"🧠","tokens_out":5995,"duration_ms":54315,"temperature":0.7,"pith_summary":"The paper argues that client drift in federated learning is not a single monolithic error but the cumulative effect of per-sample biases, with each local sample contributing differently to how the local model departs from the global model. The authors propose FedBSS, a two-stage method: a warm-up stage that trains on all samples to build diverse knowledge, then a progressive stage that ranks a client's samples by their loss under the global model, splits them into 'unbiased' and 'biased' sets using a maximum-uncertainty boundary, and adds biased samples back epoch by epoch on a cosine schedule. The claim is that this sample-level ordering reduces drift and improves global accuracy under label skew, feature skew, and noisy labels, without extra communication between clients and server.","feed_headline":"Low-loss-first training beats federated baselines","feed_subtitle":"Progressive high-loss addition improves accuracy under skewed and noisy data.","key_machinery":"The central mechanism is FedBSS's two-stage schedule. Stage 1 is a diversified knowledge acquisition warm-up of fixed rounds (50 in the experiments) where the client trains on all samples. Stage 2, in each round, evaluates every local sample with the global model, sorts by loss, and sets the split point between unbiased and biased samples at the sample with maximum uncertainty $\\alpha(x;\\vartheta)=1-(\\max p - \\min p)$. Training then starts from the unbiased set and adds biased samples progressively through $\\alpha = (1-\\cos(e/e_{\\mathrm{total}}\\pi))/2$ per local epoch. The cosine schedule matters because the loss distribution is dense near the threshold and sparse at the edges, so a linear ramp fits the sample curve poorly. This mechanism is what converts the per-sample bias ordering into an actual training schedule.","core_discovery":"The central claim is that client drift can be understood and mitigated at the level of individual samples. The paper identifies that each local sample carries a different degree of bias, that the loss of a sample under the current global model measures that bias, and that the bias ordering changes as training progresses. FedBSS operationalizes this by sorting client samples by global-model loss, using the sample with maximum uncertainty as the adaptive threshold between low-bias ('unbiased') and high-bias ('biased') samples, and training first on low-loss samples while progressively mixing in high-loss samples with a cosine ramp. The paper reports that this outperforms existing drift-calibration and aggregation methods across Fashion-MNIST, CIFAR-10, CIFAR-100, and DomainNet, and remains effective when labels are noisy.","pith_inferences":["A natural extension is to replace the fixed 50-round warm-up with an uncertainty-stability criterion; the paper's own ablation shows performance is sensitive to warm-up length, but the paper does not derive a principled stopping rule.","The 'train low-loss first' ordering is a federated analogue of curriculum learning; it could be tested as a drop-in local schedule in centralized training, where the drift notion is replaced by generalization disparity.","Because high-loss samples under the global model are also the samples the model is least certain about, the same loss-plus-uncertainty machinery could be reused for federated active learning, selecting which samples to label or request from clients.","If loss ordering is noisy early in training, the uncertainty threshold may flicker across rounds; smoothing the threshold over time is a concrete testable tweak the paper does not explore."],"forward_implications":["If the central claim holds, client drift can be reduced by ordering samples within each client, making the method orthogonal to update-calibration and aggregation techniques, so it can be stacked with them.","The method requires no extra communication; the only added cost is locally sorting samples by loss, so it is applicable to communication-constrained federated systems.","The improvement grows with heterogeneity and with model and dataset scale, suggesting the benefit is largest precisely in the settings where client drift is most severe.","Under label noise the method's robustness increases with the noise ratio, implying that the biased-sample grouping absorbs noisy samples and contains their drift contribution.","The adaptive uncertainty threshold removes the need for a fixed loss threshold across rounds, clients, and datasets, so the method does not require tuning a per-task threshold."],"supporting_citations":[{"why":"Defines client drift and provides the SCAFFOLD baseline that FedBSS must beat.","marker":"(Karimireddy et al. 2020)"},{"why":"FedProx introduces the proximal regularization baseline for heterogeneous federated learning.","marker":"(Li et al. 2020)"},{"why":"FedLC provides the logits-calibration baseline under label distribution skew.","marker":"(Zhang et al. 2022)"},{"why":"FedRS provides a restricted-softmax baseline for missing-class label skew.","marker":"(Li and Zhan 2021)"},{"why":"FedExP is an extrapolation-based speedup baseline compared in the experiments.","marker":"(Jhunjhunwala, Wang, and Joshi 2023)"},{"why":"InCo is a server aggregation baseline using internal cross-layer gradients.","marker":"(Chan et al. 2024)"},{"why":"FedCDA is a cross-round divergence-aware aggregation baseline.","marker":"(Wang et al. 2024)"},{"why":"Provides the Shards data partition protocol used to create label skew.","marker":"(McMahan et al. 2017)"},{"why":"Inspires the uncertainty measure used to adaptively split unbiased and biased samples.","marker":"(Fuchsgruber et al. 2024)"},{"why":"Precedent for sampling by uncertainty and density, adapted here for the adaptive classification point.","marker":"(Zhu et al. 2008)"}],"fun_headline_variants":["Sample-level bias selection curbs federated drift","Train on low-loss samples first to stop federated drift","Progressive high-loss addition tames federated client drift","Adaptive sample selection by loss reduces federated drift","FedBSS: pick low-loss samples first to beat FL baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a sample's loss under the current global model faithfully measures how much that sample pushes the local update away from the global update; if loss ordering and drift contribution diverge, the schedule keeps harmful samples and discards useful ones.","fun_headline_variants_meta":{"raw":{"variants":["Sample-level bias selection curbs federated drift","Train on low-loss samples first to stop federated drift","Progressive high-loss addition tames federated client drift","Adaptive sample selection by loss reduces federated drift","FedBSS: pick low-loss samples first to beat FL baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3157,"prompt_tokens":956,"completion_tokens":2201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2120}},"tokens_in":572,"tokens_out":2201,"duration_ms":15190,"temperature":1.0,"reasoning_tokens":2120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:21:10.979620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a heterogeneous benchmark, compute each sample's actual contribution to client drift (for example, the norm of the per-sample gradient projected against the global-update direction) and compare that ordering to the loss ordering FedBSS uses; if the two orderings disagree substantially on high-loss samples, FedBSS will be seen to add the wrong samples, and its accuracy should drop below FedAvg on a dataset engineered so that high-loss samples are the informative rare class.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FedExP is an extrapolation-based speedup baseline compared in the experiments."},{"cited_title":"P.; Kale, S.; Mohri, M.; Reddi, S.; Stich, S.; and Suresh, A","cited_arxiv_id":null,"evidence_quote":"Defines client drift and provides the SCAFFOLD baseline that FedBSS must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FedLC provides the logits-calibration baseline under label distribution skew."},{"cited_title":"Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning","cited_arxiv_id":"2308.11464","evidence_quote":"InCo is a server aggregation baseline using internal cross-layer gradients."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FedCDA is a cross-round divergence-aware aggregation baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Shards data partition protocol used to create label skew."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Precedent for sampling by uncertainty and density, adapted here for the adaptive classification point."}],"review_version":1}