REVIEW 4 major objections 5 minor 29 references
A server can assess each client's data quality in federated learning by scoring how well that client's locally trained model performs on the server's own test set, then adapt client selection between fairness and quality.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Federated learning client quality can be scored by testing each client's model on the server's own data, and selection fairness should be dialed down when quality scores vary widely.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection Worth reading for the empirical fairness-versus-mislabeling results, but the proposed adaptive rule is not well-defined and is never evaluated, so the method part of the paper is still a sketch. the 4 major comments →
Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that a server-based Quality Score, defined as the min-max normalized test accuracy of each client's locally trained model on the server's own labeled test set, is a valid indicator of client dataset quality under both label skew and mislabeling. Measured against 100-client MNIST and CIFAR-10 setups with sequential mislabeling as the stress test, the score rises with label diversity and falls with mislabeling rate. Alongside a Datasize Score and a Fairness Score that accumulates for unselected clients, the adaptive selection rule uses the variance of Quality Scores to decide how many slots go to fairness: low variance means mild heterogeneity and fairness is prioritized;
What carries the argument
Quality Score (Eq. 2): min-max normalized server-side test accuracy of each client's uploaded model, the privacy-preserving proxy for dataset quality; Fairness Score (Eq. 3): a per-client counter that grows by beta each round and drops by 1 when selected, so long-unselected clients become hard to ignore; and the adaptive rule (Eq. 4): the number of fairness-picked clients expands with the variance of Quality Scores among available clients. Together they let selection treat quality and fairness as tunable rather than fixed, and the variance term is what decides which way to lean.
Load-bearing premise
The server's test set represents the learning target, and a client's score on it rises monotonically with private-data quality regardless of local training effort, model capacity, or deliberate manipulation.
What would settle it
Give two clients identical data quality but different numbers of local epochs; if the longer-trained client consistently gets a higher Quality Score, the score is measuring training effort rather than data quality, and the ranking that drives selection is not what the paper claims.
If this is right
- A server can rank clients by data quality for selection without clients disclosing label distributions, feature distributions, or raw data.
- When mislabeling dominates, selecting the highest-quality clients rather than maximizing fairness raises final test accuracy; when label skew dominates, fair selection raises accuracy but requires more rounds.
- Quality Score variance is a usable real-time signal for adapting selection policy, since it tracks the degree of heterogeneity among available clients.
- The scoring scheme layers onto existing aggregation: it only changes which clients are picked, not how updates are combined.
- Handling the hardest mislabeling pattern (sequential relabeling) transfers to milder random and cyclic mislabeling, which degrade accuracy less.
Where Pith is reading between the lines
- The paper assumes all clients use the same local training epochs and model capacity; if a server allows heterogeneous local training, test accuracy conflates data quality with training effort, so a testable extension is to measure Quality Score under varied local epochs.
- The server's test set is trusted to represent the deployment distribution; a natural stress test is to shift the server test set away from client data distributions and check whether the ranking still tracks true data quality.
- Because clients can, in principle, optimize their uploaded models against the server's test set, a practical extension would add holdout or perturbed queries to detect test-set memorization.
- The authors flag that Quality Score cannot separate label skew from mislabeling; a future metric that disentangles them (e.g., by probing logit statistics or auxiliary tasks) would let the adaptive rule lean fairness even under high variance caused by label skew.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how imperfect client datasets (quantity skew, label skew, and three types of mislabeling) affect accuracy and convergence in federated learning, and how fairness in client selection interacts with these imperfections. Experiments on MNIST and CIFAR-10 show that label skew hurts CIFAR-10 more than MNIST, that mislabeling is more damaging than label skew, that fair/random selection helps under label skew but hurts when many clients have mislabeled data, and that severe sequential mislabeling can cause learning to fail. The paper then proposes a privacy-preserving client assessment scheme: a Datasize Score, a Quality Score computed by normalizing server-side test accuracy of each client's uploaded model, and a Fairness Score, with an adaptive rule (Eq. 4) that varies the number of clients selected by fairness according to the variance of Quality Scores. The Quality Score is validated by showing it increases with the number of labels per client and decreases with mislabeling rate. The paper concludes by acknowledging that the Quality Score cannot distinguish label skew from mislabeling.
Significance. If established, the paper's empirical findings would provide a useful, clearly described characterization of when fairness in client selection helps versus hurts, and the proposed Quality Score offers a simple server-side, privacy-preserving proxy for client data quality without accessing raw client data. The experimental design is transparent (100 clients, FedAvg, parameter table, two datasets, multiple skew/mislabel scenarios), and the authors explicitly concede the main limitation of their adaptive mechanism. However, the central adaptive selection rule is not evaluated end-to-end and contains a definitional inconsistency, so the claimed benefit of the proposed client-selection mechanism is not currently supported. The empirical impact study, while suggestive, also lacks error bars and has a caption inconsistency.
major comments (4)
- [§IV-C, Eq. (4)] Eq. (4) is internally inconsistent with Table I. Nmax is defined as the maximum number of clients to be selected, but L^t_fair = max(Nmax, |C_ava|(α1·Var(q)+α2)) makes Nmax a lower bound. With α2>0, L^t_fair can exceed Nmax, violating the stated constraint. If max was intended to be min, the surrounding interpretation ('α2 ensures a minimum proportion') would need to change. Moreover, the composition of the final Nmax-client set is not specified: selecting top L^t_fair clients by Fairness Score and then somehow choosing Nmax among them is not defined. This is a load-bearing flaw because the adaptive mechanism is the paper's proposed client-selection contribution.
- [§IV-C, §V] The proposed adaptive client-selection mechanism is never run end-to-end. No experiment compares the full rule (Eq. 4) against random, fair-only, or quality-only selection. The experiments in Figs. 8–9 validate only the Quality Score's ranking correlation, not the selection policy's effect on model accuracy or convergence. Consequently, the claims in the Introduction and Conclusion that the proposed scoring/selection mechanism improves client selection are unsupported. An end-to-end experiment is needed, especially because the authors themselves note the rule 'may be suboptimal when label skew dominates'—one of the paper's motivating scenarios.
- [§IV-B, Eq. (2), Figs. 8–9] The validation of Quality Score is essentially circular: Quality Score is defined as normalized server-side test accuracy, and the validation shows that this normalized accuracy correlates with the constructed ground truth (number of labels or mislabeling rate). This is expected by construction. The more important claim—that test accuracy is a monotone indicator of dataset quality—is not tested against confounds such as non-representative server test sets, heterogeneous local training epochs, or clients manipulating local training. Since the mechanism relies on this monotonicity, the paper should either add control experiments (varying server test set partition, local epochs) or explicitly narrow the claim to the specific experimental setting.
- [§III-D, Figs. 5–7] The empirical claims of 'significantly improved' and 'significantly reduced performance' are based on single learning curves averaged over ten experiments, but no error bars, standard deviations, or statistical tests are reported. Also, Fig. 4's caption says 'with 90 clients' while Table I and the rest of the paper state 100 clients. This inconsistency, together with missing variance information, makes it difficult to assess the reliability and reproducibility of the headline empirical findings.
minor comments (5)
- [Fig. 1(a)] Legend typo: 'SingelLabel' should be 'SingleLabel'.
- [Fig. 4] Caption says 'with90clients' but the experimental setting consistently uses 100 clients; verify which is correct.
- [Eq. (4)] The notation for the variance term is ambiguous: the summation expression should be parenthesized clearly as (1/|C_ava|) Σ (q_i − q)^2, not left as a loose fraction.
- [§IV-C] After defining C^t_fair ⊆ C^t_ava as 'the set of selected clients', the text immediately says 'the mechanism recruits few clients... in C^t_fair' but does not say whether C^t_fair is the final selected set or an intermediate pool; clarify this relationship to L^t_fair and Nmax.
- [General] Several figure legends use inconsistent spacing (e.g., 'SingelLabel' in Fig. 1, 'with90clients' in Fig. 4). A careful proofread is recommended.
Circularity Check
No significant circularity: Quality Score is a defined proxy (normalized server-side test accuracy), not a derived or fitted prediction; the adaptive-rule problems are consistency/validation issues, not circular reasoning.
full rationale
The central quantity, Quality Score (Eq. 2), is defined directly as min-max normalized test accuracy on the server's own test set. It is not obtained by fitting parameters to the evaluation outcome, and no claimed prediction reduces to fitted values: α1, α2, and β are free design constants. The validation experiments (Figs. 8–9) compare this defined proxy against controlled label-skew and mislabeling settings, which tests an empirical monotonicity assumption rather than disguising a fitted input as a prediction. The server-based assessment is credited to external prior work [23], not to a self-citation chain, and no uniqueness theorem or ansatz is imported from the authors' own previous work. The serious problems in Section IV-C—Eq. (4) uses max with Nmax despite Nmax being defined as the maximum number of clients to select, the composition of the final Nmax-client set from Fairness and Quality rankings is unspecified, and no experiment runs Eq. (4) end-to-end—are failures of internal consistency and validation, not circularity. The paper also explicitly concedes that Quality Score cannot distinguish label skew from mislabeling and may be suboptimal when label skew dominates, which is a stated limitation rather than a circular step. Under the requirement that circularity be exhibited as a specific reduction—such as an equation equal by construction or a fitted parameter renamed as a prediction—no such reduction is present.
Axiom & Free-Parameter Ledger
free parameters (4)
- β (Fairness Score increment)
- α1 (quality-variance sensitivity)
- α2 (minimum participation proportion)
- Local evaluation epochs =
30
axioms (4)
- domain assumption The server holds a labeled test dataset representative of the task and can evaluate each client's uploaded model on it.
- domain assumption Test accuracy on the server's test set is a monotone indicator of client dataset quality with respect to label skew and mislabeling.
- ad hoc to paper High variance of Quality Score across clients means selection should prioritize quality over fairness.
- standard math Under IID data, FedAvg's expected update equals the centralized gradient update on the union of all data.
Cite this review
Pith. "Pith review of Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning." pith.science (2026). https://pith.science/paper/ARC2OSDV
@misc{pith2026260802250,
author = {Pith},
title = {Pith review of: Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ARC2OSDV}},
note = {Machine review of arXiv:2608.02250}
}
read the original abstract
Federated learning (FL) is a popular distributed learning framework where multiple clients perform local training and a server aggregates the locally updated models. FL enables decentralized training while preserving the privacy of clients' datasets. However, non-independent and identically distributed (non-IID) or noisy datasets can lead to low model accuracy or high convergence latency. Precluding these clients through client selection may mitigate the problem, but heavily biased client selections may also degrade the learning performance. In this study, we first experimentally measure the impact of non-IID data (including skews in data quantity and label distribution), noisy data, and fairness in client selection on model accuracy and convergence. We then propose a privacy-preserving scoring method to assess each client's contribution in FL, with experiments conducted to demonstrate the effectiveness of the proposed assessment.
Figures
Reference graph
Works this paper leans on
-
[1]
Toward data heterogeneity of federated learning,
Y . Huang and C. Hu, “Toward data heterogeneity of federated learning,”arXiv:2212.08944, Dec. 2022
Pith/arXiv arXiv 2022
-
[2]
Federated learning with non-IID data,
Y . Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra, “Federated learning with non-IID data,” arXiv:1806.00582, 2018
Pith/arXiv arXiv 2018
-
[3]
Towards understanding biased client selection in federated learning,
Y . Cho, J. Wang, and G. Joshi, “Towards understanding biased client selection in federated learning,” inProc. 25th Int’l Conf. on Artificial Intelligence and Statistics, vol. 151, Mar. 2022, pp. 10 351–10 375
2022
-
[4]
Fairness-aware client selection for federated learning,
Y . Shi, Z. Liu, Z. Shi, and H. Yu, “Fairness-aware client selection for federated learning,”arXiv:2307.10738, 2023
Pith/arXiv arXiv 2023
-
[5]
Federated learning on non-IID data silos: An experimental study,
Q. Li, Y . Diao, Q. Chen, and B. He, “Federated learning on non-IID data silos: An experimental study,” inIEEE 38th Int’l Conf. on Data Engineering, May 2022
2022
-
[6]
Communication-efficient learning of deep networks from decentralized data,
H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inProc. 20th Int’l Conf. on Artificial Intelligence and Statistics, Apr. 2017, pp. 1273–1282
2017
-
[7]
Towards addressing label skews in one-shot federated learning,
Y . Diao, Q. Li, and B. He, “Towards addressing label skews in one-shot federated learning,” inProc. Int’l Conf. on Learning Representations, 2023
2023
-
[8]
Multi-task federated learning for personalised deep neural networks in edge computing,
J. Mills, J. Hu, and G. Min, “Multi-task federated learning for personalised deep neural networks in edge computing,”IEEE Trans. Parallel Distrib. Syst., vol. 33, no. 3, pp. 630–641, Mar. 2022
2022
-
[9]
Personalized cross-silo federated learning on non-IID data,
Y . Huang, L. Chu, Z. Zhou, L. Wang, J. Liu, J. Pei, and Y . Zhang, “Personalized cross-silo federated learning on non-IID data,”Proc. AAAI Conf. on Artificial Intelli- gence, vol. 35, no. 9, pp. 7865–7873, Jul. 2020
2020
-
[10]
FedClust: Optimizing federated learning on non-IID data through weight-driven client clustering,
M. S. Islam, S. Javaherian, F. Xu, X. Yuan, L. Chen, and N.-F. Tzeng, “FedClust: Optimizing federated learning on non-IID data through weight-driven client clustering,” inIEEE Int’l Parallel and Distributed Processing Symp. Workshops, San Francisco, CA, USA, May 2024
2024
-
[11]
Multi-center federated learning: Clients clustering for better personalization,
G. Long, M. Xie, T. Shen, T. Zhou, X. Wang, J. Jiang, and C. Zhang, “Multi-center federated learning: Clients clustering for better personalization,”World Wide Web, vol. 26, pp. 481–500, Feb. 2023
2023
-
[12]
FedDRL: Deep reinforcement learning- based adaptive aggregation for non-IID data in federated learning,
N. H. Nguyen, P. L. Nguyen, T. D. Nguyen, T. T. Nguyen, D. L. Nguyen, T. H. Nguyen, H. H. Pham, and T. N. Truong, “FedDRL: Deep reinforcement learning- based adaptive aggregation for non-IID data in federated learning,” inProc. ICPP, 2023
2023
-
[13]
Deep reinforcement learning based scheduling strategy for federated learning in sensor-cloud systems,
T. Zhang, K.-Y . Lam, and J. Zhao, “Deep reinforcement learning based scheduling strategy for federated learning in sensor-cloud systems,”Future Gener. Comput. Syst., vol. 144, pp. 219–229, Jul. 2023
2023
-
[14]
An optimization method for non-IID federated learning based on deep reinforcement learning,
X. Meng, Y . Li, J. Lu, and X. Ren, “An optimization method for non-IID federated learning based on deep reinforcement learning,”Sensors, vol. 23, no. 22, 2023
2023
-
[15]
MNIST handwritten digit database,
Y . LeCun, C. Cortes, and C. Burges, “MNIST handwritten digit database,”ATT Labs. Available: http://yann.lecun.com/exdb/mnist, 2010
2010
-
[16]
Learning multiple layers of features from tiny images,
A. Krizhevsky, “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep., 2009
2009
-
[17]
Mitigating sybils in federated learning poisoning,
C. Fung, C. J. Yoon, and I. Beschastnikh, “Mitigating sybils in federated learning poisoning,” arXiv:1808.04866, Jul. 2020
Pith/arXiv arXiv 2020
-
[18]
Robust federated learning with noisy labels,
S. Yang, H. Park, J. Byun, and C. Kim, “Robust federated learning with noisy labels,”IEEE Intell. Syst., vol. 37, no. 2, pp. 35–43, Mar.-Apr. 2022
2022
-
[19]
FedDiv: Collaborative noise filtering for federated learning with noisy labels,
J. Li, G. Li, H. Cheng, Z. Liao, and Y . Yu, “FedDiv: Collaborative noise filtering for federated learning with noisy labels,”arXiv:2312.12263, Feb. 2024
Pith/arXiv arXiv 2024
-
[20]
FedCorr: Multi-stage federated learning for label noise correction,
J. Xu, Z. Chen, T. Q. Quek, and K. F. E. Chong, “FedCorr: Multi-stage federated learning for label noise correction,” inProc. CVPR, Jun. 2022
2022
-
[21]
CLC: A consensus-based label correction approach in federated learning,
B. Zeng, X. Yang, Y . Chen, H. Yu, and Y . Zhang, “CLC: A consensus-based label correction approach in federated learning,”ACM Trans. Intell. Syst. Technol., vol. 13, no. 5, pp. 1–23, Jun. 2022
2022
-
[22]
Client selection for federated learning with label noise,
M. Yang, H. Qian, X. Wang, Y . Zhou, and H. Zhu, “Client selection for federated learning with label noise,” IEEE Trans. Veh. Technol., vol. 71, no. 2, pp. 2193–2197, Feb. 2022
2022
-
[23]
AUCTION: Automated and quality-aware client selection framework for efficient federated learning,
Y . Deng, F. Lyu, J. Ren, and H. Wu, “AUCTION: Automated and quality-aware client selection framework for efficient federated learning,”IEEE Trans. on Parallel and Distributed Systems, vol. 33, no. 8, pp. 1996–2009, Aug. 2022
1996
-
[24]
Labeling chaos to learning harmony: Federated learning with noisy labels,
V . Tsouvalas, A. Saeed, T. Ozcelebi, and N. Meratnia, “Labeling chaos to learning harmony: Federated learning with noisy labels,”ACM Trans. Intell. Syst. Technol., vol. 15, no. 2, pp. 1–26, Feb. 2024
2024
-
[25]
Towards fairness-aware federated learning,
Y . Shi, H. Yu, and C. Leung, “Towards fairness-aware federated learning,”IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 9, pp. 11 922–11 938, Sep. 2024
2024
-
[26]
Stochastic client selection for federated learning with volatile clients,
T. Huang, W. Lin, L. Shen, K. Li, and A. Y . Zomaya, “Stochastic client selection for federated learning with volatile clients,”IEEE Internet Things J., vol. 9, no. 20, pp. 20 055–20 070, Oct. 2022
2022
-
[27]
An efficiency-boosting client selection scheme for federated learning with fairness guarantee,
T. Huang, W. Lin, W. Wu, L. He, and K. Li, “An efficiency-boosting client selection scheme for federated learning with fairness guarantee,”IEEE Trans. Parallel Distrib. Syst., vol. 32, no. 7, pp. 1552–1564, Jul. 2021
2021
-
[28]
Lyapunov-based optimization of edge resources for energy-efficient adaptive federated learning,
C. Battiloro, P. D. Lorenzo, M. Merluzzi, and S. Bar- barossa, “Lyapunov-based optimization of edge resources for energy-efficient adaptive federated learning,”IEEE Trans. Green Commun. Netw., vol. 7, no. 1, pp. 265– 280, Mar. 2023
2023
-
[29]
Online client selection for asynchronous fed- erated learning with fairness consideration,
H. Zhu, Y . Zhou, H. Qian, Y . Shi, X. Chen, and Y . Yang, “Online client selection for asynchronous fed- erated learning with fairness consideration,”IEEE Trans. Wireless Commun., vol. 22, no. 4, pp. 2493–2506, Apr. 2023
2023
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.