Pith. sign in

REVIEW 3 major objections 4 minor 28 references

Federated Learning with Heterogeneous and Private Label Sets

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated learning can keep client label sets private with little accuracy loss.

desk verdict A useful first study of private label sets in FL with a clean per-label averaging adaptation, but the headline parity claim is not quantified in the text and the figures suggest a non-trivial gap at low label counts. read the letter →

arxiv 2508.18774 v1 pith:34HV4AGJ submitted 2025-08-26 cs.LG stat.ML

Adam Breitholtz , Edvin Listo Zec , Fredrik D. Johansson This is my paper
classification cs.LGstat.ML
keywords federatedlearninglabelsetheterogeneityprivatelabelssoftmaxaveragingclassifiercombinationrepresentationalignmentFedAvgadaptationcentraltuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies federated learning when clients hold different, private subsets of the global label set—each client knows only its own labels, and label sets are shared only with the central server. It proposes two ways to handle this: a simple adaptation of FedAvg and FedProx that averages classifier parameters per label over the clients that hold that label, and a server-side tuning procedure that aligns client representations using an unlabeled dataset. The paper's claim is that the simple per-label averaging adaptation performs nearly as well in the private setting as standard methods achieve when label sets are public. If true, clients can withhold their label sets from each other at little cost to model accuracy.

What carries the argument

The load-bearing mechanism is the restricted-softmax identity (2): for a softmax classifier, conditioning on membership in a client's label set Y_k cancels the global normalization, leaving a softmax over Y_k with the same class parameter vectors. This makes per-label averaging (Algorithm 1) unbiased under Assumption 1 and turns classifier combination into a per-label weighting problem. A second mechanism is the pairwise tuning loss (equation 5), which generalizes binary pairwise coupling to multi-class subset classifiers by penalizing disagreement between client and central models on pairs of labels the client holds.

What would settle it

A concrete test: build a federation where the global conditional distribution p(y|x) is known, split labels among clients so Assumption 1 holds, and check that per-label averaged parameters recover the global softmax parameters exactly (up to noise). Then reverse the experiment: make each client's labeling mechanism differ from the conditioned global mechanism (e.g., different label semantics for the same input), and observe whether the adapted FedAvg's accuracy drops relative to a public-label baseline. If accuracy does not drop when Assumption 1 is violated, the assumption is not the load-be

Watch

Extended reading notes

Core claim

The paper's central claim is that private label sets—where each client knows only its own subset of labels and the union is not revealed—can be handled by adapting standard parameter-averaging methods. The key identity is equation (2): under a subset-consistency assumption on client labeling mechanisms, the client's restricted softmax classifier over its own labels has exactly the same parameters as the global classifier restricted to those labels, so any convex combination of client classifier estimates is unbiased for the global classifier. The paper proves a perfect-classifier-combination proposition (Proposition 1) showing that consistent client models can be combined into a perfect cent

Load-bearing premise

The derivation that per-label averaging is unbiased rests on Assumption 1—that each client's label mechanism is simply the global mechanism conditioned on the client's label set—together with the softmax classifier being correct under an optimal encoder.

Editorial extensions

If this is right

  • Clients can keep label sets private from each other while retaining most of the accuracy of a public-label federation.
  • Simple per-label averaging of classifier parameters is a strong baseline for private heterogeneous labels, potentially removing the need for more complex alignment methods.
  • Server-side tuning on unlabeled data provides an alternative for alignment but requires the server to hold an unlabeled sample from the target marginal distribution.
  • The pairwise tuning loss is fragile when a label is absent from all clients in a round, pushing the central model to zero out the absent label; MSE tuning is more robust in the private setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The subset-consistency assumption is what makes per-label averaging work; in real federations where clients label data by different criteria (e.g., different annotation guidelines), the assumption fails and the benefit of the adaptation may shrink. A synthetic test that deliberately breaks subset-consistency would quantify this.
  • The per-class reweighting scheme could be extended beyond softmax classifiers to any model class where conditioning on label subset acts as a simple restriction on parameters, such as structured-output models.
  • The tuning approach suggests a general recipe: use the server's unlabeled data to tie together client-specific prediction heads, which could also apply to federated learning with non-identical input spaces or feature distributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies federated learning with heterogeneous and private label sets: each client observes only a subset Y_k of the global label set and is unaware of the other labels, while the server knows all labels. The authors propose (i) a per-label averaging adaptation of FedAvg/FedProx for private labels (Algorithm 1), justified under a subset-consistency assumption when the softmax model is well-specified, and (ii) server-side central tuning of the classifier on an unlabeled dataset using pairwise and MSE losses. Experiments on CIFAR-10 and Fashion-MNIST vary the number of labels per client from 2 to 10, comparing private-adapted methods with public-label baselines. The abstract claims that the proposed private adaptations perform similarly to standard public-label methods, implying that clients can retain label privacy at little accuracy cost.

Significance. If the empirical parity claim held, the paper would provide a simple, communication-efficient baseline for private label-set FL: Algorithm 1 is parameter-free and requires only per-label server averaging, while central tuning offers an alternative when unlabeled data are available. The theoretical derivation in Section 4.1 is clean under explicit assumptions, and Proposition 1 gives a principled perspective on classifier combination. The paper is transparent about limitations (e.g., pairwise-loss failure with missing labels, tuning variance). However, the central quantitative claim is currently not verifiable from the manuscript, so the significance is conditional on the requested empirical clarification.

major comments (3)
  1. [§5, Figures 2–3; Abstract] The headline claim that private adaptations achieve 'similar performance' to public standard methods is not supported by quantitative evidence. The paper reports only plots with bootstrapped CIs; no numeric test accuracies, mean differences, or significance tests across the 10 seeds are given. At low label counts (e.g., CIFAR-10, 2–4 labels per client), the private-adapted FedAvg appears several accuracy points below public FedAvg, which is hard to reconcile with 'little cost to model accuracy.' Please add tables of mean accuracies and CIs for every method and label count, and a direct measure of the private-vs-public gap (e.g., mean difference with CI).
  2. [§5, first paragraph] The experimental setup is underspecified. The number of clients m is not stated, nor is it stated whether the randomly sampled label sets are forced to cover all labels (∪_k Y_k = Y) in each seed. With random label sets and small per-client label counts, some labels will be absent from the federation in many seeds, making those classes unlearnable and confounding the comparison. The paper's own Appendix B analyzes a missing-label scenario, so this is not a remote concern. Please state m, the label-sampling distribution, whether coverage is enforced (and how), and how many seeds are affected by missing labels.
  3. [§4.1, Eq. (2)] The unbiased-averaging argument relies on the softmax being well-specified under an optimal encoder and on client estimates being unbiased. The former is acknowledged to fail in practice, but the latter is stated without qualification: 'any convex combination of unbiased estimates... is unbiased.' Client softmax parameter estimates are only asymptotically unbiased in general. Please qualify the statement as an asymptotic/consistency claim and discuss the finite-sample caveat in the theoretical justification.
minor comments (4)
  1. [Algorithm 1] The reverse index map I_k is used but not formally defined. Specify that I_k maps a global label y ∈ Y_k to the corresponding local index in the client's classifier, and similarly for the reverse.
  2. [Figure 8 caption] Figure 8 is captioned 'FashionMNIST task' but the displayed accuracy range and its placement after the CIFAR-10 private figure suggest it should be 'CIFAR10 task, public labels.' Please correct the caption (and check the other captions in Appendix C.1 for consistency).
  3. [§5.1, CIFAR10 paragraph] The sentence 'Tuning with MSE loss seems to struggle here while, in the private setting, it performs the best' is ambiguous. Clarify the referents of 'here' and 'it' to avoid confusion between the public and private settings.
  4. [§6, Discussion] The phrase 'This may be due to the alignment problem being harder with an increasing amount of labels' seems in tension with the earlier statement that tuning helps when sparsity is more extreme. Rephrase to distinguish the number of local labels from the total number of classes.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: per-label averaging and both tuning losses are derived from stated assumptions and evaluated against external test splits; the only self-citation [22] is a non-load-bearing standard variance fact.

full rationale

The paper's central claims are (1) an adaptation of FedAvg/FedProx for private label sets via per-label averaging of classifier parameters, theoretically justified in Section 4.1, and (2) server-side tuning losses (MSE and pairwise) using unlabeled data, justified in Section 4.2, plus an empirical parity claim against public-setting baselines. I find no circular step. The derivation chain in Section 4.1 is genuine: Assumption 1 (subset-consistent labeling mechanisms) together with softmax well-specification under an optimal encoder yields eq. (2), whereby each client's restricted classifier is a softmax over the same global parameter vector restricted to Yk. The conclusion that client-optimal parameters coincide with the corresponding server-optimal entries, and that any convex combination of unbiased client estimates is unbiased for the server vector, follows algebraically from those stated assumptions; the conclusion is not embedded in the premises. Proposition 1 and the pairwise loss (4)-(5) are likewise derived from Assumption 1, with the proof given and attribution to the external classifier-combination literature [6,20]. The empirical evaluation is against external benchmarks: tuning uses separate unlabeled sets (5000/6000 samples) and held-out standard test splits (10000 samples), and no fitted constant is renamed as a prediction — test accuracy is measured, not recovered from a fitted quantity. The only self-citation is [22] (Zec, Breitholtz, Johansson — two of the present authors), used in Section 3 for a remark about targeted federated learning and in Section 4.1 for 'Client weighting based on sample size (as in Algorithm 1) achieves the largest effective sample size (smallest variance)'. The latter is a parameter-free classical statistical fact, and neither use carries the central parity claim; per the review rules it does not raise the circularity score. The score of 2 reflects solely the presence of this minor, non-load-bearing self-citation. In-scope manuscript text was checked: Appendix B openly flags that the pairwise loss produces problematic gradients when a label is globally absent — a limitation, not a circular step. The abstract's 'similar performance' claim lacks numeric confidence intervals, which is an evidentiary/verifiability concern, but it does not constitute equivalence-by-construction. Overall verdict: no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central method rests primarily on Assumption 1 and the well-specified softmax assumption; the tuning methods additionally require server-side unlabeled data. No new physical or conceptual entities are introduced; the paper's contribution is an adaptation and empirical study.

assumptions (3)
  • domain assumption Subset-consistent labeling mechanisms (Assumption 1)
    Central to the unbiasedness of per-label averaging; states that client conditional label distributions given the subset are identical to the global conditional given the same subset. Cited in Section 4.1, used in proof of Proposition 1. If false, the averaging justification fails.
  • domain assumption Softmax classifier is well-specified under an optimal encoder
    Required for equation (2) to hold exactly; the paper states this before deriving the result. In practice, neural networks are approximations, so this is an idealization.
  • domain assumption Existence of an unlabeled dataset at the server for tuning methods
    The tuning losses (5) and (6) require samples from p(X) at the server. This is stated in Section 4.2 as a requirement, and is an extra burden not needed by averaging methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Learning with Heterogeneous and Private Label Sets." pith.science (2026). https://pith.science/paper/34HV4AGJ

@misc{pith2026250818774,
  author       = {Pith},
  title        = {Pith review of: Federated Learning with Heterogeneous and Private Label Sets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/34HV4AGJ}},
  note         = {Machine review of arXiv:2508.18774}
}
read the original abstract

Although common in real-world applications, heterogeneous client label sets are rarely investigated in federated learning (FL). Furthermore, in the cases they are, clients are assumed to be willing to share their entire label sets with other clients. Federated learning with private label sets, shared only with the central server, adds further constraints on learning algorithms and is, in general, a more difficult problem to solve. In this work, we study the effects of label set heterogeneity on model performance, comparing the public and private label settings -- when the union of label sets in the federation is known to clients and when it is not. We apply classical methods for the classifier combination problem to FL using centralized tuning, adapt common FL methods to the private label set setting, and discuss the justification of both approaches under practical assumptions. Our experiments show that reducing the number of labels available to each client harms the performance of all methods substantially. Centralized tuning of client models for representational alignment can help remedy this, but often at the cost of higher variance. Throughout, our proposed adaptations of standard FL methods perform well, showing similar performance in the private label setting as the standard methods achieve in the public setting. This shows that clients can enjoy increased privacy at little cost to model accuracy.

Figures

Figures reproduced from arXiv: 2508.18774 by the authors.

Figure 1
Figure 1. a): A schematic view of the two settings which we consider in our work. The private setting where the clients are unaware of the full label set and the public setting where this is known. b): Results on CIFAR10 where each client has 5 labels available in their respective dataset. The tuning methods with MSE and Pairwise losses perform the best in the private setting. Errorbars represent a 95% confidence interval. No… view at source ↗
Figure 2
Figure 2. The performance of the methods in both the private (left) and public (right) settings on CIFAR10. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The performance of the methods in both the private and public settings on FashionMNIST. We note [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The results for the private and public settings for 3 labels per client on the FashionMNIST dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Test accuracy over rounds in the FashionMNIST task. The labels sets are private. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Test accuracy over rounds in the FashionMNIST task. The labels sets are public. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Test accuracy over rounds in the CIFAR10 task. The labels sets are private. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Test accuracy over rounds in the FashionMNIST task. The labels sets are Public. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The results for the private and public settings for 5 labels per client on the FashionMNIST dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: The results for the private and public settings with 3 labels per client on the CIFAR10 dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Ablation with differing amounts of local epochs on CIFAR10. We do not observe any particular drop [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Comparison of different labels per client on CIFAR10 with E=5. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Comparison of different labels per client on CIFAR10 with E=3. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 23 canonical work pages

  1. [1]

    Federated meta-learning with fast convergence and efficient communication.arXiv:1802.07876, 2018

    Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication.arXiv:1802.07876, 2018

    arXiv 2018

  2. [2]

    Scale federated learning for label set mismatch in medical image classification

    Zhipeng Deng, Luyang Luo, and Hao Chen. Scale federated learning for label set mismatch in medical image classification. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, and Russell Taylor, editors,Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pages 118–127, 2023

    work page 2023

  3. [3]

    Adverse drug reactions: definitions, diagnosis, and management

    I Ralph Edwards and Jeffrey K Aronson. Adverse drug reactions: definitions, diagnosis, and management. The lancet, 356(9237):1255–1259, 2000

    work page 2000

  4. [4]

    An efficient framework for clustered federated learning

    Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586–19597, 2020

    work page 2020

  5. [5]

    Federated Learning with Heterogeneous Labels and Models for Mobile Activity Monitoring

    Gautham Krishna Gudur and Satheesh K Perepu. Federated learning with heterogeneous labels and models for mobile activity monitoring.arXiv preprint arXiv:2012.02539, 2020

    work page Pith review arXiv 2012

  6. [6]

    Classification by pairwise coupling.Advances in neural information processing systems, 10, 1997

    Trevor Hastie and Robert Tibshirani. Classification by pairwise coupling.Advances in neural information processing systems, 10, 1997

    work page 1997

  7. [7]

    Fedcrl: Personalized federated learning with contrastive shared representations for label heterogeneity in non-iid data.arXiv preprint arXiv:2404.17916, 2024

    Chenghao Huang, Xiaolu Chen, Yanru Zhang, and Hao Wang. Fedcrl: Personalized federated learning with contrastive shared representations for label heterogeneity in non-iid data.arXiv preprint arXiv:2404.17916, 2024

    arXiv 2024

  8. [8]

    Federated semi-supervised learning with inter-client consistency & disjoint learning

    Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. InInternational Conference on Learning Representations,

    work page

Show all 28 references
  1. [9]

    Improving federated learning personal- ization via model agnostic meta learning.arXiv preprint arXiv:1909.12488, 2019

    Yihan Jiang, Jakub Konečn` y, Keith Rush, and Sreeram Kannan. Improving federated learning personal- ization via model agnostic meta learning.arXiv preprint arXiv:1909.12488, 2019

    1909 arXiv

  2. [10]

    SCAFFOLD: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. InPro- ceedings of the 37th International Conference on Machine Learning, volume 119 ofProceeding...

    2020

  3. [11]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto, 2009

    2009

  4. [12]

    Predicting risk of adverse drug reactions in older adults

    Amanda Hanora Lavan and Paul Gallagher. Predicting risk of adverse drug reactions in older adults. Therapeutic advances in drug safety, 7(1):11–22, 2016

    2016

  5. [13]

    Feder- ated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Feder- ated optimization in heterogeneous networks. InProceedings of Machine Learning and Systems, volume 2, 2020

    2020

  6. [14]

    Ditto: Fair and robust federated learning through personalization

    Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. InInternational conference on machine learning, pages 6357–6368. PMLR, 2021

    2021

  7. [15]

    Fedrs: Federated learning with restricted softmax for label distribution non-iid data

    Xin-Chun Li and De-Chuan Zhan. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. InProceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2021

    2021

  8. [16]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  9. [17]

    Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints.IEEE transactions on neural networks and learning systems, 32(8):3710–3722, 2020. 10

    2020

  10. [18]

    Improving predictive inference under covariate shift by weighting the log-likelihood function

    Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000

    2000

  11. [19]

    An improved federated clustering algorithm with model-based clustering

    Harsh Vardhan, Avishek Ghosh, and Arya Mazumdar. An improved federated clustering algorithm with model-based clustering. Transactions on Machine Learning Research, 2024

    2024

  12. [20]

    Probability estimates for multi-class classification by pairwise coupling

    Ting-Fan Wu, Chih-Jen Lin, and Ruby Weng. Probability estimates for multi-class classification by pairwise coupling. Advances in Neural Information Processing Systems, 16, 2003

    2003

  13. [21]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  14. [22]

    Overcoming label shift in targeted federated learning

    Edvin Listo Zec, Adam Breitholtz, and Fredrik D Johansson. Overcoming label shift in targeted federated learning. arXiv preprint arXiv:2411.03799, 2024

    2024 arXiv

  15. [23]

    Navigating alignment for non-identical client class sets: A label name-anchored federated learning framework

    Jiayun Zhang, Xiyuan Zhang, Xinyang Zhang, Dezhi Hong, Rajesh K Gupta, and Jingbo Shang. Navigating alignment for non-identical client class sets: A label name-anchored federated learning framework. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Da...

    2023

  16. [25]

    Zero client probabilities for label 9:Because no client has seen label 9, their models will output near-zero (or zero) probabilities for this label: hk,y9 ≈ 0 ∀k

  17. [26]

    Problematic loss calculation when label 9 is involved:The loss calculation becomes problematic when either i = 9 or j = 9. Let’s analyze both cases: • Case 1: i = 9 loss_tensor = (hk,yj · p9 − hk,y9 · pj)2 ≈ (hk,yj · p9 − 0 · pj)2 = (hk,yj · p9)2 Thelossdependsontheglobalmodel...

  18. [27]

    The loss pushes p9 towards zero because that’s the only way to reduce the loss when paired with the near-zero client probabilities

    Global model degradation:The global model’s representation for label 9 is negatively impacted. The loss pushes p9 towards zero because that’s the only way to reduce the loss when paired with the near-zero client probabilities. This harms the global model’s ability to generaliz...

  19. [28]

    C Additional empirical results Here we show some additional results

    Unfair penalization: The global model receives gradients that are based on a comparison against the absent label, creating unstable behaviour. C Additional empirical results Here we show some additional results. C.1 Test accuracy over communication rounds Here we present the t...

  20. [2021]

    URL https://openreview.net/forum?id=ce6CFXBh30h

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.