ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes

Alberto Huertas Celdr\'an; Enrique Tom\'as Mart\'inez Beltr\'an; G\'er\^ome Bovet; Gregorio Mart\'inez P\'erez; Miguel Fern\'andez Llamas; Pedro Miguel S\'anchez S\'anchez

REVIEW 4 major objections 5 minor 1 cited by

ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ProFe claims a 40–50% communication cut in decentralized federated learning by exchanging distilled student models and class prototypes instead of full models.

desk verdict ProFe is a useful empirical composition of KD, prototypes, and quantization for DFL, but the prototype aggregation in Eq. (4) as written can't be right, and the FedAvg baseline architecture is unspecified; check the code before relying on the 40–50% number. read the letter →

arxiv 2412.11207 v1 pith:G2DEICEW submitted 2024-12-15 cs.LG cs.AIcs.DCcs.NI

Pedro Miguel Sánchez Sánchez , Enrique Tomás Martínez Beltrán , Miguel Fernández Llamas , Gérôme Bovet , Gregorio Martínez Pérez , Alberto Huertas Celdrán This is my paper · ORCID

classification cs.LGcs.AIcs.DCcs.NI

keywords DecentralizedFederatedLearningCommunicationOptimizationKnowledgeDistillationPrototypeQuantizationNon-IIDdataModelcompression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProFe is a decentralized federated learning algorithm that combines three techniques — knowledge distillation, prototype learning, and 16-bit quantization — to shrink the amount of data nodes exchange each round. The paper's central claim is that this combination cuts communication costs by roughly 40–50% compared with FedAvg and other literature baselines on MNIST, CIFAR10, and CIFAR100, while maintaining or improving model F1-score, especially under non-IID data splits. The price is an 18–20% increase in training time on the harder datasets, which the authors frame as a deliberate trade-off for bandwidth-constrained settings. If true, the result matters because communication overhead is the main bottleneck in decentralized federated learning, and ProFe would offer a way to keep accuracy without a central server.

What carries the argument

The central object is the ProFe training loop: each node keeps a large teacher model and a small student model, and the student is trained with cross-entropy, a prototype-matching mean-squared-error loss, a knowledge-distillation KL divergence, and an additional mean-squared-error loss between the student's and teacher's intermediate representations. The teacher is trained with cross-entropy and the prototype-matching loss. The artifacts exchanged between nodes are the student model and per-class prototypes — averages of intermediate representations taken before the final prediction layer — and both are quantized to 16 bits before transmission to halve the byte count. A schedule called Professor Importance Decay halves the distillation weight after each federated round and zeroes it below a threshold, so the student eventually stops imitating the teacher.

What would settle it

A direct test is to rerun the CIFAR10 non-IID 20% experiment with clients whose local label sets are disjoint, and compare nearest-prototype accuracy against predictions made by the student's own classifier head. If the nearest-prototype rule does not correctly classify classes a client never saw, the prototype mechanism's claimed benefit fails; a second check is to recompute global prototypes with a properly count-normalized average and see whether accuracy changes.

Watch

Extended reading notes

Core claim

ProFe's central discovery is that a decentralized federated system can exchange only small student models and class prototypes instead of full models, and still match or beat a full-model baseline. On MNIST, CIFAR10, and CIFAR100, with 20 nodes in a fully connected topology, average bytes sent and received drop by roughly 40–50% relative to FedAvg, while average node F1 stays within the same range or improves in non-IID settings; the cost is an 18–20% rise in training time on the harder datasets. The mechanism is a joint loss that trains a small student to imitate a large local teacher's softened logits and intermediate representations, while also pulling the student's representations toward global class prototypes computed from other nodes. Classification during testing switches to the nearest global prototype, which is how the system claims to handle classes a client never observed locally.

Load-bearing premise

The method's correctness rests on the assumption that the class summaries (prototypes) exchanged between nodes remain accurate when averaged by the paper's formula, and that classifying a point by the closest summary works even for classes a node never saw in its local data.

Editorial extensions

If this is right

In a fully connected decentralized topology, replacing full model exchange with a distilled student plus quantized prototypes cuts average bytes sent and received by roughly 40–50% relative to FedAvg on MNIST, CIFAR10, and CIFAR100.
Model quality, measured by average node F1-score, stays close to FedAvg and beats FedProto and FML in most non-IID settings, so the communication saving does not come at the cost of accuracy in the tested configurations.
Because only the student model and class prototypes are exchanged, the method supports heterogeneous local model sizes; each node can keep its large teacher locally for distillation.
The 16-bit quantization step halves the byte size of every transmitted model and prototype, which is the direct source of the 50% part of the saving.
The added 18–20% training time on CIFAR10 and CIFAR100 is the cost of running both teacher and student plus prototype losses, a trade-off that matters when bandwidth is scarcer than compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Implicit in the design but not tested: the 16-bit quantization alone accounts for the 50% part of the saving, so lower-precision quantization (8-bit) or quantization-aware training is a natural way to push the reduction further.
The paper reports communication costs only under IID; it assumes the ratio of savings transfers to non-IID settings because byte counts are distribution-independent. A direct measurement under the Dirichlet splits used for accuracy would test that assumption.
Because Eq. (4) divides the count-weighted prototype sum by the number of clients rather than by the total count, re-normalizing it could change the global prototypes; re-running the experiments with the corrected average would isolate whether the current formula helps or hurts the reported accuracy.
The teacher–student representation-matching term in Eq. (8) is an extra constraint; ablating it would show how much of ProFe's accuracy retention comes from distillation versus prototype regularization.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the combination: knowledge distillation, prototype learning, and quantization in a decentralized topology, with real Docker-based communications measured directly rather than simulated. That measurement setup is a real strength. The F1 comparisons across MNIST, CIFAR10, and CIFAR100 with five data distributions are broad, and the claim is honestly scope-limited: roughly 40–50% communication savings, about 20% training time overhead, accuracy roughly on par with FedAvg and better than FedProto and FML in non-IID settings. I believe the empirical work is mostly solid, and the related work is cited fairly.

The soft spots are not fatal, but they are real. First, Eq. (4) is wrong as written. The global prototype is defined as (1/|N^j|) * sum_{i in N^j} (|D_{i,j}|/N_j) * C_i^(j). Since sum_i |D_{i,j}|/N_j = 1, the extra 1/|N^j| factor scales each class prototype down by the number of clients that know that class. The nearest-prototype rule in Eq. (5) and the MSE loss in Eq. (6) are not invariant to per-class scaling, so in non-IID settings with partial label overlap, this biases predictions. Either the code implements a corrected weighted average and Eq. (4) is a typo, or the code follows the paper and the reported F1 curves are suspect. This needs to be checked before I would trust the central non-IID claim.

Second, the FedAvg baseline architecture is not specified. If FedAvg transmits teacher-size models while ProFe transmits student models, part of the 40–50% savings is just model-size reduction, not the algorithm. That has to be stated. Third, the abstract says ≈40–50% while the introduction says ≈50–60%; minor but embarrassing. Fourth, key hyperparameters (initial KD weight, beta_limit, alpha_s, beta_s, temperature) are missing, so reproduction is impossible without the code. The communication table only shows IID, but the authors give a reasonable justification that the proportional savings are distribution-independent; I would still want one non-IID row.

The citation pattern is clean and the code is linked. This is an engineering composition paper, not a conceptual leap, and it does not overclaim. I would send it to peer review with a request for major revision: fix Eq. (4), specify the FedAvg model, add the missing hyperparameters, and reconcile the savings numbers. If the code fixes the prototype formula, this is a useful contribution for DFL practitioners. If it does not, the paper's main non-IID results are in doubt.

For a reading group: maybe, mainly to discuss the prototype aggregation bug. I would not cite it in my own work until the code and equation are checked.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes ProFe, a decentralized federated learning algorithm that combines knowledge distillation from a local teacher to a smaller student, prototype aggregation, and 16-bit quantization to reduce communication cost. The evaluation uses MNIST, CIFAR10, and CIFAR100 with 20 nodes and five data distributions, measuring average node F1, network bytes sent/received, and wall-clock training time against FedAvg, FedProto, FedGPD, and FML. The headline claims are a 40-50% communication reduction with maintained or improved model performance and approximately 19% training-time overhead.

Significance. If the implementation matches the description, the paper offers a practical communication-efficiency recipe for decentralized federated learning and includes direct communication measurements over real Docker-based node-to-node traffic, which is a strength. The comparison across several literature baselines and the public code availability are also positive features. However, two load-bearing issues must be resolved before the central claim can be accepted: the global prototype aggregation in Eq. (4) appears improperly normalized, and the FedAvg baseline architecture is unspecified, making the reported communication savings difficult to attribute to the proposed algorithm.

major comments (4)

[Section III-B, Eq. (4)] The global prototype formula is not a correctly weighted average. The weights |D_{i,j}|/N_j sum to 1 by definition, so the additional factor 1/|N^j| introduces a class-dependent scale factor equal to 1 divided by the number of clients that know class j. Because Eq. (5) compares Euclidean distances to these prototypes and Eq. (6) anchors the representation loss to them, the scale factor biases predictions against classes known by many clients in the partial-label non-IID settings of Section IV. Please correct Eq. (4) to a properly normalized weighted average, or show from the released code that the implemented aggregation differs from the formula as written. The notation N^j versus N_j is also inconsistent, so the formula cannot be disambiguated by notation alone.
[Section IV-B, Table II] The FedAvg baseline architecture and communication details are not specified. ProFe transmits a smaller student model (half-channel CNN for MNIST, ResNet8 for CIFAR10, ResNet18 for CIFAR100) under 16-bit quantization, while the FedAvg model size, architecture, and bit width are not given. The reported 40-50% communication reduction could therefore reflect the smaller model and quantization choices rather than the proposed distillation and prototype mechanism. The paper should report the FedAvg architecture, parameter count, bytes per update, and bit width so that the comparison isolates the algorithmic contribution.
[Section IV-B, Table III] The approximately 19% training-time overhead for CIFAR10 and CIFAR100 is reported as a single aggregate value, with no breakdown among teacher training, student training, prototype computation, and communication. Since ProFe trains two networks per node, the near-zero overhead on MNIST and the exact 19% figures across datasets are hard to interpret without such a breakdown. Adding a time decomposition, or at least separate measurements for the main components, would support the communication-versus-training-time trade-off claimed in the paper.
[Section IV] The hyperparameters controlling the losses in Eqs. (2), (8), and (9) are not reported: the initial distillation weight beta, the decay limit beta_limit, the prototype and distillation weights beta_s and alpha_s, the temperature T, and the quantization step Delta of Section III-D. Because the performance and communication trade-off depends directly on these values, the experimental results are not reproducible without them. Please add a hyperparameter table and, ideally, a brief sensitivity study.

minor comments (5)

[Abstract and Section I] The abstract states a communication reduction of 'up to ~40-50%' while the introduction states '~50–60%'; the two numbers should be reconciled.
[Section IV] The text says validation used the Nebula framework and cites reference [12], but [12] is the Fedstellar paper and the NEBULA repository is reference [13]; the citation should be corrected.
[Figure 2] The F1 curves are shown without error bars or confidence bands, although Table II reports standard deviations; adding uncertainty information would strengthen the claim that performance is maintained.
[Section IV-A] Phrases such as '+95% F1-Score' are likely proportions (0.95), but the percentage notation is ambiguous; please use standard decimal notation for F1 values.
[Section III-D] The quantization function is described, but the manuscript does not state how Delta is chosen per tensor or whether gradients and prototypes are quantized in the same way; please clarify the quantization scope.

Circularity Check

0 steps flagged · score 0.0 of 10

ProFe is a self-contained empirical evaluation against external baselines; the author self-citations are background/platform references and are not load-bearing.

full rationale

ProFe's central claims are empirical measurements: communication bytes and elapsed time are reported in Tables II and III, and model quality is reported as F1 scores in Fig. 2 against FedAvg, FedGPD, FML, and FedProto. These claims are not derived by construction from the method's definitions. The communication reduction follows from the design choice to transmit a smaller student model and quantized values, but the paper measures the resulting bytes rather than defining the improvement into existence. The prototype equations (3)-(7) implement a prototype-based loss and nearest-prototype classifier; they are algorithmic components used in the evaluation, not a fitted parameter renamed as a prediction. The paper does not fit a parameter to a subset of the evaluation data and then predict a closely related quantity. The self-citations to prior work by the same authors, specifically [2] as background on decentralized federated learning and [12]/[13] as the framework and code used for experiments, are not load-bearing for the claim that ProFe performs well; the comparisons are against external methods and standard benchmarks. No uniqueness theorem from the authors is invoked, and no ansatz is smuggled in via citation. The possible normalization issue in Eq. (4) is a correctness or implementation concern, not a circularity, because the experimental results are not logically reduced to that formula. Therefore no circular step is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method's accuracy rests on the prototype aggregation being a valid average and on hyperparameters chosen without reported values. The communication measurement is straightforward, but the fairness of the FedAvg baseline is not demonstrated.

free parameters (5)

initial KD weight beta = not reported
Controls the balance between cross-entropy and distillation loss in Eq. (2); value is not given.
KD decay threshold beta_limit = not reported
Sets when distillation loss is turned off; value is not given.
distillation weight alpha_s = not reported
Scales the KD plus MSE losses in the student loss, Eq. (8); value is not given.
prototype weight beta_s = not reported
Scales the prototype MSE loss for the student in Eq. (8); teacher uses beta_t = 1.
KD temperature T = not reported
Smooths teacher and student probability distributions in KD; value is not given.

assumptions (5)

standard math KL divergence, cross-entropy, and MSE are appropriate loss functions for the stated objectives.
Used in Eqs. (1), (2), (6), (8), and (9).
domain assumption Intermediate feature representations from different clients are comparable in a shared Euclidean space.
The prototype loss and nearest-prototype classification in Eqs. (5) and (6) assume this.
domain assumption The simulated non-IID data partitions (60%, 40%, 20%, Dirichlet) represent realistic decentralized FL heterogeneity.
Used for all experiments in Section IV; conclusions are limited to these splits.
domain assumption A fully connected 20-node topology is representative enough for the communication claims.
All experiments use this topology; sparse topologies are not tested.
ad hoc to paper The global prototype aggregation in Eq. (4) is correctly normalized.
The formula as written divides a count-weighted sum by the number of clients, which is not a proper average under the paper's own definitions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes." pith.science (2026). https://pith.science/paper/G2DEICEW

@misc{pith2026241211207,
  author       = {Pith},
  title        = {Pith review of: ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2DEICEW}},
  note         = {Machine review of arXiv:2412.11207}
}

read the original abstract

Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off.

Figures

Figures reproduced from arXiv: 2412.11207 by the authors.

**Figure 2.** F1-Score Comparison Between the Proposed Algorithm (ProFe) and the Literature [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

S-VOTE: Similarity-based Voting for Client Selection in Decentralized Federated Learning
cs.LG 2025-01 conditional novelty 4.0 of 10

S-VOTE selects clients by cosine similarity of model weights and lets low-vote clients sometimes skip training, reducing communication and energy while improving non-IID accuracy in some settings.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelli- gence and statistics . PMLR, 2017, pp. 1273–1282

work page 2017
[2]

Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,

E. T. M. Beltr ´an et al., “Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,” IEEE COMST, 2023

work page 2023
[3]

Decentralized feder- ated learning: Balancing communication and computing costs,

W. Liu, L. Chen, and W. Zhang, “Decentralized feder- ated learning: Balancing communication and computing costs,” IEEE Transactions on Signal and Information Processing over Networks , vol. 8, pp. 131–143, 2022

2022
[4]

Communication-efficient federated learning via knowl- edge distillation,

C. Wu, F. Wu, L. Lyu, Y . Huang, and X. Xie, “Communication-efficient federated learning via knowl- edge distillation,” Nature Communications, vol. 13, 2022

work page 2022
[5]

Knowledge distillation: A survey,

J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

2021
[6]

Rethinking federated learning with domain shift: A prototype view,

W. Huang, M. Ye, Z. Shi, H. Li, and B. Du, “Rethinking federated learning with domain shift: A prototype view,” in IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR) , 2023, pp. 16 312–16 322

work page 2023
[8]

Federated mutual learning: a collaborative machine learning method for heterogeneous data, mod- els, and objectives,

T. Shen et al., “Federated mutual learning: a collaborative machine learning method for heterogeneous data, mod- els, and objectives,” Frontiers of Information Technology & Electronic Engineering , vol. 24, no. 10, pp. 1390– 1402, 2023

work page 2023
[9]

Fedproto: Federated prototype learning across heterogeneous clients,

Y . Tan et al. , “Fedproto: Federated prototype learning across heterogeneous clients,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8432–8440

work page 2022

Show all 12 references

[10]

Global prototype distillation for hetero- geneous federated learning,

S. Wu et al. , “Global prototype distillation for hetero- geneous federated learning,” Scientific Reports , vol. 14, no. 1, p. 12057, 2024

2024
[11]

Ac- celerating federated learning for iot in big data analytics with pruning, quantization and selective updating,

W. Xu, W. Fang, Y . Ding, M. Zou, and N. Xiong, “Ac- celerating federated learning for iot in big data analytics with pruning, quantization and selective updating,” IEEE Access, vol. 9, pp. 38 457–38 466, 2021

2021
[12]

Fedstellar: A platform for decentralized federated learning,

E. T. M. Beltr ´an et al. , “Fedstellar: A platform for decentralized federated learning,” Expert Systems with Applications, vol. 242, p. 122861, 2024

2024
[13]

NEBULA: A Platform for Decentralized Federated Learning - Communication Optimization,

M. Fern ´andez Llamas et al. , “NEBULA: A Platform for Decentralized Federated Learning - Communication Optimization,” https://github.com/CyberDataLab/nebula/ tree/feature/communication optimization, 2024, [Online; accessed 30-October-2024]

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelli- gence and statistics . PMLR, 2017, pp. 1273–1282

work page 2017

[2] [2]

Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,

E. T. M. Beltr ´an et al., “Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,” IEEE COMST, 2023

work page 2023

[3] [3]

Decentralized feder- ated learning: Balancing communication and computing costs,

W. Liu, L. Chen, and W. Zhang, “Decentralized feder- ated learning: Balancing communication and computing costs,” IEEE Transactions on Signal and Information Processing over Networks , vol. 8, pp. 131–143, 2022

2022

[4] [4]

Communication-efficient federated learning via knowl- edge distillation,

C. Wu, F. Wu, L. Lyu, Y . Huang, and X. Xie, “Communication-efficient federated learning via knowl- edge distillation,” Nature Communications, vol. 13, 2022

work page 2022

[5] [5]

Knowledge distillation: A survey,

J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

2021

[6] [6]

Rethinking federated learning with domain shift: A prototype view,

W. Huang, M. Ye, Z. Shi, H. Li, and B. Du, “Rethinking federated learning with domain shift: A prototype view,” in IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR) , 2023, pp. 16 312–16 322

work page 2023

[7] [8]

Federated mutual learning: a collaborative machine learning method for heterogeneous data, mod- els, and objectives,

T. Shen et al., “Federated mutual learning: a collaborative machine learning method for heterogeneous data, mod- els, and objectives,” Frontiers of Information Technology & Electronic Engineering , vol. 24, no. 10, pp. 1390– 1402, 2023

work page 2023

[8] [9]

Fedproto: Federated prototype learning across heterogeneous clients,

Y . Tan et al. , “Fedproto: Federated prototype learning across heterogeneous clients,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8432–8440

work page 2022