Pith. sign in

REVIEW 5 major objections 3 minor 58 references

FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation

T0 review · 5 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Heterogeneous black-box RL agents can federate by sharing action probability distributions instead of model parameters, and this distillation improves both system- and per-agent sample efficiency.

desk verdict A plausible distillation-based FedRL method, but the theoretical claims are unsound and the experiments too thin to compensate. read the letter →

arxiv 2502.00870 v1 pith:NXPPQGOH submitted 2025-02-02 cs.LG cs.AIcs.MA

Wenzheng Jiang , Ji Wang , Xiongtao Zhang , Weidong Bao , Cheston Tan , Flint Xiaofeng Fan This is my paper · ORCID
classification cs.LGcs.AIcs.MA MSC 68T0590C4068W15
keywords federatedreinforcementlearningpolicydistillationagentheterogeneityblack-boxmodelsknowledgesampleefficiencygradientconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether reinforcement learning agents with different network architectures, activation functions, and learning rates, visible to one another only as black boxes, can still learn collaboratively in a federated setting. It argues they can, provided the shared currency is the action probability distribution rather than model parameters. The proposed method, FedHPD, periodically has each agent compute its action distributions on a fixed public state set, averages these distributions at the server, and then pulls each local policy toward that global average with a KL-divergence penalty. The paper claims this improves both the system-level average reward and each individual agent's sample efficiency, and proves that the scheme converges with a sample-complexity ordering between vanilla REINFORCE and REINFORCE with knowledge distillation.

What carries the argument

The central object is the action probability distribution as a black-box-shared knowledge medium, aggregated into a global consensus over a public state set. The load-bearing mechanism is the KL-divergence digestion step, in which each agent minimizes the KL divergence between its own action distribution and the global average on the public state set, added as a penalty to the REINFORCE objective, together with the distillation interval that controls how often this collaborative step interrupts local training. The proof machinery is the L-smoothness of the combined objective and a variance-reduction argument: the KL penalty lowers the variance of the policy-gradient estimator when the gradient of the KL term and the gradient of the return are sufficiently aligned, which is what yields the claimed sample-complexity ordering.

What would settle it

Run FedHPD with a public state set drawn from a distribution that barely overlaps the states the agents actually visit (for example, random noise inputs on Cartpole) and compare system average reward against independent training; if performance still improves, the public-state requirement is not load-bearing, while if it does not, the method's dependence on a usable state set is exposed. Alternatively, measure the quantities in the variance-reduction condition during early training; if the variance of the KL gradient is at least twice the covariance between the return gradient and the KL gradient for most rounds, the claimed variance reduction fails and the sample-complexity ordering would not follow.

Watch

Extended reading notes

Core claim

FedHPD's central claim is that periodic policy distillation over a public state set resolves agent heterogeneity in federated policy-gradient reinforcement learning. Each heterogeneous agent trains locally with REINFORCE; every d rounds, each agent uploads its action probability vectors on a shared state set, the server averages them into a global consensus distribution, and each agent takes a gradient step against the KL divergence between its own distribution and that consensus. The paper shows this objective is L-smooth by combining the smoothness of the policy-gradient return with the smoothness of the KL term, proves convergence of REINFORCE with the distillation penalty under Robbins-Monro learning rates, and argues that the distillation term reduces gradient variance when reward-seeking and consensus-seeking gradients align, yielding the sample-complexity ordering N_REINFORCEwithKD ≤ N_FedHPD ≤ N_REINFORCE. Experiments on Cartpole, LunarLander, and InvertedPendulum with ten heterogeneous agents report higher system average rewards and faster per-agent convergence than independent training, with smaller distillation intervals giving larger gains.

Load-bearing premise

The method collapses if the server cannot obtain a useful public state set before training, or if the uniform average of heterogeneous action distributions is not a beneficial distillation target; the paper relies on both without proving that the average helps.

Editorial extensions

If this is right

  • Heterogeneous black-box agents can federate without sharing parameters or gradients; only action probability vectors on a public state set need to be exchanged.
  • Smaller distillation intervals give larger system-level gains but at increased communication cost, so the interval trades individual improvement against overhead.
  • The variance-reduction argument predicts that the sample complexity of FedHPD sits between vanilla REINFORCE and REINFORCE with per-round distillation, which the experiments support by showing performance degrading toward NoFed as the interval grows.
  • Agents that train poorly in isolation benefit most from distillation, since the global consensus pulls suboptimal policies toward better ones.
  • The method extends to continuous action spaces by using Gaussian output distributions for the KL term, not just softmax logits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the variance-reduction condition is the real driver, FedHPD's benefit should be reproducible in a single-agent setting: a locally trained REINFORCE agent penalized by KL toward a slowly moving target distribution should show the same sample-efficiency gain, which would separate the effect of distillation from the effect of federation per se.
  • The public state set need not be drawn from the exact task distribution, since the states are only used as inputs for comparing distributions; states from a related task or a coarse simulator could substitute, which the paper's own experiments on different public state sets begin to support.
  • Weighted rather than uniform averaging of local distributions, with weights based on estimated policy quality, is an untested variant that could address the paper's own concession that averaging may pull policies toward suboptimal solutions in extreme heterogeneity.
  • The convergence result is stationarity rather than global optimality, and the paper explicitly leaves specific sample-complexity bounds for heterogeneous settings open; tightening the ordering into an explicit rate would be the natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper proposes FedHPD, a federated reinforcement learning method for heterogeneous black-box agents. Agents train locally with REINFORCE, periodically upload action probability distributions over a public state set, and are distilled toward the global average of these distributions (the 'global consensus'). The authors claim that this mechanism improves system-level and per-agent sample efficiency, provide a convergence theorem for the regularized objective and a corollary giving a sample-complexity ordering N_REINFORCEwithKD ≤ N_FedHPD ≤ N_REINFORCE, and report experiments on Cartpole, LunarLander, and InvertedPendulum.

Significance. FedHPD addresses a relevant and under-studied problem: collaborative RL with heterogeneous architectures and black-box policies. The proposed knowledge-transfer mechanism is simple and communication-efficient, and the empirical results on three Gym tasks suggest that the method can improve over independent training. The paper also makes its code available. However, the theoretical analysis is not reliable: the smoothness constant in Theorem 1 has a sign error, the Gaussian part of Lemma 1 applies a closed-form formula to a mixture distribution where it does not apply, and the fast-convergence corollary depends on an unproved alignment condition that the paper itself later disavows. These issues affect the central claims, so the contribution is presently an empirical study with unresolved theoretical backing.

major comments (5)
  1. [Appendix A.2, Eq. (37)] The proof of Theorem 1 defines the smoothness constant of J' = J - λ D_KL as L = L_J - λ L_KL. For a difference of smooth functions, the standard composition rule gives L = L_J + λ L_KL. With the paper's sign, the descent inequality (47) can fail even when both J and D_KL are individually smooth, so the claimed convergence to a stationary point is not established.
  2. [Appendix A.1, Eqs. (31)-(32)] The Gaussian half of Lemma 1 uses the closed-form KL divergence between two Gaussians to compute D_KL(π_θk || π_global), but π_global is a mixture of K Gaussian components, not a Gaussian. Therefore Eq. (32) is not the KL divergence of the actual distributions, and the L-smoothness of the KL term for continuous action spaces is unproven. This directly affects the validity of Theorem 1 for the continuous-action experiments.
  3. [Corollary 1, Eq. (17)] The fast-convergence claim rests on Eq. (17), which requires the gradient of the reward to align with the gradient of the KL divergence term. This condition is assumed, not derived, and no argument is given for why it should hold for heterogeneous agents. Section 6 concedes that averaging may lead to suboptimal policies. Moreover, Eq. (18) is asserted without proof, and Section 7 states that the paper 'does not provide specific sample complexity results,' directly contradicting the ordering in Eq. (10).
  4. [Section 3.3, Table 1] The paper claims independence from a server-side MDP (Table 1, ISM column), but Section 3.3 requires a simulated server-side environment with a virtual agent to generate the public state set S_p. When such states are not available, the method cannot be executed. This is a precondition for the algorithm's operation rather than a minor implementation detail, and it should be stated as a limitation rather than a claimed advantage.
  5. [Section 5, Table 2] The empirical evaluation reports a single number per condition in Table 2, despite Section 5.1 stating that each experiment is run with five random seeds. Without variance estimates or confidence intervals, and with NoFed as the only baseline in Table 2, the claimed improvements are not statistically supported. The comparison with DPA-FedRL in Figure 4 covers only two tasks and also lacks error bars.
minor comments (3)
  1. [Section 2.2] The statement that 'REINFORCE ... does not require trajectory sampling' is inaccurate; REINFORCE estimates the policy gradient from sampled trajectories and should be corrected.
  2. [Section 3.1, Eqs. (4)-(5)] The quantities ψ'_k and ψ_k are not formally defined; the problem formulation would be clearer if these were specified as expected returns for a fixed training budget.
  3. [Throughout] Several typos and notation inconsistencies remain, such as 'foucs' in Section 3.2, 'widly' in Section 4, and the use of P^{i+1}_k versus π_θ(·|S_p); please standardize the notation and proofread.

Circularity Check

2 steps flagged · score 6.0 of 10

The fast-convergence result is assumed rather than derived: the global-consensus teacher is an average of the students, and the key variance inequality (Eq. 18) is asserted without proof.

  1. self definitional [Section 3.3, Algorithm 1 lines 8-13; Eq. (6)]
    "Server aggregates the uploaded probability distributions to obtain global consensus P^{i+1} ... Local agents calculate the KL divergence between their predicted distributions P^{i+1}_k and global distributions P^{i+1} to update their policy parameters ... global action probability distributions π_global(a|S_p) = 1/K Σ_{k=1}^K π_{θ_k}(a|S_p)."

    The 'teacher' is formed by averaging the students' own current action distributions, so the distillation loss L = D_KL(P_k || P_global) contains no information beyond the parameters being optimized. Consequently the claimed improvement—'By periodically distilling the local policies, it effectively enhance system performance and individual sample efficiency'—is a statement about a self-averaging regularizer, not about importing external knowledge. Section 6 concedes 'If we use average aggregation, it might lead the policy towards suboptimal solution,' confirming that the self-referential target does not by itself guarantee the claimed benefit.

  2. other [Section 4, Corollary 1 proof, Eq. (18)]
    "Combined with Eq. (16), for FedHPD, we have: Var[∇θJ′(θ)]≤ Var[∇θJ′′(θ)]≤ Var[∇θJ(θ)], (18) where the equality is obtained for d = 1 and d = ∞."

    Eq. (16) is conditional on the unproved alignment condition Eq. (17) and compares only J′ with J. The proof then asserts, without derivation, Var[J′] ≤ Var[J′′] ≤ Var[J] for intermediate d—exactly the fast-convergence claim in variance form—and the Chebyshev step converts this asserted inequality into the sample-complexity ordering N_REINFORCEwithKD ≤ N_FedHPD ≤ N_REINFORCE of Eq. (10). Thus the theorem's conclusion is restated as an input rather than derived from the assumptions. Section 7 later states the paper 'does not provide specific sample complexity results,' confirming that Eq. (10) is not an established consequence.

full rationale

The paper's central claim of provable sample-efficiency improvement is not an independent derivation. The global consensus is defined as the average of the local policies themselves (Eq. 6), so the distillation target is self-referential rather than an external teacher; any benefit depends on a regularizer over the same parameters being optimized. The fast-convergence proof then assumes the load-bearing variance inequality Eq. (18) without deriving it, after already relying on the unproven alignment condition Eq. (17). This is a conclusion-assumed argument rather than a reduction from standard assumptions. Additional non-circular correctness problems exist (the wrong smoothness sign L = L_J − λL_KL in Theorem 1 and the Gaussian-to-mixture KL closed form in Eq. 32), and Section 7 explicitly disclaims specific sample complexity results, but those are proof-quality issues rather than circularity. There is no load-bearing self-citation chain: Proposition 1 is attributed to Xu et al. and the smoothness constant to Lemma 4.4 of Yuan et al., both external. Because the central sample-complexity prediction effectively reduces to an asserted variance bound and a self-defined consensus target, the circularity score is 6.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The framework depends on hand-selected hyperparameters, on smoothness assumptions for policy distributions, on the untested usefulness of a self-generated average target, and on the availability of a server-side simulator or public states. The theoretical results further assume a gradient-alignment condition that is not proved.

free parameters (3)
  • λ (regularization coefficient) = 1
    Chosen by hand in Appendix A.3 to minimize gradient variance; the value affects Theorem 1's condition L = L_J - λL_KL > 0 and Corollary 1.
  • Distillation interval d = 5, 10, 20 in main experiments; also 2, 40, 80 in Section 5.5
    Hand-selected; the paper's performance and sample-complexity claims depend on d, and Corollary 1 relates d to sample size.
  • Public state set size |S_p| = 5000 (Cartpole, InvertedPendulum), 10000 (LunarLander)
    Chosen in Appendix B to balance communication cost and training utility; it determines the amount of distillation signal.
assumptions (6)
  • domain assumption Assumption 1: policy parameterization regularity (bounded first and second derivatives of log policy).
    Invoked in Section 4 to guarantee Lipschitz smoothness of the policy gradient objective; standard in policy gradient analysis.
  • domain assumption Assumption 2: softmax distribution is L-smooth with respect to parameters.
    Needed for Lemma 1's smoothness of the KL divergence in discrete action spaces.
  • domain assumption Assumption 3: Gaussian mean and variance are L-smooth functions of parameters.
    Needed for Lemma 1's smoothness of the KL divergence in continuous action spaces; the proof then incorrectly treats a Gaussian mixture as a single Gaussian.
  • domain assumption All agents interact with separate copies of the same MDP M.
    Section 3.1 states 'agent k interacts in a separate copy of the MDP M', so heterogeneity is limited to policy networks and training configurations, not environments.
  • ad hoc to paper The average of heterogeneous action distributions is a useful training target.
    The paper asserts global consensus helps guide suboptimal policies (Section 5.3) without proving it, and Section 6 acknowledges averaging may lead to suboptimal solutions.
  • ad hoc to paper Variance-reduction condition Eq. (17) holds for FedHPD.
    Corollary 1 requires Var[∇D_KL] < 2Cov(∇J,∇D_KL), i.e., alignment of reward and consensus gradients; the proof asserts 'it typically tends to lower values' without establishing it.
invented entities (1)
  • Virtual agent trained in a server-side simulated environment
    purpose: Generate the offline public state set S_p used as shared input for knowledge distillation
    The paper postulates this component without external validation; its states must be representative enough to support distillation, and its existence conflicts with the claim of independence from a server-side MDP.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation." pith.science (2026). https://pith.science/paper/NXPPQGOH

@misc{pith2026250200870,
  author       = {Pith},
  title        = {Pith review of: FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NXPPQGOH}},
  note         = {Machine review of arXiv:2502.00870}
}
read the original abstract

Federated Reinforcement Learning (FedRL) improves sample efficiency while preserving privacy; however, most existing studies assume homogeneous agents, limiting its applicability in real-world scenarios. This paper investigates FedRL in black-box settings with heterogeneous agents, where each agent employs distinct policy networks and training configurations without disclosing their internal details. Knowledge Distillation (KD) is a promising method for facilitating knowledge sharing among heterogeneous models, but it faces challenges related to the scarcity of public datasets and limitations in knowledge representation when applied to FedRL. To address these challenges, we propose Federated Heterogeneous Policy Distillation (FedHPD), which solves the problem of heterogeneous FedRL by utilizing action probability distributions as a medium for knowledge sharing. We provide a theoretical analysis of FedHPD's convergence under standard assumptions. Extensive experiments corroborate that FedHPD shows significant improvements across various reinforcement learning benchmark tasks, further validating our theoretical findings. Moreover, additional experiments demonstrate that FedHPD operates effectively without the need for an elaborate selection of public datasets.

Figures

Figures reproduced from arXiv: 2502.00870 by the authors.

Figure 1
Figure 1. Illustration of FedHPD. 1. Generate public state set [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparisons of system performance under different distillation intervals ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparisons of selected individual performance under different distillation intervals ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Comparisons of system performance between DPA [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparisons of system performance under differ [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Complete comparisons of individual agents with different distillation intervals ( [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Comparisons of individual agent between DPA-FedRL and FedHPD ( [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Comparisons of individual agent with different distillation intervals in Cartpole ( [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Distributions of different public datasets (Employing PCA for dimensionality reduction). [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Comparisons of system performance with different datasets in Cartpole ( [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Comparisons of individual agent with different datasets in Cartpole ( [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 43 canonical work pages

  1. [1]

    Zeyuan Allen-Zhu and Elad Hazan. 2016. Variance Reduction for Faster Non- Convex Optimization. In ICML, Vol. 48. 699–707

    work page 2016

  2. [2]

    Jonathan Baxter and Peter L Bartlett. 2001. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research 15 (2001), 319–350

    work page 2001

  3. [3]

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schul- man, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016)

    arXiv 2016

  4. [4]

    Runqi Chai, Hanlin Niu, Joaquin Carrasco, Farshad Arvin, Hujun Yin, and Barry Lennox. 2022. Design and experimental validation of deep reinforce- ment learning-based fast trajectory planning and control for mobile robot in unknown environment. IEEE TNNLS 35, 4 (2022), 5778–5792

    work page 2022

  5. [5]

    Giuseppe Cuccu, Luca Rolshoven, Fabien Vorpe, Philippe Cudré-Mauroux, and Tobias Glasmachers. 2022. DiBB: distributing black-box optimization. In Genetic and Evolutionary Computation Conference . 341–349

    work page 2022

  6. [6]

    Zhongxiang Dai, Flint Xiaofeng Fan, Cheston Tan, Trong Nghia Hoang, Bryan Kian Hsiang Low, and Patrick Jaillet. 2024. Chapter 14 - Federated sequential decision making: Bayesian optimization, reinforcement learning, and beyond. In Federated Learning. Academic Press, 257–279. https://doi.org/10.1016/B978-0- 44-319037-7.00023-5

    doi:10.1016/b978-0- 2024

  7. [7]

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In CoRL. 1–16

    work page 2017

  8. [8]

    Flint Xiaofeng Fan, Yining Ma, Zhongxiang Dai, Wei Jing, Cheston Tan, and Bryan Kian Hsiang Low. 2021. Fault-tolerant federated reinforcement learning with theoretical guarantee. In Advances in Neural Information Processing Systems , Vol. 34. 1007–1021

    work page 2021

Show all 58 references
  1. [9]

    Flint Xiaofeng Fan, Yining Ma, Zhongxiang Dai, Cheston Tan, and Bryan Kian Hsiang Low. 2023. FedHQL: Federated Heterogeneous Q-Learning. In Inter- national Conference on Autonomous Agents and Multiagent Systems . 2810–2812

    2023

  2. [10]

    Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong, Roger Wattenhofer, and Wei- Tsang Ooi. 2024. FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF. arXiv preprint arXiv:2412.15538 (2024)

    2024 arXiv

  3. [11]

    Ilyas Fatkhullin, Anas Barakat, Anastasia Kireeva, and Niao He. 2023. Sto- chastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non- degenerate Policies. In ICML, Vol. 202. 9827–9869

    2023

  4. [12]

    Bin Feng, Zhuping Liu, Gang Huang, and Chuangxin Guo. 2023. Robust federated deep reinforcement learning for optimal control in multiple virtual power plants with electric vehicles. Applied Energy 349 (2023), 121615

    2023

  5. [13]

    Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. 2023. Dense reinforcement learning for safety validation of autonomous vehicles. Nature 615, 7953 (2023), 620–627

    2023

  6. [14]

    Matilde Gargiani, Andrea Zanelli, Andrea Martinelli, Tyler Summers, and John Lygeros. 2022. PAGE-PG: A simple and loopless variance-reduced policy gradient method with probabilistic gradient estimation. In ICML, Vol. 162. 7223–7240

    2022

  7. [15]

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139–144

    2020

  8. [16]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

    2015 arXiv

  9. [17]

    Chenying Jin, Xiang Feng, and Huiqun Yu. 2024. Embracing Multiheterogeneity and Privacy Security Simultaneously: A Dynamic Privacy-Aware Federated Reinforcement Learning Approach. IEEE TNNLS (2024), 1–15

    2024

  10. [18]

    Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2022. Fed- erated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics . 18–37

    2022

  11. [19]

    Philip Jordan, Florian Grötschla, Flint Xiaofeng Fan, and Roger Wattenhofer. 2024. Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergence. In International Conference on Autonomous Agents and Multiagent Systems. 964–972

    2024

  12. [20]

    Hao Ju, Rongshun Juan, Randy Gomez, Keisuke Nakamura, and Guangliang Li

  13. [21]

    Sajad Khodadadian, Pranay Sharma, Gauri Joshi, and Siva Theja Maguluri. 2022. Federated Reinforcement Learning: Linear Speedup Under Markovian Sampling. In International Conference on Machine Learning , Vol. 162. 10997–11057

    2022

  14. [22]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  15. [23]

    Daliang Li and Junpu Wang. 2019. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019)

    2019 arXiv

  16. [24]

    Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. 2020. Ensemble distillation for robust model fusion in federated learning. In NeurIPS 2020

    2020

  17. [25]

    Qiang Liu and Dilin Wang. 2016. Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm. In NeurIPS, Vol. 29

    2016

  18. [26]

    Weiming Mai, Jiangchao Yao, Gong Chen, Ya Zhang, Yiu-Ming Cheung, and Bo Han. 2023. Server-client collaborative distillation for federated reinforcement learning. ACM Transactions on Knowledge Discovery from Data 18, 1 (2023), 1–22

    2023

  19. [27]

    Hei Yi Mak, Flint Xiaofeng Fan, Luca A Lanzendörfer, Cheston Tan, Wei Tsang Ooi, and Roger Wattenhofer. 2024. CAESAR: Enhancing Federated RL in Hetero- geneous MDPs through Convergence-Aware Sampling with Screening. arXiv preprint arXiv:2403.20156 (2024)

    2024 arXiv

  20. [28]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics. 1273–1282

    2017

  21. [29]

    Volodymyr Mnih. 2016. Asynchronous Methods for Deep Reinforcement Learn- ing. arXiv preprint arXiv:1602.01783 (2016)

    2016 arXiv

  22. [30]

    Xinlei Pan, Weiyao Wang, Xiaoshuai Zhang, Bo Li, Jinfeng Yi, and Dawn Song

  23. [31]

    Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Mar- cello Restelli. 2018. Stochastic Variance-Reduced Policy Gradient. InInternational Conference on Machine Learning , Vol. 80. 4026–4035

    2018

  24. [32]

    Jing Qiao, Zuyuan Zhang, Sheng Yue, Yuan Yuan, Zhipeng Cai, Xiao Zhang, Ju Ren, and Dongxiao Yu. 2024. BR-DeFedRL: Byzantine-Robust Decentralized Federated Reinforcement Learning with Fast Convergence and Communication Efficiency. In IEEE Conference on Computer Communications ...

    2024

  25. [33]

    Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola

    Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola

  26. [34]

    Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. 2015. Policy distillation. arXiv preprint arXiv:1511.06295 (2015)

    2015 arXiv

  27. [35]

    Sefutsu Ryu and Shinya Takamaeda-Yamazaki. 2022. Model-based federated reinforcement distillation. In IEEE Global Communications Conference . 1109– 1114

    2022

  28. [36]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

  29. [37]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  30. [38]

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Mar- tin Riedmiller. 2014. Deterministic policy gradient algorithms. In International Conference on Machine Learning . 387–395

    2014

  31. [39]

    Lichao Sun and Lingjuan Lyu. 2021. Federated Model Distillation with Noise-Free Differential Privacy. In IJCAI. 1563–1570

    2021

  32. [40]

    Richard S Sutton. 2018. Reinforcement learning: An introduction. A Bradford Book (2018)

    2018

  33. [41]

    Ahmed Touati, Amy Zhang, Joelle Pineau, and Pascal Vincent. 2020. Stable Policy Optimization via Off-Policy Divergence Regularization. In Conference on Uncertainty in Artificial Intelligence, Vol. 124. 1328–1337

    2020

  34. [42]

    Qing Wang, Yingru Li, Jiechao Xiong, and Tong Zhang. 2019. Divergence- Augmented Policy Optimization. In Advances in Neural Information Processing Systems, Vol. 32

    2019

  35. [43]

    Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (1992), 229–256

    1992

  36. [44]

    Jiin Woo, Gauri Joshi, and Yuejie Chi. 2023. The blessing of heterogeneity in federated q-learning: Linear speedup and beyond. In International Conference on Machine Learning. 37157–37216

    2023

  37. [45]

    Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2022. Supervised advantage actor-critic for recommender systems. In International Conference on Web Search and Data Mining . 1186–1196

    2022

  38. [46]

    Pan Xu, Felicia Gao, and Quanquan Gu. 2020. An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient. In Conference on Uncertainty in Artificial Intelligence, Vol. 115. 541–551

    2020

  39. [47]

    Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. 2023. Hetero- geneous federated learning: State-of-the-art and research challenges. Comput. Surveys 56, 3 (2023), 1–44

    2023

  40. [48]

    Yang Yu. 2018. Towards sample efficient reinforcement learning. In International Joint Conference on Artificial Intelligence . 5739–5743

    2018

  41. [49]

    Gower, and Alessandro Lazaric

    Rui Yuan, Robert M. Gower, and Alessandro Lazaric. 2022. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics, Vol. 151. 3332–3380

    2022

  42. [50]

    Chenyu Zhang, Han Wang, Aritra Mitra, and James Anderson. 2024. Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning. In International Conference on Learning Representations

    2024

  43. [51]

    Zhong Zheng, Fengyu Gao, Lingzhou Xue, and Jing Yang. 2024. Federated Q- Learning: Linear Regret Speedup with Low Communication Cost. InInternational Conference on Learning Representations

    2024

  44. [52]

    Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. 2021. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning. 12878–12889. APPENDIX A PROOF A.1 Proof of Lemma 1 Proof. For different tasks, we use different distributio...

    2021

  45. [58]

    ∑︁ 𝑠,𝑎 𝜋𝜃(𝑎|𝑠) log 𝜋𝜃(𝑎|𝑠) 𝜋𝑔𝑙𝑜𝑏𝑎𝑙(𝑎|𝑠) !# = ∑︁ 𝑠,𝑎

    has a well-known closed form: 𝐷KL(N(𝜇1,𝜎 2 1)∥N( 𝜇2,𝜎 2 2)) = 1 2 𝜎2 1 𝜎2 2 +(𝜇2−𝜇1)2 𝜎2 2 − 1+ log 𝜎2 2 𝜎2 1 ! . (31) Integrating over𝑎 and using Eq. (31), we obtain: 𝐷KL(𝜋𝜃𝑘∥𝜋𝑔𝑙𝑜𝑏𝑎𝑙) = 1 2[ 1 𝐾 𝐾∑︁ 𝑗=1 𝜎2(𝜃𝑗) 𝜎2(𝜃𝑘)+ 𝜇(𝜃𝑘)− 1 𝐾 Í𝐾 𝑗=1𝜇(𝜃𝑗) 2 1 𝐾 Í𝐾 𝑗=1𝜎2(𝜃𝑗) − 1− log 𝜎2(𝜃𝑘) ...

    2000

  46. [2015]

    In International Conference on Machine Learning, Vol

    Trust Region Policy Optimization. In International Conference on Machine Learning, Vol. 37. 1889–1897

  47. [2016]

    In Interna- tional Conference on Machine Learning , Vol

    Stochastic Variance Reduction for Nonconvex Optimization. In Interna- tional Conference on Machine Learning , Vol. 48. 314–323

  48. [2017]

    arXiv preprint arXiv:1707.06347 (2017)

    Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  49. [2019]

    In AAMAS 2019, Vol

    How You Act Tells a Lot: Privacy-Leaking Attack on Deep Reinforcement Learning.. In AAMAS 2019, Vol. 19. 368–376

    2019

  50. [2022]

    Nature Machine Intelligence 4, 12 (2022), 1077–1087

    Transferring policy of deep reinforcement learning from simulation to reality for robotics. Nature Machine Intelligence 4, 12 (2022), 1077–1087

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.