Pith. sign in

REVIEW 3 major objections 4 minor 35 references

The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In federated PPO, updating the actor before the critic with the global critic from the previous round makes the convergence bound independent of data heterogeneity while retaining the $O(\sqrt{\log|\mathcal{A}|/T})$ rate.

desk verdict A simple, plausible update-order reversal for federated PPO with a useful convergence analysis, but the 'immune to heterogeneity' claim overreaches because the κ-free bound rests on an unproven critic-accuracy condition. read the letter →

arxiv 2506.01261 v1 pith:WE5X327L submitted 2025-06-02 cs.LG

Zhijie Xie , Shenghui Song This is my paper · ORCID
classification cs.LG
keywords federatedreinforcementlearningPPOactor-criticupdateorderdataheterogeneityconvergenceboundsautonomousdriving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a small scheduling question with a large consequence in federated reinforcement learning: should each client update its actor or its critic first inside a round of PPO? It argues that the conventional order (critic first, then actor) couples the global policy update to the divergence of the clients' local critics, so the convergence bound carries a heterogeneity penalty that grows with how different the clients' environments are. The proposed algorithm, FedRAC, simply reverses the order: the actor is updated first, using the global critic broadcast from the previous round, and only then are critics improved locally. The paper proves that, under bounded heterogeneity and sufficiently accurate policy evaluation, FedRAC keeps the same $O(\sqrt{\log|\mathcal{A}|/T})$ rate as the baseline while dropping the pairwise critic-divergence term $\kappa$ from its convergence upper bound. If the proof is right, federated PPO practitioners can reduce the damage caused by heterogeneous environments by changing an update order rather than adding communication or regularization.

What carries the argument

The load-bearing machinery is the update-order swap, expressed as a choice between two policy-evaluation targets: the baseline's purely local $Q$-target and FedRAC's global weighted $Q$-target, where the target mixes all clients' action-value functions with weights proportional to their optimal-policy visitations. The proof decomposes the per-round error into policy improvement error, worst-case stepwise log-probability difference, and aggregation error in log probability; the decisive step is Lemma 4.6, where the baseline's bound contains $\beta_t^{-1}\kappa$ while FedRAC sets the indicator $p=0$ and the term vanishes. The argument is carried by a two-layer ReLU network parameterization for policy and value functions, a bounded concentrability assumption, and the eighth-moment critic-accuracy condition, which bridges the practical local updates to the ideal global target.

What would settle it

Run FedRAC and the baseline on a deliberately heterogeneous benchmark with undertrained critics — for instance, one critic epoch per round or a much smaller critic network — and track whether FedRAC's convergence advantage grows with the heterogeneity level; if the gap narrows or vanishes as the critic quality drops, the claimed $\kappa$-free bound depends on an assumption the implementation does not guarantee.

Watch

Extended reading notes

Core claim

FedRAC's central claim is that reversing the actor-critic update order in federated PPO removes the dominant aggregation error caused by client heterogeneity. The baseline updates critics first, so each client improves its actor with its own locally drifted critic; the aggregated policy then inherits a term $\beta_t^{-1}\kappa$, where $\kappa$ measures the pairwise divergence of the clients' action-value functions. Under FedRAC, all clients update their actors using the same global critic from the previous round, and the same derivation yields a bound with no $\kappa$ term, for both Softmax and Gaussian policies, at the rate $O(\sqrt{\log|\mathcal{A}|/T})$. The paper formalizes this through an ideal critic target in which each client's critic approximates a weighted average of all clients' $Q$-functions, and shows that the practically implemented local-critic updates need to satisfy an eighth-moment accuracy condition to make that ideal target valid. Empirical results on three classical environments and an autonomous-driving traffic simulation show FedRAC achieving higher cumulative rewards and faster convergence than the baseline, including when layered on FedAvg, FedProx, and SCAFFOLD.

Load-bearing premise

The load-bearing assumption is that each client's local critic remains accurate enough, in an eighth-moment sense, on every other client's state-action distribution; the paper assumes this rather than proving it from the local critic updates, and Remark 4.3 admits the condition tightens as heterogeneity grows.

Editorial extensions

If this is right

  • For heterogeneous federated PPO, actor-first updates make the convergence upper bound independent of the pairwise critic-divergence level $\kappa$, without changing the asymptotic rate.
  • The benefit holds under both discrete Softmax and continuous Gaussian policy parameterizations, since the vanishing of $\kappa$ appears in both cases.
  • The theoretical gain is conditional on accurate policy evaluation: with narrow or undertrained critics, the accuracy conditions in Lemma 4.2 fail, and the experimental section shows FedRAC's advantage disappears.
  • FedRAC can be combined with standard federated schemes such as FedAvg, FedProx, and SCAFFOLD with only minor modifications, and in the tested environments it attains higher cumulative reward and faster convergence.
  • With sufficiently wide networks and exact enough critics, both FedRAC and the baseline converge to the globally optimal policy at $O(\sqrt{\log|\mathcal{A}|/T})$; the baseline additionally stays within a distance bounded by $\kappa$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the theory would measure the eighth-moment critic accuracy in condition (10) on real heterogeneous tasks; if that quantity is large in the operational regime where FedRAC wins, the practical gain may come from a different mechanism than the proof identifies.
  • The same update-order argument likely transfers to other federated actor-critic algorithms beyond PPO, since the cancellation of $\kappa$ comes from all actors sharing a single global critic rather than from the KL-regularized objective; testing TD3-based or SAC-based federated variants would show whether this is the active ingredient.
  • Remark 4.3 admits that condition (10) becomes stricter as heterogeneity grows, so at extreme heterogeneity the assumption may be the binding constraint rather than the update order; the performance gap between FedRAC and the baseline could therefore narrow or reverse in regimes the current experiments do not cover.
  • The argument also reframes update order as a communication design choice: any federated decomposition that puts a synchronization barrier between actor and critic updates should re-examine which side of the barrier uses which model, including asynchronous or bandwidth-limited settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the order of actor and critic updates in federated PPO and proposes FedRAC, which reverses the conventional order so that local actors are updated using the global critic from the previous round before local critics are refined. The authors claim that this removes the pairwise critic-difference term κ from the convergence bound, making FedRAC's bound immune to data heterogeneity under "mild conditions." They provide a convergence analysis for both Softmax and Gaussian policies, and they report experiments on classical RL environments and a SUMO-based autonomous driving scenario showing that FedRAC outperforms the baseline. The paper also includes ablations on network width, heterogeneity level, and scalability.

Significance. If the theoretical claim were fully established, the update-order insight would be a simple, practical contribution to federated reinforcement learning, complementing recent work on federated policy optimization. The paper's strengths include a unified treatment of Softmax and Gaussian policies, a clear algorithm description, and reproducible experiments with a provided GitHub repository. However, the headline claim of heterogeneity immunity is currently overstated: the bound still depends on heterogeneity through other quantities, and the main theoretical result rests on a strong, unproven assumption about the accuracy of local critics. The empirical findings are plausible but do not by themselves establish the theoretical claim.

major comments (3)
  1. [Section 4.1, condition (10) and Remark 4.3] Condition (10) can be unsatisfiable in heterogeneous regimes. The threshold defining \hat{ε}^{t+1}_n subtracts the nonnegative term (E^t)^{1/2} from the nonnegative term (( \dot{ε}^{t+1}_n)^2 - 8R_w^2)(Ξ_t^n)^{-1/2}. As the heterogeneity level grows, (E^t)^{1/2} can exceed the first term, making \hat{ε}^{t+1}_n negative while the left-hand side of (10) is always nonnegative. The paper provides no argument that a critic obtained by minimizing the local squared error (7) satisfies (10), nor that the cross-client visitation ratios ρ_{π*,n}/ρ_{π*,i} entering Ξ_t^n and E^t remain bounded under the stated "mild conditions." This is load-bearing because the κ-free bound for FedRAC in Lemma 4.6 and Theorem 4.8 uses Lemma 4.2, whose FedRAC case requires (10). If (10) fails, the κ-free bound is not established for the implemented algorithm.
  2. [Section 4.3 and abstract] The claim that FedRAC's convergence bound is "immune to data heterogeneity" is overstated. Even with p=0 in Lemma 4.6, the bound in Theorem 4.8 depends on the concentrability quantities ψ_t^n, the aggregation-error quantities ξ_t^n, and the coefficients Ξ_t^n and E^t used in condition (10), all of which encode heterogeneity across clients. The explicit rate in Appendix H contains the terms Σ_{t,n} q_n ε(ψ_t^n+2ξ_t^n) and Σ_{t,n} q_n \dot{ε}ψ_t^n, which are heterogeneity-dependent. The precise statement proven is that the pairwise critic-difference term κ is absent from the FedRAC bound, not that the bound is independent of heterogeneity. The abstract and Section 4.3 should be reworded to reflect this limitation.
  3. [Section 4.1, Lemma 4.2 and Algorithm 1] There is a gap between the implemented algorithm and the theoretical ideal target (8). Algorithm 1, line 7, trains each local critic by minimizing the local MSE (7) against Q^{π_{θ_t}}_n under the client's own visitation. The proof of Lemma 4.2 for FedRAC instead bounds the error of the global critic against the weighted average in (8), and it does so by assuming condition (10) for all pairs (n,i). The paper does not show that the minimizer of (7), even if it perfectly approximates the local action-value function, satisfies (10), nor that the N^2 constraints implied by (10) are compatible with the stated bounded-heterogeneity condition. Remark 4.3 concedes that the constraints become stricter as heterogeneity increases, but it does not resolve the possibility that (10) is violated for large heterogeneity. This gap should be closed by either proving (10) under explicit verifiable conditions or by presenting the theorem as conditional on (10) with a clear statement that the κ-free property applies only to that idealized setting.
minor comments (4)
  1. [Table 1 and Lemma 4.6] The quantity ξ_t^n is used in Lemma 4.6 and in the derived rate in Appendix H but is never defined in Table 1 or in the surrounding text. Please add a definition.
  2. [Section 4.3, Theorem 4.8] The notation is inconsistent: the theorem defines ε_t = Σ_n q_n ε^{t+1}_n but the derived rate and Appendix H index the same quantity as ε_t^n and ε^{t+1}_n. Please harmonize the notation.
  3. [Section 1] The sentence "As far as the author knows, the question regarding the optimal update order for federated Actor-Critic remains unanswered" should use the plural "authors" for a two-author paper.
  4. [Introduction, related work] The background discussion of FedPG-BR, PAvg, FedKL, and FedNAC is useful, but the paper would benefit from explicitly stating how the proposed analysis differs from or improves upon the convergence guarantees in FedKL and FedNAC regarding dependence on heterogeneity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the κ-free FedRAC bound is a structural consequence of the reversed update order, and the assumptions that bridge local and global critics are explicit rather than hidden fits.

full rationale

The central claim that FedRAC's convergence upper bound is free of the heterogeneity level κ is not obtained by fitting or by renaming inputs. In Algorithm 1, line 6 ('Common: Optimize θ_t for E epochs to obtain θ_{t+1}^k in (6)') runs before line 7 ('FedRAC: Optimize w_t for E epochs ... in (7)'), so the actor is updated with the shared global critic Q^{w_t}. Consequently, in the FedRAC branch of Lemma E.1's proof, the middle term Σ_i q_i |(β^{-1} Q^{w_t} + τ^{-1} f_{θ_t}) - (β^{-1} Q^{w_t} + τ^{-1} f_{θ_t})| is identically zero, and the pairwise critic-divergence term κ disappears by algebra rather than by a fitted parameter. This is the intended mechanism of the reversed order, not circularity. The paper does not conceal the bridge between the implemented local update (7) and the theoretical target (8): Section 3.4 states (8) is 'only for theoretical analysis purposes,' and Lemma 4.2 explicitly assumes condition (10), with Remark 4.3 conceding that 'as the heterogeneity level increases, the condition (10) tends to be more stringent.' That condition may indeed be hard or impossible to satisfy in highly heterogeneous regimes, since the definition of \hatt{ε}^{t+1}_n subtracts (E_t)^{1/2} and can become negative; but this is a correctness and plausibility limitation of the assumption, not a circular reduction of the conclusion to its inputs. The only self-citation is Lemma B.3, whose proof is deferred to the authors' prior FedPOHCS [33]; this is a generic two-layer ReLU linearization bound contributing the O(m^{-1/10}) term common to both FedRAC and the baseline, and it does not carry the κ-free claim. The experiments (Figures 1-3) provide independent, externally falsifiable support. Hence no step in the derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fit to data; all constants in the theory are structural (radii, temperatures, KL penalties). In the experiments, hyperparameters such as learning rate and KL target are standard choices and are listed in the appendix. No new physical or algorithmic entities are introduced beyond the FedRAC procedure itself.

assumptions (5)
  • domain assumption Assumption 4.1: The Radon-Nikodym derivatives of ρ_{π*,n}π* and ρ_{π*,n} with respect to ρ_{π_θ_t,n}π_θ_t and ρ_{π_θ_t,n} are bounded for all parameterized policies in the function class.
    Used in Lemmas 4.2 and 4.6 to control the error propagation terms; assumes different clients' state-action visitation distributions are not too far apart.
  • domain assumption Conditions (9) and (10): the local policy evaluation error is bounded; for FedRAC this must hold for every client i under client n's distribution in the eighth moment.
    These are assumptions, not derived properties; condition (10) is the key unproven requirement that makes the κ-free bound valid.
  • standard math Two-layer ReLU networks in the lazy-training regime: output layer fixed, first-layer parameters projected to a ball of radius R_θ, and Lemma B.3 bounds the linearization error by O(R^{6/5}_θ m^{-1/10} R̂^{2/5}_θ).
    Imported from prior neural-network policy-gradient work; needed for the aggregation error analysis.
  • domain assumption Softmax or Gaussian policy parameterizations with monotonically decreasing temperature τ_t or variance σ_t that are not trainable.
    Restricts the policy class; used to derive the explicit rates and the O(1/T) terms.
  • domain assumption The policy improvement error assumption in Lemma 4.2: the MSE between the updated log-policy and the target β^{-1}_t Q_w + log π_θ_t is bounded by c^{-1}_{t+1} e^{t+1}_n.
    Assumes local PPO updates do not deviate too much from the KL-regularized optimum, which is standard in neural PPO analyses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning." pith.science (2026). https://pith.science/paper/WE5X327L

@misc{pith2026250601261,
  author       = {Pith},
  title        = {Pith review of: The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WE5X327L}},
  note         = {Machine review of arXiv:2506.01261}
}
read the original abstract

In the context of Federated Reinforcement Learning (FRL), applying Proximal Policy Optimization (PPO) faces challenges related to the update order of its actor and critic due to the aggregation step occurring between successive iterations. In particular, when local actors are updated based on local critic estimations, the algorithm becomes vulnerable to data heterogeneity. As a result, the conventional update order in PPO (critic first, then actor) may cause heterogeneous gradient directions among clients, hindering convergence to a globally optimal policy. To address this issue, we propose FedRAC, which reverses the update order (actor first, then critic) to eliminate the divergence of critics from different clients. Theoretical analysis shows that the convergence bound of FedRAC is immune to data heterogeneity under mild conditions, i.e., bounded level of heterogeneity and accurate policy evaluation. Empirical results indicate that the proposed algorithm obtains higher cumulative rewards and converges more rapidly in five experiments, including three classical RL environments and a highly heterogeneous autonomous driving scenario using the SUMO traffic simulator.

Figures

Figures reproduced from arXiv: 2506.01261 by the authors.

Figure 1
Figure 1. Comparisons on different simulations. (a) Hoppers; (b) Half Cheetahs; (c) HongKongOSMs. [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Learning curves on Hoppers with different neural network settings: (a) 4-layer MLPs for [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Learning curves on Mountain Cars with different levels of heterogeneity and a large [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: OSM data of ten areas in Hong Kong. (a) Causeway; (b) Central; (c) Chai Wan; (d) Clear [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 27 canonical work pages

  1. [1]

    Fitted q-iteration in continuous action-space mdps

    Antos, A., Szepesvári, C., and Munos, R. Fitted q-iteration in continuous action-space mdps. In Platt, J., Koller, D., Singer, Y ., and Roweis, S. (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007

    work page 2007

  2. [2]

    S., and Guin, S

    Bhatnagar, S., Borkar, V . S., and Guin, S. Actor–critic or critic–actor? a tale of two time scales. IEEE Control Systems Letters, 7:2671–2676, 2023. doi: 10.1109/LCSYS.2023.3288931

    arXiv 2023

  3. [3]

    Openai gym

    Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. CoRR, abs/1606.01540, 2016

    arXiv 2016

  4. [4]

    D., and Wang, Z

    Cai, Q., Yang, Z., Lee, J. D., and Wang, Z. Neural temporal-difference learning converges to global optima. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019

    work page 2019

  5. [5]

    W., Hilton, J., Klimov, O., and Schulman, J

    Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2020–2027. PMLR, 18–24 Jul 2021

    work page 2020

  6. [6]

    Linear off-policy actor-critic

    Degris, T., White, M., and Sutton, R. Linear off-policy actor-critic. In ICML, 2012

    work page 2012

  7. [7]

    A theoretical analysis of deep q-learning

    Fan, J., Wang, Z., Xie, Y ., and Yang, Z. A theoretical analysis of deep q-learning. In Bayen, A. M., Jadbabaie, A., Pappas, G., Parrilo, P. A., Recht, B., Tomlin, C., and Zeilinger, M. (eds.), Proceedings of the 2nd Conference on Learning for Dynamics and Control , volume 120 of Proceedings of Machine Learning Research, pp. 486–489. PMLR, 10–11 Jun 2020

    work page 2020

  8. [8]

    Fan, X., Ma, Y ., Dai, Z., Jing, W., Tan, C., and Low, B. K. H. Fault-tolerant federated reinforcement learning with theoretical guarantee. Advances in Neural Information Processing Systems, 34, 2021

    work page 2021

Show all 35 references
  1. [9]

    Error propagation for approximate policy and value iteration

    Farahmand, A.-m., Szepesvári, C., and Munos, R. Error propagation for approximate policy and value iteration. In Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., and Culotta, A. (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, In...

    2010

  2. [10]

    Reinforcement learning with deep energy- based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy- based policies. In Precup, D. and Teh, Y . W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 13...

    2017

  3. [11]

    Federated reinforcement learning with environment heterogeneity

    Jin, H., Peng, Y ., Yang, W., Wang, S., and Zhang, Z. Federated reinforcement learning with environment heterogeneity. In Camps-Valls, G., Ruiz, F. J. R., and Valera, I. (eds.),Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 1...

    2022

  4. [12]

    P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A

    Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. SCAFFOLD: Stochastic controlled averaging for federated learning. In III, H. D. and Singh, A. (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedin...

    2020

  5. [13]

    and Tsitsiklis, J

    Konda, V . and Tsitsiklis, J. Actor-critic algorithms. In Solla, S., Leen, T., and Müller, K. (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999

    1999

  6. [14]

    K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V

    Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V . Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, volume 2, pp. 429–450, 2020

    2020

  7. [15]

    On the convergence of fedavg on non-iid data

    Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. In ICLR 2020 : Eighth International Conference on Learning Representations, 2020

    2020

  8. [16]

    Neural trust region/proximal policy optimization attains globally optimal policy

    Liu, B., Cai, Q., Yang, Z., and Wang, Z. Neural trust region/proximal policy optimization attains globally optimal policy. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , vol...

    2019

  9. [17]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. Communication- Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54, pp. 1273–1282, 20–22 Apr 2017

    2017

  10. [18]

    On the global convergence rates of softmax policy gradient methods

    Mei, J., Xiao, C., Szepesvari, C., and Schuurmans, D. On the global convergence rates of softmax policy gradient methods. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Res...

    2020

  11. [19]

    A., Veness, J., Bellemare, M

    Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  12. [20]

    and Szepesvári, C

    Munos, R. and Szepesvári, C. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(27):815–857, 2008

    2008

  13. [21]

    Planet dump retrieved from https://planet.osm.org

    OpenStreetMap contributors. Planet dump retrieved from https://planet.osm.org . https: //www.openstreetmap.org, 2017

    2017

  14. [22]

    I., and Abbeel, P

    Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. Trust region policy optimiza- tion. CoRR, abs/1502.05477, 2015

    2015 arXiv

  15. [23]

    Proximal policy optimiza- tion algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms. CoRR, abs/1707.06347, 2017

    2017 arXiv

  16. [24]

    Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3 (1):9–44, 1988

    1988

  17. [25]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y . Policy gradient methods for reinforce- ment learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, 2000. 11

    2000

  18. [26]

    Boosted fitted q-iteration

    Tosatto, S., Pirotta, M., D’Eramo, C., and Restelli, M. Boosted fitted q-iteration. In Precup, D. and Teh, Y . W. (eds.),Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3434–3443. PMLR, 06–11 Aug 2017

    2017

  19. [27]

    Deep reinforcement learning with double q-learning

    Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  20. [28]

    E., Srivastava, S., Tuia, D., and Falcão, A

    Vargas-Munoz, J. E., Srivastava, S., Tuia, D., and Falcão, A. X. Openstreetmap: Challenges and opportunities in machine learning and remote sensing. IEEE Geoscience and Remote Sensing Magazine, 9(1):184–199, 2021. doi: 10.1109/MGRS.2020.2994107

    2021

  21. [29]

    L., Kheterpal, N., Jang, K., Wu, C., Wu, F., Liaw, R., Liang, E., and Bayen, A

    Vinitsky, E., Kreidieh, A., Flem, L. L., Kheterpal, N., Jang, K., Wu, C., Wu, F., Liaw, R., Liang, E., and Bayen, A. M. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Proceedings of The 2nd Conference on Robot Learning, volume 87, pp. 399–409, 2018

    2018

  22. [30]

    Neural policy gradient methods: Global optimality and rates of convergence

    Wang, L., Cai, Q., Yang, Z., and Wang, Z. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150, 2019

    1909 arXiv

  23. [31]

    P., and Kakade, S

    Wang, R., Foster, D. P., and Kakade, S. M. What are the statistical limits of offline rl with linear function approximation?, 2020

    2020

  24. [32]

    and Song, S

    Xie, Z. and Song, S. Fedkl: Tackling data heterogeneity in federated reinforcement learning by penalizing kl divergence. IEEE Journal on Selected Areas in Communications , 41(4): 1227–1242, 2023. doi: 10.1109/JSAC.2023.3242734

    2023

  25. [33]

    and Song, S

    Xie, Z. and Song, S. H. Client selection for federated policy optimization with environment heterogeneity. arXiv preprint arXiv:1812.06210, 2023

    2023 arXiv

  26. [34]

    Federated natural policy gradient and actor critic methods for multi-task reinforcement learning

    Yang, T., Cen, S., Wei, Y ., Chen, Y ., and Chi, Y . Federated natural policy gradient and actor critic methods for multi-task reinforcement learning. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Informa...

    2024

  27. [35]

    Qwt i (s, a) − ρπ∗,i(s) Zπ∗ (s) Qπθt i (s, a) 2# . By the fact that (a + b)2 ≤ 2a2 + 2b2 and ab − cd = a(b − d) + d(a − c), we have Es∼ρπθt ,n,a∼πθt

    Zhao, Y ., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V . Federated learning with non-iid data. CoRR, abs/1806.00582, 2018. 12 APPENDICES A Additional Implementation Details and Experiment Setting The hyperparameters for different experiments and the general FRL settin...

    2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.