Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Local running statistics for each agent stop mismatched weight norms from wrecking federated RL under environmental heterogeneity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 23:01 UTC pith:NHVBOPRN

load-bearing objection Clean empirical fix for heterogeneous FedRL: private running observation stats help, shared ones hurt; the weight-norm story is unmeasured. the 3 major comments →

arxiv 2605.27385 v1 pith:NHVBOPRN submitted 2026-04-10 cs.LG cs.AI

Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity

Yiran Pang , Zhen Ni , Xiangnan Zhong This is my paper
classification cs.LG cs.AI
keywords federated reinforcement learningobservation normalizationheterogeneous environmentspersonalized FedRLdistribution shiftPPOMuJoCo
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When reinforcement-learning agents train together by averaging network weights but live in environments with different physics, their raw state streams have different scales. Those mismatched scales produce weight matrices of very different sizes, so the averaging step lets large-norm agents drown out small-norm ones. The paper shows that simply giving every agent its own continuously updated mean and variance, used only to normalize its own observations and never shared, restores balanced aggregation. On three MuJoCo robots whose body-part lengths are deliberately randomized per agent, the method yields both faster learning and substantially higher final returns than either independent learning or federated averaging without local normalization. Sharing a single global mean and variance across agents fails, confirming that the statistics themselves must stay personal.

Core claim

In federated reinforcement learning the dominant damage from heterogeneous state-transition dynamics is an imbalance of weight norms under averaging: agents whose observation variance is smaller develop larger weight norms, which then dominate the global model. Keeping a private, online-updated mean and variance for each agent and normalizing its observations with those statistics alone equalizes the norms, accelerates training, and raises final performance on morphologically heterogeneous MuJoCo tasks.

What carries the argument

Personalized observation normalization (PON): each agent maintains its own Chan-style running mean and variance of raw states, normalizes every new observation with those local statistics, and never uploads the statistics themselves; only the policy and value networks are averaged.

Load-bearing premise

The main harm of environmental heterogeneity is assumed to be input-scale mismatch that produces unequal weight norms under averaging; if other forms of non-i.i.d. behavior dominate, local normalization alone will not fix the problem.

What would settle it

On the same morphologically randomized MuJoCo suite, replace local statistics with a single shared mean/variance (or with no normalization) and check whether the performance gap reported in Table I disappears or reverses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • FedRL practitioners can keep observation normalizers completely local without any extra communication cost.
  • Sharing normalization statistics across heterogeneous agents is actively harmful and should be avoided.
  • Both actor and critic networks should be averaged together once local input scales are equalized.
  • Simple online mean/variance tracking is sufficient to stabilize federated PPO under moderate morphological diversity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same local-statistics idea may transfer to other non-i.i.d. federated RL settings (different sensors, different reward scales) once the dominant mismatch is known to be input scale.
  • If policy-induced non-stationarity outruns the running statistics, a forgetting factor or windowed variance estimator would be a natural next control.
  • The method suggests that many existing federated RL failures blamed on 'non-i.i.d. data' may actually be fixable at the input layer rather than by complicated personalization of the network itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Personalized Observation Normalization (PON) for federated reinforcement learning (FedRL) under environmental heterogeneity. Differing state-transition dynamics induce non-identical observation distributions, which the authors argue produce mismatched weight norms that cause imbalanced FedAvg updates. PON lets each agent maintain its own online running mean and variance (Chan-style incremental updates) to normalize local observations; these statistics are never shared or aggregated. The authors also show that a shared-ON variant is unstable. They integrate PON into a FedRL-PPO algorithm that averages both actor and critic parameters, and evaluate on three MuJoCo tasks (HalfCheetah, Ant, Walker2d) whose morphologies are randomized per agent. Reported results (five seeds) show faster convergence and higher final returns than independent PPO and FedRL-PPO without normalization (Table I, Fig. 5).

Significance. If the empirical gains hold under broader conditions, PON is a simple, low-overhead, privacy-compatible preprocessing step that improves FedRL sample efficiency on heterogeneous continuous-control tasks. The explicit ablation of shared versus private statistics is a clear, falsifiable contribution, and the multi-environment MuJoCo suite with morphological randomization is a useful testbed. The work does not claim theoretical convergence rates or machine-checked proofs; its value is primarily empirical and algorithmic. The mechanistic story of “norm overshadowing” is secondary and currently unmeasured, so the paper’s lasting impact rests on the practical recipe rather than the toy derivation.

major comments (3)
  1. [Section III.A, Eqs. (8)–(10), Fig. 2] Section III.A (Eqs. 8–10 and the surrounding equilibrium argument) asserts that heterogeneous input variances force ||W_i||_2 ∝ 1/σ_xi, so that large-norm agents dominate FedAvg (illustrated by Fig. 2). The abstract and introduction therefore claim that PON “ensures consistent scaling o without overshadowing across agents.” No weight-norm statistics, layer-scale ratios, gradient magnitudes, or pre-/post-aggregation imbalance metrics are ever reported. Consequently the large lifts in Table I / Fig. 5 could be explained entirely by the well-known stabilizing effect of running observation normalization on PPO, with personalization required only because a shared (μ,σ) mismatches the local state distributions. The central causal claim linking PON to balanced aggregation therefore remains untested.
  2. [Section IV.B, Table I, Fig. 5] The experimental design compares only independent PPO and FedRL-PPO without any normalization. While the shared-ON ablation (Fig. 6) is informative, the paper does not benchmark against other personalization techniques already cited (Jin et al. embedding layers, Xiong et al. perFedDC, Fallah-style meta-initialization) nor against a non-federated PPO that simply applies local running ON. Without these controls it is impossible to isolate how much of the reported gain is due to federation-plus-personalization versus ordinary local normalization.
  3. [Section IV.A] All experiments use exactly three agents and morphological randomization only. The motivating overshadowing argument is an aggregation phenomenon whose severity should grow with the number of clients and with additional sources of non-i.i.d. behavior (reward-scale differences, sensor noise, etc.). The current regime is too narrow to support the general claim that PON “addresses the challenge of input distribution heterogeneity in FedRL.”
minor comments (4)
  1. [Section IV] Typographical errors: “Impliment details” (IV.A), “parameters shearing” (caption of Fig. 4), “INDL” used without definition (IV.B).
  2. [Figures 1, 3] Fig. 1(b) and Fig. 3 captions refer to “three agents” but do not state which morphological parameters were used for the plotted trajectories; a short table of the sampled lengths/scales would improve reproducibility.
  3. [Section IV.A] The PPO network description (two 64-unit layers, Tanh) is given, yet the precise clipping range, GAE λ, and entropy coefficient are omitted; these are known to interact with observation normalization.
  4. [Section II.B] Equation (2) writes y_i = W_i x_i + b while later aggregation treats a single global b; the notation for per-agent versus global bias should be made consistent.

Circularity Check

0 steps flagged

No circularity: empirical FedRL recipe with standard online normalization and an illustrative (unfitted) toy model; performance claims rest on measured returns, not self-referential derivation.

full rationale

The paper's central claims are (1) that local running-mean/variance observation normalization (PON) stabilizes FedRL-PPO under morphological heterogeneity and (2) that sharing the same statistics is harmful. Both are established by direct experiment (Figs. 5-6, Table I) on held-out environment steps; the Chan-style online updates (Eqs. 11-15) are standard, parameter-free relative to the policy objective, and independent of the reported returns. The Sec. III.A toy argument (Eqs. 8-10 equating output variance to a global equilibrium Δy and therefore ||W_i|| ∝ 1/σ_xi) is purely motivational and never fitted to, or claimed to predict, the MuJoCo numbers; no uniqueness theorem, self-citation chain, or fitted quantity is re-labeled as a first-principles result. Consequently the derivation chain contains no self-definitional step, no fitted-input-called-prediction, and no load-bearing self-citation. Score 0 is the correct, non-manufactured finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The paper rests on standard RL/FL machinery plus a small set of modeling choices that define the heterogeneity regime and the normalization procedure. No new physical entities are postulated; free parameters are ordinary hyper-parameters and morphology ranges chosen by the authors.

free parameters (4)
  • learning rate = 3e-4
    Fixed at 3×10^{-4}; controls local PPO updates and therefore the rate at which observation statistics and policy co-evolve.
  • morphology sampling intervals = bthigh [0.10,0.19], fthigh [0.10,0.17], etc.
    Hand-chosen ranges for thigh/shin lengths and leg scale factors that define the degree of heterogeneity; different ranges would change the severity of the input-distribution shift.
  • network width / depth = 64-64 MLP
    Two hidden layers of 64 units; affects capacity and therefore how strongly weight norms can diverge.
  • epsilon for variance
    Small positive constant added under the square root to avoid division by zero; standard but still a free numerical choice.
axioms (4)
  • domain assumption PPO clipped surrogate and value-function losses remain stable under federated averaging of both actor and critic parameters.
    Invoked throughout Sec. II.B and III.D; no proof that simultaneous actor-critic averaging preserves the trust-region property under non-i.i.d. transitions.
  • ad hoc to paper At equilibrium the global model’s output feature variance converges to a common value Δy across agents (toy model).
    Sec. III.A, Eqs. 8–10; used to derive ||W_i|| ∝ 1/σ_xi and thereby motivate normalization. Not derived from first principles.
  • domain assumption Chan et al. online mean/variance update correctly tracks the non-stationary state distribution induced by a changing policy.
    Sec. III.B, Eqs. 11–14; assumed sufficient for the entire training horizon.
  • domain assumption FedAvg of policy and value networks yields a policy that is useful on the averaged MDP of the heterogeneous environments.
    Cited from Jin et al. [21]; used to justify the global aggregation step.
invented entities (1)
  • Personalized Observation Normalization (PON) no independent evidence
    purpose: Local, never-shared running mean and variance used to normalize each agent’s raw observations before the policy network.
    The central algorithmic contribution; defined by the update rules in Sec. III.B and the decision not to average the statistics.

pith-pipeline@v1.1.0-grok45 · 15170 in / 3035 out tokens · 39610 ms · 2026-07-12T23:01:34.197369+00:00 · methodology

0 comments
read the original abstract

Federated reinforcement learning (FedRL) enables multiple agents to collaboratively train a global policy without sharing raw data, making it ideal for privacy-sensitive applications. However, FedRL faces challenges in heterogeneous environments where differing state-transition dynamics lead to non-identical input distributions and imbalanced parameter updates during aggregation. Therefore, this paper develops a personalized observation normalization (PON) method, allowing each agent to locally normalize raw state inputs using a continuously updated running mean and variance. This design ensures consistent scaling of local feature without overshadowing across agents during aggregation. Furthermore, we demonstrate that sharing normalization parameters across agents is ineffective due to the diverse local input distributions, which highlights the necessity of personalized statistics. Experiments on heterogeneous MuJoCo tasks show that our developed PON accelerates training and achieves superior performance compared to baseline methods.

Figures

Figures reproduced from arXiv: 2605.27385 by Xiangnan Zhong, Yiran Pang, Zhen Ni.

Figure 1
Figure 1. Figure 1: (a) Illustration of the FedRL system. The global model is distributed to and aggregated from local models, with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the issue where a model with a smaller norm is overshadowed by one with a larger norm during [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The train observation variation over environment steps [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The mean observation trajectories over training steps [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of independent learning PPO, FedRL-PPO (PPO with FedAvg for Actor and Critic), and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustrates the FedRL training curves of PON and its [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 6 linked inside Pith

  1. [1]

    A method of path planning and intelligent exploration for robot based on deep reinforcement learning,

    X. Lyu, Z. Zang, and S. Li, “A method of path planning and intelligent exploration for robot based on deep reinforcement learning,” in2024 International Joint Conference on Neural Networks (IJCNN), 2024, pp. 1–8

    2024

  2. [2]

    Learning from demonstrations: A computationally efficient inverse reinforcement learning approach with simplified implementation,

    Y . Lin, Z. Ni, and X. Zhong, “Learning from demonstrations: A computationally efficient inverse reinforcement learning approach with simplified implementation,”IEEE Transactions on Emerging Topics in Computational Intelligence, pp. 1–13, 2025

    2025

  3. [3]

    Deliverai: Reinforcement learning based distributed path-sharing network for food deliveries,

    A. Mehra, S. Saha, V . Raychoudhury, and A. Mathur, “Deliverai: Reinforcement learning based distributed path-sharing network for food deliveries,” in2024 International Joint Conference on Neural Networks (IJCNN), 2024, pp. 1–9

    2024

  4. [4]

    Simion zoo: A training workbench for reinforcement learning allowing distributed experimentation,

    B. Fernandez-Gauna and M. Gra ˜na, “Simion zoo: A training workbench for reinforcement learning allowing distributed experimentation,”Neu- rocomputing, vol. 568, p. 127030, 2024

    2024

  5. [5]

    Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents,

    A. Das, Z. Ni, and X. Zhong, “Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents,”International Journal of Electrical Power & Energy Systems, vol. 152, p. 109210, 2023

    2023

  6. [6]

    An imitation learning method with multi-virtual agents for microgrid energy optimization under interrupted periods,

    Y . Lin, Z. Ni, and Y . Tang, “An imitation learning method with multi-virtual agents for microgrid energy optimization under interrupted periods,” in2024 IEEE Power & Energy Society General Meeting (PESGM), 2024, pp. 1–5

    2024

  7. [7]

    Stfl: Utilizing a semi-supervised, transfer-learning, federated-learning approach to detect phishing url attacks,

    I. Sakazi, E. Grolman, Y . Elovici, and A. Shabtai, “Stfl: Utilizing a semi-supervised, transfer-learning, federated-learning approach to detect phishing url attacks,” in2024 International Joint Conference on Neural Networks (IJCNN), 2024, pp. 1–10

    2024

  8. [8]

    Federated learning for crowd counting in smart surveillance systems,

    Y . Pang, Z. Ni, and X. Zhong, “Federated learning for crowd counting in smart surveillance systems,”IEEE Internet of Things Journal, vol. 11, no. 3, pp. 5200–5209, 2024

    2024

  9. [9]

    Federated reinforce- ment learning for smart building joint peer-to-peer energy and carbon allowance trading,

    D. Qiu, J. Xue, T. Zhang, J. Wang, and M. Sun, “Federated reinforce- ment learning for smart building joint peer-to-peer energy and carbon allowance trading,”Applied Energy, vol. 333, p. 120526, 2023

    2023

  10. [10]

    Federated reinforcement learning in iot: applications, opportunities and open challenges,

    E. C. Pinto Neto, S. Sadeghi, X. Zhang, and S. Dadkhah, “Federated reinforcement learning in iot: applications, opportunities and open challenges,”Applied Sciences, vol. 13, no. 11, p. 6497, 2023

    2023

  11. [11]

    A fast federated reinforcement learning approach with phased weight-adjustment technique,

    Y . Pang, Z. Ni, and X. Zhong, “A fast federated reinforcement learning approach with phased weight-adjustment technique,” Neurocomputing, vol. 626, p. 129550, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S092523122500222X

    2025

  12. [12]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” inProceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 1–16

    2017

  13. [13]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,”Proceedings of Machine learning and systems, vol. 2, pp. 429–450, 2020

    2020

  14. [14]

    The non-iid data quagmire of decentralized machine learning,

    K. Hsieh, A. Phanishayee, O. Mutlu, and P. Gibbons, “The non-iid data quagmire of decentralized machine learning,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 4387–4398

    2020

  15. [15]

    Calfat: Calibrated federated ad- versarial training with label skewness,

    C. Chen, Y . Liu, X. Ma, and L. Lyu, “Calfat: Calibrated federated ad- versarial training with label skewness,”Advances in Neural Information Processing Systems, vol. 35, pp. 3569–3581, 2022

    2022

  16. [16]

    A review of personalized federated reinforcement learning,

    G. Chen and Q. Wu, “A review of personalized federated reinforcement learning,”International Journal of Computer Science and Information Technology, vol. 3, no. 1, pp. 1–9, 2024

    2024

  17. [17]

    Personalized federated learning: A meta-learning approach,

    A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning: A meta-learning approach,”arXiv preprint arXiv:2002.07948, 2020

    Pith/arXiv arXiv 2002

  18. [18]

    Federated reinforcement learn- ing for fast personalization,

    C. Nadiger, A. Kumar, and S. Abdelhak, “Federated reinforcement learn- ing for fast personalization,” in2019 IEEE Second International Con- ference on Artificial Intelligence and Knowledge Engineering (AIKE). IEEE, 2019, pp. 123–127

    2019

  19. [19]

    Federated reinforcement learning: Techniques, applications, and open challenges,

    J. Qi, Q. Zhou, L. Lei, and K. Zheng, “Federated reinforcement learning: Techniques, applications, and open challenges,”arXiv preprint arXiv:2108.11887, 2021

    Pith/arXiv arXiv 2021

  20. [20]

    Dpfed: Toward fair personalized federated learning with fast convergence,

    J. Wu, X. Liu, J. Liu, M. Hu, and D. Wu, “Dpfed: Toward fair personalized federated learning with fast convergence,” in2022 18th International Conference on Mobility, Sensing and Networking (MSN). IEEE, 2022, pp. 510–517

    2022

  21. [21]

    Federated rein- forcement learning with environment heterogeneity,

    H. Jin, Y . Peng, W. Yang, S. Wang, and Z. Zhang, “Federated rein- forcement learning with environment heterogeneity,” inInternational Conference on Artificial Intelligence and Statistics. PMLR, 2022, pp. 18–37

    2022

  22. [22]

    Personalized federated reinforcement learning: Balancing personalization and experience shar- ing via distance constraint,

    W. Xiong, Q. Liu, F. Li, B. Wang, and F. Zhu, “Personalized federated reinforcement learning: Balancing personalization and experience shar- ing via distance constraint,”Expert Systems with Applications, vol. 238, p. 122290, 2024

    2024

  23. [23]

    Fedbn: Feder- ated learning on non-iid features via local batch normalization,

    X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “Fedbn: Feder- ated learning on non-iid features via local batch normalization,”arXiv preprint arXiv:2102.07623, 2021

    Pith/arXiv arXiv 2021

  24. [24]

    Crossq: Batch normalization in deep rein- forcement learning for greater sample efficiency and simplicity,

    A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters, “Crossq: Batch normalization in deep rein- forcement learning for greater sample efficiency and simplicity,” in International Conference on Learning Representations (ICLR), 2024

    2024

  25. [25]

    Implementation matters in deep policy gradients: A case study on ppo and trpo,

    L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry, “Implementation matters in deep policy gradients: A case study on ppo and trpo,”arXiv preprint arXiv:2005.12729, 2020

    Pith/arXiv arXiv 2005

  26. [26]

    Stable-baselines3: Reliable reinforcement learning implementations,

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,”Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021. [Online]. Available: http://jmlr.org/papers/v22/20- 1364.html

    2021

  27. [27]

    Observation normalization,

    EnliteAI GmbH, “Observation normalization,” https://maze- rl.readthedocs.io, 2021, revision 94e91250

    2021

  28. [28]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  29. [29]

    Re- thinking normalization methods in federated learning,

    Z. Du, J. Sun, A. Li, P.-Y . Chen, J. Zhang, H. H. Li, and Y . Chen, “Re- thinking normalization methods in federated learning,” inProceedings of the 3rd International Workshop on Distributed Machine Learning, 2022, pp. 16–22

    2022

  30. [30]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics. PMLR, 2017, pp. 1273– 1282

    2017

  31. [31]

    Updating formulae and a pairwise algorithm for computing sample variances,

    T. F. Chan, G. H. Golub, and R. J. LeVeque, “Updating formulae and a pairwise algorithm for computing sample variances,” inCOMPSTAT 1982 5th Symposium held at Toulouse 1982: Part I: Proceedings in Computational Statistics. Springer, 1982, pp. 30–41

    1982

  32. [32]

    Gymnasium: A standard interface for reinforcement learning environments,

    M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goul ˜ao, A. Kallinteris, M. Krimmel, A. KGet al., “Gymnasium: A standard interface for reinforcement learning environments,”arXiv preprint arXiv:2407.17032, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Pytorch,

    S. Imambi, K. B. Prakash, and G. Kanagachidambaresan, “Pytorch,” Programming with TensorFlow: solution for edge computing applica- tions, pp. 87–104, 2021

    2021

  34. [34]

    Tianshou: A highly modularized deep reinforcement learning library,

    J. Weng, H. Chen, D. Yan, K. You, A. Duburcq, M. Zhang, Y . Su, H. Su, and J. Zhu, “Tianshou: A highly modularized deep reinforcement learning library,”Journal of Machine Learning Research, vol. 23, no. 267, pp. 1–6, 2022. [Online]. Available: http://jmlr.org/papers/v23/21-1127.html

    2022