{"id":"c8f01209-dd7c-4370-924c-a8e902e9aeb2","arxiv_id":"2505.09959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Federated averaging of behavior-metric-based state projection networks improves cross-environment generalization in federated reinforcement learning, while the claimed privacy protection is not demonstrated.","lead":"This paper proposes FedRAG, a federated reinforcement learning method in which clients share only the parameters of a learned state projection network through a central server. The authors report that this sharing improves generalization to unseen environments on DeepMind Control Suite tasks and argue that it protects privacy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The privacy claim central to the abstract is unsupported and likely false: Eq. 15 trains omega on raw private states and rewards, so shared omega encodes state information despite Section 4.4's assertion to the contrary.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the Section 4.4 privacy argument incorrectly claims omega is independent of private states. My reading confirms this is not merely an overstatement but an internal contradiction. Eq. 15 trains omega on raw state observations through phi_omega(s), the RAG representation loss, and the reward/dynamics approximators; therefore omega is a function of the private data. The Bayesian inference attack framework in Eq. 16 explicitly measures dependency between D_k and exposed W^S_k, and the authors do not compute or bound that dependency. The abstract's claim 'shares no sensitive task-specific information' is thus not supported. This is the central claim because privacy preservation is a stated motivation of FedRAG, not a side remark. The empirical performance claim is separate and may hold, so the appropriate resolution is to require the authors to either weaken the privacy claim to a comparative statement, add a formal privacy guarantee such as differential privacy, or provide attack experiments. Since the reader's CONDITIONAL verdict already captures this need, no verdict change is required. I do not see a more damaging internal error in the empirical method itself; the reported improvements are plausible and the ablation in Appendix A.2 supports the two sharing mechanisms. The unproven Theorem 2 is a secondary issue and does not affect the central privacy claim.","tokens_in":14085,"tokens_out":2225,"duration_ms":25711,"concrete_test":"Run a model-inversion or property-inference attack against the shared omega. Concretely: train a client for a subset of episodes, expose only its uploaded omega_k (or the updates), then train a decoder or logistic-regression probe to reconstruct raw observations or to infer the client's environment parameter (e.g., pole length in cartpole-swing). Compare classification accuracy or reconstruction error against a random-guessing baseline and against a baseline that uses a randomly initialized, untrained phi. If the probe performs significantly better than chance, the shared parameters carry recoverable private information and the 'nothing to do with our private data state' assertion is falsified. An analytic complement: compute d L_FedRAG / d s for a raw input state; if this gradient is nonzero, omega depends on private states by construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that FedRAG 'shares no sensitive task-specific information, yet provides information gain.' The performance component is plausible from the reported curves, but the privacy component is the load-bearing part and it is not established. Section 4.4 argues that omega 'is only related to the mapped state and reward, and has nothing to do with our private data state.' This is internally inconsistent: the mapped state phi_omega(s) is itself a function of the private raw state s, and the FedRAG loss in Eq. 15 is an expectation over D_k containing tuples (s, a, r, s'). The gradient of Eq. 15 with respect to omega therefore depends on the raw private states through both the representation loss and the dynamics/reward approximators. Sharing omega thus leaks information about the local state distribution, and the paper's own Bayesian inference attack formalism (Eq. 16) defines exactly the dependency that the authors then assert, without proof, to be absent. No differential privacy mechanism, noise injection, or attack evaluation is provided. In the absence of such support, the abstract's unconditional privacy claim should not be accepted as stated; the defensible claim is at most that sharing omega reduces exposure compared with sharing raw states or gradients, not that it shares no sensitive information.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedRAG, a federated reinforcement learning framework in which each client learns a state projection function fitted to an approximate behavioral metric (the RAG distance) and only the parameters of this projection function are shared with and aggregated by a central server. Clients train local SAC policies on the projected states, with an L2 penalty pulling local projection parameters toward the global average. The authors claim that sharing the projection-function parameters improves cross-environment generalization and simultaneously protects privacy by avoiding the exchange of raw states, rewards, and dynamics. Experiments on modified DeepMind Control Suite tasks (cartpole, cheetah, finger, walker) compare FedRAG with a baseline labeled FedAvg, with FeSAC, and with local-only training, reporting improved performance in other-environment evaluation. The appendix includes proofs that the RAG distance is a contraction and that the distance upper-bounds the value-function difference.","tokens_in":14359,"tokens_out":7036,"duration_ms":66458,"significance":"Representation sharing through a behavioral-metric encoder is a plausible and relatively underexplored idea in federated RL, and the paper's empirical results, if reproducible, would demonstrate that such sharing can improve cross-environment generalization with low communication cost. The simplicity of the method (parameter averaging plus L2 regularization) is a practical strength. However, the paper's strongest advertised claim—that FedRAG 'shares no sensitive task-specific information'—is not established: the encoder is trained on private states and the shared parameters can encode information about the local state distribution, yet no formal privacy guarantee or attack evaluation is provided. The performance claim is also weakened by a nonstandard FedAvg baseline and by selection of the key hyperparameter lambda on the evaluation metric. With these issues addressed, the method could be a useful contribution to federated representation learning for RL; in its current form, the significance is largely conditional.","major_comments":[{"comment":"The privacy claim is not supported and is internally inconsistent. The text asserts that omega 'is only related to the mapped state and reward, and has nothing to do with our private data state,' but Eq. (15) defines L_FedRAG as an expectation over D_k, which contains raw private states (s, a, r, s'), and the representation phi_omega(s) is a deterministic function of the private state s. The gradient of Eq. (15) with respect to omega therefore depends on the raw state distribution, so sharing omega can leak information about local state observations (e.g., via model inversion or membership inference). The paper provides no formal privacy guarantee (e.g., differential privacy), no noise mechanism, and no attack evaluation. The abstract's assertion that FedRAG 'shares no sensitive task-specific information' should be withdrawn or replaced by a bounded statement such as 'does not directly transmit raw states or rewards'; otherwise the central claim is unsupported.","section":"Section 4.4"},{"comment":"The FedAvg baseline is defined as FedRAG with lambda=0, but this is not standard FedAvg. With lambda=0, the client still optimizes the RAG loss in Eq. (13) and the global encoder parameters are still averaged; this is a representation-learning variant, not the standard federated averaging of policy or Q networks. Consequently, the reported improvement over 'FedAvg' may be attributable to the behavioral-metric representation loss rather than to the proposed collaboration mechanism. A proper FedAvg baseline (e.g., federated averaging of the SAC actor-critic parameters, or of the encoder without the RAG loss) is needed to support the claim that FedRAG's specific design is responsible for the gains.","section":"Section 5.2"},{"comment":"The regularization weight lambda is selected on the same evaluation metric used to report the final results. Section 5.3 reports that 'the optimal performance was achieved at lambda=0.001' based on performance in other environments, and this value is then reused in Figs. 4, 5, 6, 9, and 10. Since the selection is made on the test environments, the reported cross-environment improvements may reflect hyperparameter fitting rather than a general property of the algorithm. The paper should either report a separate validation split or demonstrate that the gains are stable across a range of lambda values.","section":"Section 5.3 and Figs. 3-6"},{"comment":"Equation (13) as displayed defines L_RAG as a difference of two squared terms, i.e., (d_hat(...) - gamma*d_hat(...))^2 - (|r_i - r_j|^2 - sigma_i^2 - sigma_j^2)^2. This objective is not a valid regression loss: it can be negative, is unbounded below, and does not become zero when the estimated distance matches the RAG distance. The intended loss from the RAG literature should be the square of the difference between the embedding distance and the target metric (including the square-rooted reward-variance term), not the difference of two independent squared terms. If this is a typesetting error, the correct expression must be given, because Algorithm 1 and Eq. (15) inherit the same malformed form.","section":"Equation (13)"},{"comment":"The manuscript does not state the number of random seeds, does not show error bars or confidence intervals in most figures, and does not provide code or a description of the compute environment. The text in Section 5.2 mentions 'standard deviation,' but the figures appear to show single curves without uncertainty bands. Without these details, the claimed improvements cannot be distinguished from training variance, which is essential for an empirical paper in this area.","section":"Sections 5.1-5.6 and Appendix A"}],"minor_comments":[{"comment":"The main text states that an episode consists of 125 environment steps and that 4000 episodes yield 500,000 steps, but Table 1 lists 'Episode length 1000.' Please reconcile this inconsistency.","section":"Section 5.1 and Table 1"},{"comment":"The hyperparameter K in the MICo-style distance (Eq. 14) is introduced as a hyperparameter but is not listed in Table 1, despite the table reporting other hyperparameters such as alpha_RAG and alpha_P. Please report its value or explain how it is set.","section":"Table 1 and Eq. (14)"},{"comment":"The justification 'According to the central limit theorem, we approximate the global Gaussian distribution by aggregating the mean of all local omega_k' is imprecise: averaging parameters does not follow from the central limit theorem, and the notion of a 'global Gaussian distribution' over parameters is not defined. This should be reworded as a design choice.","section":"Section 4.3"},{"comment":"The synchronization period is described as 'every 4 episodes,' which seems short relative to one episode of 1000 steps; if this is a typo, the intended number of local update steps should be specified.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The privacy claim is the most fragile part of the paper. If the authors cannot provide a rigorous privacy analysis or an attack evaluation, the abstract and introduction should be revised to remove the claim that FedRAG 'shares no sensitive task-specific information.' The paper may then be acceptable as an empirical representation-learning method, provided the baseline and hyperparameter-selection issues are resolved. The manuscript also needs a reproducibility pass (seeds, error bars, code or a detailed implementation description) before it can be judged on its empirical merits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is modest but real: taking the RAG behavioral-metric encoder from Chen and Pan and federating only the encoder parameters is a sensible combination that hasn't been tried, and the experiments are reasonably thorough for a conference paper. Five DMC tasks, ablations over the replace-vs-regularize components, a background-distraction set, and an unseen-environment test give the core empirical claim—better cross-environment generalization—decent support. The contraction proof in Appendix B is a known result, but it's correct and not a problem.\n\nThe soft spots are in proportion. The abstract's \"shares no sensitive task-specific information\" is not supported. Section 4.4's argument that omega \"is only related to the mapped state and reward, and has nothing to do with our private data state\" contradicts Eq. 15, where the RAG loss is an expectation over D_k containing raw states, and omega is trained through phi_omega(s). A neural encoder trained on private states encodes information about them. The paper offers no differential privacy, no noise injection, and no attack evaluation; its own Bayesian inference formalism (Eq. 16) defines the dependency the authors then assume away. The defensible claim is at most that sharing projection parameters reduces exposure compared with sharing raw states or gradients, not that no sensitive information is shared. This needs fixing, not just polishing.\n\nThe FedAvg baseline is also misleading. FedAvg is defined as FedRAG with lambda=0, which still federates the projection parameters and replaces local ones with the global average. That's a stripped-down FedRAG, not standard FedAvg. The comparison shows the effect of the L2 proximal term, not FedRAG vs a standard benchmark. The authors should compare against sharing Q-policy parameters (e.g., FeSAC) or a FedProx-style method, otherwise the claim \"outperforms FedAvg\" is not the contribution it appears to be.\n\nReproducibility is a real concern: no code, no reported number of seeds, and an inconsistency between Section 5.1 (episode length 125) and Table 1 (episode length 1000). These are fixable but need attention before the empirical results can be fully trusted.\n\nThe core idea is worth taking seriously, and the empirical pattern is plausible. This is not a desk reject; it deserves peer review, but with major revision: tone down the privacy claim to what is actually shown, add a genuine FedAvg baseline, provide code and seeds, and fix the experimental description. A careful referee should be able to steer the authors toward a defensible version.\n\nBottom line: worth engaging, not worth taking the privacy claim at face value.","headline":"Plausible empirical extension—sharing RAG-based state projection parameters in federated RL—but the privacy claim overreaches and the FedAvg baseline is not what it claims to be.","tokens_in":14858,"tokens_out":2452,"would_cite":false,"duration_ms":26417,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated RL clients can generalize across environments by sharing only a behavior-metric state projection, keeping raw observations private.","keywords":["federated reinforcement learning","behavioral metric","state projection","representation learning","heterogeneous environments","privacy-preserving federated learning","RAG distance","Soft Actor-Critic"],"falsifier":"Run a model-inversion or gradient-leakage attack on the uploaded $\\omega_k$ in the natural-video CartPole setup from Appendix A.3: if frames from a client's local replay buffer can be reconstructed or classified above chance from the shared projection parameters, the claim that $\\omega$ has 'nothing to do with our private data state' is refuted. A positive estimate of mutual information between $\\omega_k$ and local observations would also settle the question.","tokens_in":1497,"feed_emoji":"🤖","tokens_out":2086,"duration_ms":104347,"temperature":0.7,"pith_summary":"FedRAG aims to show that federated reinforcement learning does not need to share raw states, rewards, or policies to help clients learn from one another. Each client learns a state-projection function that maps high-dimensional observations into a space where distances approximate a behavioral metric, and only the parameters of these projection functions are averaged on a central server. The paper argues that this sharing improves cross-environment generalization on DeepMind Control tasks and, as a side effect, protects sensitive task-specific information. If true, the proposal separates what is shared, a task-relevant representation, from what remains private, raw observations and transition data, which is the core trade-off federated RL must manage.","feed_headline":"Projection sharing, not raw states, boosts federated RL generalization","feed_subtitle":"Clients exchange only projection parameters, gaining cross-environment skill without exposing raw observations.","key_machinery":"The load-bearing object is the approximated behavioral-metric state projection function $\\phi_\\omega$, trained with the RAG distance. RAG is a recursive behavioral metric: the distance between two states is the difference in expected reward under the policy plus a discounted distance between expected next states; the paper approximates the intractable reward-variance term with a learned Gaussian reward model and the next-state term with a learned Gaussian dynamics model, and uses a MICo-style embedding distance with nonzero self-distance. This projection compresses each client's high-dimensional observation into task-relevant features, and it is the only object whose parameters are shared, averaged, and regularized in federated learning.","core_discovery":"The central claim is that aggregating the parameters $\\omega$ of locally learned behavioral-metric state projections gives each federated RL client useful knowledge from the others without exchanging any raw state, reward, or transition data. Each client's projection $\\phi_{\\omega_k}$ is trained so that embedding distances approximate the Reducing Approximation Gap (RAG) distance, a behavioral metric comparing expected rewards and expected next states under the local policy. The server forms a global projection by averaging the local $\\omega_k$'s, and each client's loss adds an L2 pull toward that global value while periodically replacing local parameters with global ones. The paper reports better performance in other, unseen, and visually distracted environments than local training, FedAvg, or FeSAC, and concludes that sharing projection functions gives each client information gain while task-specific data never leaves the client.","pith_inferences":["A testable extension the paper leaves implicit: run model-inversion or gradient-based attacks on the aggregated projection parameters; if local observation features can be recovered, the privacy side-effect would need explicit defenses such as noise injection or encryption.","The same parameter-sharing recipe could be applied to other behavioral metrics, such as bisimulation or contrastive similarities, since the mechanism only requires a trainable state distance; the paper demonstrates one metric.","The paper assumes the global projection parameters follow a Gaussian and averages them; comparing this with robust aggregators would show whether the gains come from averaging itself or from the particular shape assumed for the shared model distribution.","Because the projection is trained to discard task-irrelevant background, the approach suggests a broader link between behavioral regularization and privacy: representations that ignore environmental noise may leak less, a connection the paper does not quantify."],"forward_implications":["A client trained in one physical configuration can act in previously unseen variants of its task, because the shared projection encodes task-relevant behavior rather than environment-specific pixels.","Increasing environmental heterogeneity among clients no longer degrades cross-environment performance as sharply as in FedAvg, since the behavioral metric filters environment-specific noise.","Communication cost is reduced: only lightweight projection parameters are exchanged, not raw states, transitions, or full policy gradients.","Tuning the regularization weight $\\lambda$ controls a concrete trade-off: larger values improve out-of-environment generalization, while very large values slow local adaptation and hurt same-environment performance.","Replacing local parameters with global ones and applying L2 regularization are both necessary; the paper's ablation shows removing either component causes significant performance drops."],"supporting_citations":[{"why":"Supplies the MICo sampling-based state-similarity approximation whose nonzero self-distance form makes the RAG projection loss computationally practical.","marker":"[Castro et al., 2021]"},{"why":"Introduces the Reducing Approximation Gap distance, the behavioral metric that FedRAG approximates and adopts as its representation-learning objective.","marker":"[Chen and Pan, 2022]"},{"why":"Provides the maximum-entropy Soft Actor-Critic objective that each client optimizes on the projected state representation.","marker":"[Haarnoja et al., 2018a]"},{"why":"Defines FeSAC, the federated SAC formulation that FedRAG extends by sharing only projection-function parameters.","marker":"[Tang et al., 2022]"},{"why":"Supplies the FedAvg server-side averaging rule that FedRAG uses to aggregate local projection parameters into the global model.","marker":"[McMahan et al., 2017]"},{"why":"Establishes the bisimulation-metric foundation for behavioral state similarity that RAG generalizes to continuous MDPs.","marker":"[Ferns et al., 2011]"},{"why":"Provides the DeepMind Control Suite environments and tasks used for all experiments.","marker":"[Tassa et al., 2018]"}],"fun_headline_variants":["Sharing projections, not states, boosts federated RL","FedRAG: Aggregate state projections for private federated RL","Privacy-preserving FRL via behavioral metric projections","State projection aggregation enhances federated RL","Projection sharing beats raw state sharing in federated RL"],"cache_read_input_tokens":17024,"weakest_assumption_plain":"The privacy guarantee rests on the assertion that the shared projection parameters $\\omega$ are unrelated to the raw private states; if an adversary can recover state features from those parameters, the paper's central privacy claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Sharing projections, not states, boosts federated RL","FedRAG: Aggregate state projections for private federated RL","Privacy-preserving FRL via behavioral metric projections","State projection aggregation enhances federated RL","Projection sharing beats raw state sharing in federated RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1154,"prompt_tokens":818,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":434,"tokens_out":336,"duration_ms":3598,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:19:36.447095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a model-inversion or gradient-leakage attack on the uploaded $\\omega_k$ in the natural-video CartPole setup from Appendix A.3: if frames from a client's local replay buffer can be reconstructed or classified above chance from the shared projection parameters, the claim that $\\omega$ has 'nothing to do with our private data state' is refuted. A positive estimate of mutual information between $\\omega_k$ and local observations would also settle the question.","supporting_citations":[{"cited_title":"Mico: Im- proved representations via sampling-based state similarity for markov decision processes.Advances in Neural Infor- mation Processing Systems, 34:30113–30126,","cited_arxiv_id":null,"evidence_quote":"Supplies the MICo sampling-based state-similarity approximation whose nonzero self-distance form makes the RAG projection loss computationally practical."},{"cited_title":"Learn- ing representations via a robust behavioral metric for deep reinforcement learning.Advances in Neural Information Processing Systems, 35:36654–36666,","cited_arxiv_id":null,"evidence_quote":"Introduces the Reducing Approximation Gap distance, the behavioral metric that FedRAG approximates and adopts as its representation-learning objective."},{"cited_title":"Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks","cited_arxiv_id":"2212.02075","evidence_quote":"Defines FeSAC, the federated SAC formulation that FedRAG extends by sharing only projection-function parameters."},{"cited_title":"Bisimulation metrics for continuous markov decision processes.SIAM Journal on Computing, 40(6):1662–1714,","cited_arxiv_id":null,"evidence_quote":"Establishes the bisimulation-metric foundation for behavioral state similarity that RAG generalizes to continuous MDPs."}],"review_version":1}