{"id":"a9906477-9c26-4a93-b253-f2e2b51e76a8","arxiv_id":"2412.15538","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"FedRLHF decentralizes RLHF across clients that locally shape rewards with private human feedback, with claimed convergence guarantees and a personalization-performance trade-off.","lead":"This paper proposes FedRLHF, a federated learning framework that lets each client fine-tune a shared policy using private human feedback without uploading raw data or feedback. The authors claim convergence guarantees, sample complexity bounds, a personalization-performance trade-off, and experiments on MovieLens and IMDb.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 4.2's cross-client term does not vanish: the proof drops E⟨∇J_k,∇J_k'⟩, which is nonzero under heterogeneity; Theorem 4.1 is therefore unsupported.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing flaw in the paper's central argument. The main advertised contribution is the convergence guarantee in Theorem 4.1, and its proof depends on Lemma 4.2's one-step descent inequality. That lemma is invalid: the cross-client terms in the squared aggregation error do not vanish merely because clients are independent, and the means of the stochastic gradients are the true client gradients, which are not zero-mean in the relevant sense. My concrete K=2, τ=1 deterministic-gradient check shows the claimed bound can be violated by a factor of K. Without an additional gradient-dissimilarity assumption (e.g., a bound on ∥∇J_k−∇J∥²), the stated O(1/T) convergence and the sample complexity bound do not follow. The rest of the paper—the personalization trade-off theorems (which also rely on an unflagged D∞≤log2 assumption) and the small-scale experiments—does not repair this hole in the main theoretical claim. Therefore I agree with the reader's REJECT verdict and have no basis to change it. My read is an independent confirmation rather than a new objection, so the verdict is unchanged.","tokens_in":28947,"tokens_out":3284,"duration_ms":29778,"concrete_test":"Take K=2 clients, τ=1 local step, deterministic gradients (σ²=0), and set ∇J_1(θ_0)=∇J_2(θ_0)=g with ∥g∥=G. Then the actual averaged-update norm squared is E∥(1/2)Σ_k(θ^k_0−θ_0)∥² = η²/4 ∥g_1+g_2∥² = η²G². The bound used in Lemma 4.2 is η²τ²/K(G²+σ²)=η²G²/2. Since η²G² > η²G²/2 for η,G≠0, the lemma's second-term bound is violated. Re-derive Lemma 4.2 without the cross-term assumption; the missing term (K−1)/K · η²τ²G² appears, confirming the descent inequality does not hold under the stated assumptions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Lemma 4.2 (Appendix A), the bound on E∥(1/K)Σ_k(θ^k_t−θ_t)∥² is justified by 'assuming independence across clients and zero-mean stochastic gradients, the cross terms vanish.' This is invalid: the stochastic gradient noise has zero mean, but the stochastic gradient itself has expectation ∇J_k(θ), which is not zero. The cross term E⟨θ^k_t−θ_t, θ^{k'}_t−θ_t⟩ equals η² Σ_{i,j} ⟨∇J_k(θ^k_{t,i}), ∇J_{k'}(θ^{k'}_{t,j})⟩ plus noise-only terms that do vanish. These inner products are generally nonzero because client objectives are heterogeneous; no assumption in the paper bounds them. For a simple homogeneous case with K clients, σ²=0, and ∇J_k ≡ g for all k, the actual second term is η²τ²∥g∥², while the proof claims (η²τ²/K)(∥g∥²+σ²), a factor K smaller for K>1. Thus Lemma 4.2's inequality is false as stated, and the contraction step in Theorem 4.1 (Appendix B) does not follow from Assumptions 1–6. The error is not a loose constant but an omitted heterogeneity term scaling with (K−1)/K; without a gradient-dissimilarity assumption, the O(1/K) variance term in the theorem has no basis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FedRLHF, a federated RLHF framework in which clients keep raw data and human feedback local, shape rewards as R_k = R0_k + λH_k, run local policy-gradient updates, and share only model deltas with a server. The main claimed contributions are a convergence guarantee for the averaged global policy under a PL condition (Theorem 4.1), a sample-complexity bound (Theorem 4.2), and a personalization–performance trade-off analysis (Theorems 5.1 and 5.2), together with experiments on MovieLens and IMDb. The central theoretical claims are not established: the one-step descent lemma relies on an invalid cancellation of cross-client terms, the human-feedback penalty is introduced by assertion rather than derivation, and the impact-of-human-feedback theorem analyzes an update rule that is not Algorithm 1.","tokens_in":29207,"tokens_out":7183,"duration_ms":60053,"significance":"The high-level direction — combining federated averaging with locally shaped RLHF rewards — is timely, and the authors make a good-faith effort to provide guarantees rather than only heuristics. The paper ships code and experiments on two datasets, and the attempt to quantify a personalization cost is conceptually interesting. However, the value of the paper rests on the rigor of the convergence and trade-off proofs, and those proofs contain load-bearing gaps. Because the convergence theorem is the paper's primary theoretical contribution, the current manuscript does not support the advertised guarantees.","major_comments":[{"comment":"The proof bounds E||(1/K)Σ_k(θ^k_t−θ_t)||² by (1/K)E||θ^k_t−θ_t||², saying that under independence across clients and zero-mean stochastic gradients 'the cross terms vanish'. The stochastic noise has zero mean, but the true gradients do not: the cross term equals η² Σ_{i,j}⟨∇J_k(θ^k_{t,i}), ∇J_{k'}(θ^{k'}_{t,j})⟩ plus a noise part that does vanish, and this term is not bounded by any assumption. For the homogeneous example with K clients, σ²=0 and ∇J_k≡g for all k, the actual value of the squared-average term is η²τ²∥g∥², while the proof claims (η²τ²/K)(∥g∥²+σ²), which is a factor K smaller. Thus Lemma 4.2 is false as stated under Assumptions 1–6, and the contraction argument in Theorem B.1 has no basis.","section":"Appendix A, Lemma A.2 (also Theorem 4.1)"},{"comment":"The final line of the proof subtracts the term −λHmax from the lower bound with the sentence 'we subtract this term from our lower bound'; this is an assertion, not a derivation. Since the local objectives J_k already include the shaped reward R_k=R0_k+λH_k, the global objective J already contains the human-feedback contribution, and Assumption 6 only bounds |H_k| pointwise. No step in the proof connects the bounded feedback to a worst-case loss of λHmax in the descent inequality. Moreover, the earlier simplification step drops the η³τ³ term as 'higher order' without conditions, even though the later choice η=1/(Lτ) gives η³τ³=1/(L³τ²), which is not automatically negligible.","section":"Appendix A, Lemma A.2 ('Accounting for human feedback')"},{"comment":"The proof of the O(λ²) personalization scaling assumes a policy update rule θ'_k = θ_k + ηλ∇_θ H_k(θ_k). No such update appears in Algorithm 1; the algorithm updates local parameters with policy-gradient estimates of J_k, where the shaped reward enters through R_k, not as a separate H_k gradient. The proof also introduces a new constant G_H without defining it from Assumptions 1–6. The final sample-complexity claim 'N = O(L(G²+σ²)/(μ²ε²) + LKλHmax/(με))' is not the expression derived in Theorem B.2, which contains no λ-dependent additive term. Consequently the O(λ²), O(λ), and O(λ) rates in Theorem 5.2 are unsupported.","section":"Appendix C, Theorem C.2"},{"comment":"The proof uses the assertion D∞(d^{π_k}_k∥ρ) ≤ log 2 to replace the state distribution d^{π_k}_k by ρ in the personalization score, which is needed to obtain the constant C. This inequality is not derived from Assumptions 1–6, the reward shaping, or the FedRLHF update; the text says only that the authors 'can assert' it. The bound is not a standard property of federated averaging, and without a proof of this divergence bound, the personalization–performance trade-off theorem does not follow.","section":"Appendix C, Theorem C.1"}],"minor_comments":[{"comment":"Theorem 4.1 is described as a 'linear convergence rate', but the statement is an O(1/T) bound on the average gap; the per-round contraction ρ=1−μ/L applies only to the homogeneous part of the bound. The wording should distinguish geometric contraction of the initial-condition term from the O(1/T) average.","section":"Section 4.2"},{"comment":"The descent coefficient is written as 1−Lη/2 in one displayed equation and later as 1−Lητ/2; the two forms are not equivalent and the proof should be consistent.","section":"Appendix A, Lemma A.2"},{"comment":"The experimental section uses Q-learning for the MovieLens task while the theory explicitly assumes policy-gradient local updates; this limits the extent to which the experiments validate Theorem 4.1 and should be stated more prominently as a limitation.","section":"Section 6.1.3"},{"comment":"The abstract and introduction claim 'robust privacy preservation', but the protocol shares model updates without formal privacy guarantees; the text correctly notes that differential privacy could be added, so the claims should be qualified accordingly.","section":"Abstract and Section 3.3"},{"comment":"The IMDb experiments are reported with a single random seed while the MovieLens experiments use five seeds; this asymmetry should be acknowledged in the main text.","section":"Appendix D.2.4"}],"recommendation":"reject","confidential_remarks":"The theoretical core is the stated contribution, and the proof gaps are systematic rather than typographical: the main descent lemma is false as stated, the human-feedback term is imposed by assertion, and the personalization theorem analyzes a different algorithm. I do not see a simple local patch that preserves the stated theorems; a revision would need to add a bounded-heterogeneity assumption to the descent analysis and re-derive the personalization trade-off from the actual update dynamics. Given the journal's standards, I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the FedRLHF framework is a reasonable and timely combination of federated averaging with per-client human-feedback-shaped rewards, and the paper is well structured. But the central convergence theorem (Theorem 4.1) is not established. Lemma 4.2's proof drops cross-client terms in the average local-global difference, and the -lambda Hmax term is inserted by assertion. The personalization theorems also rely on unproven assumptions.\n\nThe genuinely new piece is the KL-based personalization score and the explicit O(lambda^2)/O(lambda) trade-off framing. That is a useful way to quantify the cost of personalization. The experiments, though small (5 rounds, 10 or 5 clients, single-seed IMDb), show the framework running end to end, and the code is public. I give credit for that.\n\nThe soft spots are load-bearing, not cosmetic. In Lemma A.2, after expanding the average squared local-global difference, the proof says cross terms vanish because stochastic gradients are zero-mean. That is wrong: the noise is zero-mean, but the true gradients are not, so cross terms contain E<grad J_k, grad J_k'>, which is generically nonzero under heterogeneity. A simple homogeneous example with sigma=0 and identical clients makes the proof's bound off by a factor of K. You would need a gradient-dissimilarity assumption, which is not in the paper. That kills the O(1/K) variance term in Theorem 4.1. The -lambda Hmax term is also not derived from the reward shaping; it is just subtracted. Similarly, Theorem 5.1 needs D_infinity <= log2, which is asserted without proof, and Theorem 5.2 uses a postulated update rule theta'_k = theta_k + eta lambda grad H_k that is not the algorithm. So the personalization-performance trade-off, as stated, is not proven.\n\nI would still send this to serious review. The topic is important, the framework is plausible, and the errors are fixable if the authors add proper heterogeneity assumptions and derive the human-feedback term. A referee could guide that revision. But as it stands, the theoretical results do not hold.\n\nFor a reading group, this is a good case study in how federated averaging proofs can go wrong. I would not cite the convergence results, though I might cite the framework if I needed a reference for federated RLHF.","headline":"The framework is sensible and clearly described, but the main convergence proof has an invalid cross-term cancellation that leaves Theorem 4.1 unsupported; the personalization theorems also rely on unproven assumptions.","tokens_in":29791,"tokens_out":3085,"would_cite":false,"duration_ms":26722,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedRLHF claims that RLHF can run fully federated—each client updates a local policy from private human feedback—and converges at rate O(1/T) up to a bias governed by client diversity and feedback strength.","keywords":["federated reinforcement learning","RLHF","privacy-preserving machine learning","personalization","convergence guarantee","sample complexity","human feedback","policy gradient"],"falsifier":"Run Algorithm 1 with two clients whose intrinsic rewards push the policy in opposite directions and measure the cross-client term $(1/K^2)\\sum_{k\\ne k'}\\mathbb{E}\\langle\\theta_t^k-\\theta_t,\\theta_t^{k'}-\\theta_t\\rangle$ across rounds. If this term is not negligible, the one-step descent lemma's bound is violated and the empirical optimality gap will plateau above the bias predicted by Theorem 4.1, refuting the universal convergence claim for heterogeneous clients.","tokens_in":28675,"feed_emoji":"🔒","tokens_out":8892,"duration_ms":71220,"temperature":0.7,"pith_summary":"FedRLHF is a proposed way to run reinforcement learning with human feedback without moving users' raw data or preference labels to a server: each client shapes its own reward with local human feedback, takes several policy-gradient steps, and sends only the model update to be averaged. The paper's central claim is that this federated procedure converges to the optimal global policy at rate O(1/T), with an unavoidable bias that shrinks as the number of clients grows and is proportional to the human-feedback weight. This matters because centralized RLHF—the standard behind instruction-following systems—collects user data and feedback in one place, creating both privacy risks and a one-size-fits-all policy. If the claim holds, personalized, privacy-preserving fine-tuning is not a separate compromise but a method with the same asymptotic sample efficiency as centralized training.","feed_headline":"Federated RLHF converges at 1/T without sharing user feedback","feed_subtitle":"New framework keeps raw data and human preferences on-device, matches centralized RLHF, and quantifies the personalization trade-off.","key_machinery":"The load-bearing mechanism is a one-step descent lemma for the FedAvg trajectory $\\theta_{t+1}=\\theta_t+(1/K)\\sum_k(\\theta^k_t-\\theta_t)$, where each client's local update is a sum of $\\tau$ policy-gradient steps on its shaped reward $R_k=R_k^0+\\lambda H_k$. Smoothness bounds the descent, the PL condition turns the gradient-norm term into a contraction, and the human-feedback bound appears as the additive $\\lambda H_{\\max}$ bias. The personalization analysis uses the personalization score $P_k(\\theta)=\\mathbb{E}_{s\\sim\\rho}[D_{\\mathrm{KL}}(\\pi_k(\\cdot|s,\\theta)\\|\\pi(\\cdot|s,\\theta))]$ and the performance difference lemma to convert KL divergence into a global-performance penalty.","core_discovery":"Under $L$-smoothness, $G$-bounded gradients, $\\sigma^2$-bounded variance, the Polyak–Łojasiewicz condition with parameter $\\mu$, and $|H_k(s,a)|\\le H_{\\max}$, the paper proves that with learning rate $\\eta=1/(L\\tau)$ the averaged output $\\theta_{\\mathrm{avg}}$ of Algorithm 1 satisfies $$\\mathbb{E}\\left[J(\\$\\theta$^*)-J(\\theta_{\\mathrm{avg}})\\right]\\le \\frac{L}{\\mu T}\\left(J(\\$\\theta$^*)-J(\\theta_0)\\right)+\\frac{1}{2\\mu K}\\left($G^{2}$+\\$sigma^{2}$\\right)+\\frac{L}{\\mu}\\$\\lambda$ H_{\\max}.$$ The proof's one-step descent lemma converts gradient norm into progress and leaves two bias terms: stochastic variance, which shrinks as $1/K$, and human-feedback influence $\\lambda H_{\\max}$. Theorem 4.2 translates this into total sample complexity $N=O(L(G^2+\\sigma^2)/(\\mu^2\\epsilon^2))$, and Theorem 5.1 bounds global intrinsic performance below by average client performance minus a penalty proportional to the mean square-root KL personalization score.","pith_inferences":["The bound suggests a practical calibration rule the paper leaves implicit: choose $\\lambda$ no larger than $O(\\mu\\epsilon/L)$ to keep the human-feedback bias within an error budget $\\epsilon$.","The personalization score $P_k$ could be measured online and used as a stopping criterion or as a regularizer on local updates, turning Theorem 5.1's inequality into an adaptive controller.","Because the privacy guarantee is architectural rather than formal, composing FedRLHF with a mechanism such as differential privacy would make the stated privacy property a formal one; the paper notes this as future work.","Since both bias terms are independent of $T$, extra communication rounds give diminishing returns once variance and feedback bias dominate, whereas adding clients directly improves the bound—a practical consequence the paper does not spell out."],"forward_implications":["Only policy updates—never raw trajectories or preference labels—leave the client, so the framework is compatible with on-device data-governance constraints.","Increasing the number of clients $K$ tightens the convergence bound through the $1/K$ variance term and lowers per-client sample complexity.","The human-feedback weight $\\lambda$ is a controllable dial: larger $\\lambda$ raises the personalization score at order $\\lambda^2$ while lowering global performance and raising sample complexity at order $\\lambda$.","To reach expected optimality gap $\\epsilon$, total sample complexity scales as $O(\\epsilon^{-2})$ with more clients reducing the per-client burden.","On MovieLens and IMDb, FedRLHF matches or surpasses centralized RLHF in reward and accuracy while reported client accuracy and Spearman correlation improve over rounds."],"supporting_citations":[{"why":"Supplies the federated reinforcement-learning convergence framework and per-client speed-up that FedRLHF extends.","marker":"[9]"},{"why":"Justifies the Polyak-Łojasiewicz condition used to turn the descent lemma into a contraction.","marker":"[16]"},{"why":"Provides the FedAvg aggregation rule used in Algorithm 1 and in the experiments.","marker":"[24]"},{"why":"Defines the human-preference reward signal that FedRLHF localizes to each client.","marker":"[3]"},{"why":"Supplies the PPO policy-gradient update used as the local RLHF optimizer in the IMDb experiments.","marker":"[34]"},{"why":"Supplies the MovieLens dataset validating convergence and personalization in recommendation.","marker":"[13]"},{"why":"Supplies the IMDb dataset used for the sentiment-controlled generation experiment.","marker":"[22]"}],"fun_headline_variants":["Federated RLHF: provable convergence, private feedback, per-client personalization","Privacy-preserving RLHF with convergence guarantee and personalization","FedRLHF: converge at 1/T, match centralized, keep feedback local","Federated human feedback: on-device personalization, global convergence proof","Sample-efficient federated RLHF: private feedback, provable O(1/T) convergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of Lemma 4.2 assumes the cross-client terms in the average of local updates vanish because gradient noise has zero mean, ignoring that clients' true gradients differ, so without a gradient-dissimilarity bound the stated O(1/T) guarantee does not follow from the proof.","fun_headline_variants_meta":{"raw":{"variants":["Federated RLHF: provable convergence, private feedback, per-client personalization","Privacy-preserving RLHF with convergence guarantee and personalization","FedRLHF: converge at 1/T, match centralized, keep feedback local","Federated human feedback: on-device personalization, global convergence proof","Sample-efficient federated RLHF: private feedback, provable O(1/T) convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4408,"prompt_tokens":963,"completion_tokens":3445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3342}},"tokens_in":579,"tokens_out":3445,"duration_ms":22131,"temperature":1.0,"reasoning_tokens":3342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:20:55.827960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 with two clients whose intrinsic rewards push the policy in opposite directions and measure the cross-client term $(1/K^2)\\sum_{k\\ne k'}\\mathbb{E}\\langle\\theta_t^k-\\theta_t,\\theta_t^{k'}-\\theta_t\\rangle$ across rounds. If this term is not negligible, the one-step descent lemma's bound is violated and the empirical optimality gap will plateau above the bias predicted by Theorem 4.1, refuting the universal convergence claim for heterogeneous clients.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the Polyak-Łojasiewicz condition used to turn the descent lemma into a contraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FedAvg aggregation rule used in Algorithm 1 and in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the federated reinforcement-learning convergence framework and per-client speed-up that FedRLHF extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the human-preference reward signal that FedRLHF localizes to each client."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MovieLens dataset validating convergence and personalization in recommendation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the IMDb dataset used for the sentiment-controlled generation experiment."}],"review_version":1}