{"id":"00e85aa1-b5c1-4ee3-9cca-3267718b19f2","arxiv_id":"2511.16992","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Regularizing each client's local MGDA step in federated multi-objective LLM alignment provably controls client disagreement drift and yields Pareto-stationary convergence in a simplified actor-critic setting.","lead":"FIRM is a federated-learning method that lets multiple clients fine-tune a language model on several goals at once while sending only one set of updated parameters to the server. It adds a regularization term to each client's local multi-goal optimizer to stop clients from drifting apart, and reports both a convergence proof (for a simplified version) and experiments showing smoother training and adjustable helpfulness/safety trade-offs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's advertised disagreement-drift term O(√M^3/(β√B)·αK) does not follow from the proof; the proof yields separate additive terms O(√M^3/(β√B)) and O(√M^3·αK/β).","rationale":"The reader's weakest_assumption focused on the gap between TFIRM (the analyzed simplification) and the deployed FIRM/PPO algorithm, which is a legitimate scope concern. My check goes one level deeper: even within the TFIRM setting, the proof's own equations do not yield the displayed disagreement-drift term. The bound in Eq. (24) is a sum of independent error terms, not a product, so the O(√M^3/(β√B)·αK) in Theorem 1 and the corresponding intuition in Remark 1 ('controlled by the product of regularization and batch size') are not supported. This is a concrete algebraic inconsistency rather than a mere scope gap. The reader did partially anticipate this by flagging Lemma 8's regularization-constant mismatch, which is one ingredient of the drift bound, but the main issue is the propagation into Theorem 1. The defect is fixable: a corrected additive bound still shows that β and B control drift and that a Pareto-stationary point is approached up to the usual bias terms, so the paper's core idea survives. Therefore the CONDITIONAL verdict remains appropriate; no change to the overall recommendation is needed, though the authors should correct the theorem's statement and proof before publication.","tokens_in":29490,"tokens_out":13792,"duration_ms":123901,"concrete_test":"Re-derive the chain from Eq. (23) through Eq. (35) symbolically, tracking the factors 1/√B and αK separately. Substitute Lemma 9's bound on max_j E||g^{c}_j − g^{c′}_j|| into Eq. (23) and simplify using Σ_i w_i = 1. If the result contains a product term ∝ αK/√B, Theorem 1's drift formula is supported; if, as the written proof suggests, the 1/√B and αK terms remain separate, replace the advertised term with O(√M^3/β(1/√B+αK)) and revise Remark 1 and the abstract accordingly. Also verify the strong-convexity parameter in Lemma 8 against the β/2 in Eq. (1): the correct modulus is β, not 2β, so the stated 4RM/β constant needs rechecking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is internal to Theorem 1: the headline drift formula appears unsupported by the proof. Eq. (4) states the disagreement-drift contribution as O(√M^3/(β√B)·αK). However, in Appendix F.1.2, the bound on E[T_disagr|F_t] is (Eq. 24): ≤ (8R^3M^{3/2}/β) Σ_i w_i · max{4√ζapprox + 4√εcritic + 4√(V/B) + 4L_J R αK}, with V = (r_max+R_w)^2(1−ρ+4κρ)/(1−ρ). Because Σ_i w_i = 1, this simplifies to O(√M^3/β · (√ζ + √ε + 1/√B + αK)). The factors 1/√B and αK appear additively, not multiplicatively. No step in the proof multiplies the batch-size variance term by the local-update drift term. The final bound (Eq. 35) repeats the same unsupported product. Thus the paper's central quantitative claim about disagreement drift is not established by the given derivation. Separately, Lemma 8's f_G uses β||λ||² while the local objective (Eq. 1) uses (β/2)||λ||², so the strong-convexity constant used for the 4RM/β factor is mismatched; the constant needs re-derivation. These are fixable defects, but the theorem as stated is not internally supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FIRM, a federated algorithm for multi-objective RLHF alignment. Each client computes PPO gradients for M reward objectives, solves a locally regularized MGDA subproblem to obtain a single update direction, and transmits only the updated LoRA adapter to the server, reducing communication cost from O(Md) to O(d). The theoretical section analyzes an idealized variant, TFIRM, which replaces PPO with a linear-approximation actor-critic and assumes homogeneous clients. Theorem 1 claims convergence to an ε-Pareto stationary point with a bound containing a novel disagreement-drift term O(√M^3/(β√B)·αK). Experiments on TinyLlama with helpfulness/harmlessness rewards compare FIRM against a server-centric baseline, ablate the regularization, test robustness to non-IID data and heterogeneous reward models, and demonstrate preference-guided alignment.","tokens_in":29872,"tokens_out":13474,"duration_ms":116473,"significance":"The central idea—resolving multi-objective conflicts at the client and using a simple ℓ2 regularizer to control the resulting disagreement drift—is novel, well-motivated, and practically relevant. The communication-cost reduction from O(Md) to O(d) is significant for large-model federated alignment. The empirical study is thorough: it includes reward trajectories, MGDA-weight trajectories, ablations on β, non-IID data, heterogeneous reward models, scalability to 16 clients, and preference-control experiments. The overall proof structure is a credible extension of existing actor-critic/FedAvg analyses, and the explicit decomposition of client drift into a classical term and a disagreement-drift term is a useful conceptual contribution. However, the theorem's headline disagreement-drift rate is not supported by the proof as written, and the theoretical guarantee is for an idealized variant that differs from the deployed algorithm. With correction, the qualitative message that β and batch size control drift appears salvageable, but the manuscript currently overstates its theoretical contribution.","major_comments":[{"comment":"The advertised disagreement-drift term O(√M^3/(β√B)·αK) is not derived. Eq. (24) bounds E[T_disagr|F_t] by a constant times [4√ζ_approx + 4√ε_critic + 4√(V/B) + 4L_J R αK], with V=(r_max+R_w)^2(1−ρ+4κρ)/(1−ρ). After multiplying by 8R^3 M^{3/2}/β, the resulting terms are additive: O(M^{3/2}/(β√B)) and O(M^{3/2} αK/β). No step in the proof multiplies the batch-size term by αK. Eq. (35) repeats the unsupported product. This affects the central quantitative claim emphasized in the abstract and Remark 1; the theorem statement and the surrounding discussion must be corrected to the additive form or the product must be proved.","section":"Theorem 1 / Eq. (4), Appendix F.1.2 (Eq. (24))"},{"comment":"Theorem 1 is proved for Theoretical-FIRM only: linear function approximation, homogeneous clients (P_c=P, r_c=r), and a simplified actor-critic (Algorithm 2). The deployed method (Algorithm 1) uses PPO, LoRA, and in the experiments heterogeneous reward models and non-IID data. The abstract's claim that 'we prove that our algorithm converges to Pareto-stationary points' is therefore overbroad. The authors should clearly separate the idealized TFIRM guarantee from the practical FIRM algorithm and either extend the analysis or temper the claim. The structural discrepancy between Algorithm 1 (K local updates per communication round) and Algorithm 2 (one update per round, aggregation every K rounds) also needs clarification.","section":"Section 4.1 vs. Algorithm 1 / Algorithm 2; Section 5"}],"minor_comments":[{"comment":"The left-hand side of Eq. (31) drops the λ_t factor: it should read E[||∇θJ(θ̄_t)λ_t||²], matching Eq. (30) and the theorem statement.","section":"Appendix F.3.1, Eq. (31)"},{"comment":"The proof defines f_G(λ)=λ^T G λ + β||λ||², whereas the algorithm's subproblem Eq. (1) uses (β/2)||λ||². The 4RM/β constant is in fact recoverable because the actual objective is β-strongly convex (Hessian 2G+βI), but the notation should be aligned to avoid the apparent mismatch.","section":"Appendix G.1, Lemma 8, Eq. (37)"},{"comment":"The proof uses a global λ_t in the descent and in the T1 decomposition, but Algorithm 2 only maintains per-client λ_t^c. The authors should explicitly define λ_t (e.g., as the average of client weights) and verify that the descent and telescoping arguments remain valid for that definition.","section":"Proof of Theorem 1, Eqs. (13)–(14)"},{"comment":"The bound appears to be r_max/(1−γ)(2+Σ 2η_t) rather than (1+Σ 2η_t); the constant difference does not affect the O(log T) rate but should be corrected.","section":"Appendix F.3.2, Lemma 10"},{"comment":"The experimental curves do not report error bars or multiple seeds. Given the known variance of PPO and MGDA weight trajectories, adding confidence bands or stating the number of seeds would strengthen the empirical claims.","section":"Section 5, Figures 1–6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising algorithm and a careful empirical study, but the main theorem's advertised disagreement-drift rate is not supported by the proof. This is a load-bearing error that affects the abstract, Remark 1, and the theorem statement. The proof can likely be repaired by changing the product form to an additive bound, but the presentation must be updated accordingly. The gap between TFIRM and the deployed FIRM is also significant and should be addressed honestly. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely new idea — control client disagreement in federated multi-objective RL by regularizing the local MGDA subproblem, so clients only need to send one parameter vector — and it backs it with an attempted finite-time Pareto-stationary bound for a simplified actor-critic surrogate. But the headline drift term in the bound does not follow from the proof. The appendix derives additive terms (1/√B and αK), while the theorem advertises their product. That is a load-bearing internal inconsistency, not a cosmetic typo. The paper also proves the theorem for TFIRM, not the PPO/LoRA FIRM actually run, and Lemma 8 has a constant mismatch: it states the subproblem with β||λ||² while Algorithm 1/Eq. (1) uses (β/2)||λ||², so the strong-convexity constant needs re-derivation. All of these are fixable, but the theorem as stated is not currently supported.\n\nWhat is good: the identification of multi-objective disagreement drift is real and clearly explained. The regularization trick is simple and the empirical plots do show smoother λ trajectories and better reward trade-offs than the server-centric baseline. The preference-vector extension is a nice touch, and the related work is thorough.\n\nWhat is weak: besides the theorem issue, the experiments lack error bars, seeds, and code. The ablation with different reward models is suggestive but appears to be single-run. The comparison to FedCMOO-A is fair in spirit (no compression), though the baseline is their own adaptation.\n\nBottom line: the core idea deserves referee time, and the paper has potential after a substantive revision that re-derives the drift bound, reconciles the constants, and either proves the theorem for the actual algorithm or clearly scopes it as a surrogate. I would send it to review, but I would not currently cite the convergence guarantee.","headline":"New idea, clean problem, but the headline convergence bound is internally inconsistent — the advertised product term does not follow from the proof's additive terms.","tokens_in":30379,"tokens_out":3366,"would_cite":false,"duration_ms":32323,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple ℓ2 regularizer added to each client's local multi-objective optimization provably controls the instability that plagues federated LLM alignment, letting clients send a single parameter vector per round while still converging to a P","keywords":["federated multi-objective optimization","LLM alignment","MGDA","client drift","Pareto stationarity","regularization","communication efficiency","actor-critic convergence"],"falsifier":"Compare the β=0 baseline with the regularized version on the same task: if the unregularized client λ vectors are already smooth and tightly clustered (no larger pairwise distance than the regularized run), the claimed disagreement drift and the necessity of the ℓ2 term would not hold. A sharper numeric check: with two clients on the same model and known gradient noise, solve the regularized MGDA for many batches and test whether ∥λ^c − λ^{c'}∥₂ ≤ (4RM/β) max_j ∥g_j^c − g_j^{c'}∥₂ always holds; any violation would falsify Lemma 8, the key bound.","tokens_in":29382,"feed_emoji":"⚖️","tokens_out":6839,"duration_ms":61231,"temperature":0.7,"pith_summary":"This paper introduces FIRM, a federated way to align LLMs with multiple conflicting objectives (like helpfulness and harmlessness) without centralizing data or shipping multiple gradients to a server. The core claim is that a lightweight ℓ2 penalty in each client's local multi-objective (MGDA) subproblem makes the locally chosen objective weights stable under sampling noise, eliminating what the authors call 'multi-objective disagreement drift.' If true, federated multi-objective alignment becomes both communication-efficient (one parameter vector per client per round) and provably convergent, with what the authors call the first finite-time guarantee for a federated multi-objective actor-critic algorithm. Experiments on helpfulness/harmlessness alignment show smoother training, reduced drift, and improved reward trade-offs versus a server-centric baseline, and the same regularizer doubles as a preference dial to trace Pareto trade-offs.","feed_headline":"One ℓ2 penalty provably fixes federated alignment drift","feed_subtitle":"A local regularizer lets clients share one update each while still hitting Pareto-optimal trade-offs.","key_machinery":"The central object is the regularized MGDA subproblem solved locally by each client: λ* = argmin_{λ∈ΔM} λᵀ(G + (β/2)I)λ, where G is the Gram matrix of the M objective gradients and β > 0 is the regularization coefficient. The ℓ2 term makes the subproblem strongly convex, which the paper uses to bound the distance between any two clients' λ solutions in terms of the maximum per-objective gradient difference (Lemma 1/8). That bound directly tames the disagreement-drift term in Theorem 1, converting a previously unstudied source of federated instability into a term controlled by β and batch size B.","core_discovery":"The paper's central discovery is that the instability in federated multi-objective RL comes not mainly from classical local-update drift but from the MGDA solver's non-smooth dependence on stochastic gradients: small sampling variations across clients produce very different λ weights, hence diverging descent directions. Adding an ℓ2 penalty with coefficient β to the local MGDA objective makes it β-strongly convex, and the paper proves (for its theoretical variant TFIRM, using linear value-function approximation and homogeneous clients) that pairwise differences between clients' optimal λ vectors are bounded by (4RM/β) times the worst per-objective gradient difference. Plugging this into a de","pith_inferences":["The theorem's explicit M^{3/2} dependence suggests a concrete engineering rule the authors leave implicit: to hold disagreement drift below a fixed threshold as the number of objectives grows, scale the regularization β or batch size B roughly as M^{3/2}. This is testable by running the same experimental setup with M=3 or M=4 objectives.","The preference-encoding trick is a remarkably cheap mechanism for Pareto-front exploration — it changes nothing about the federated communication — so it may transfer directly to centralized multi-objective RLHF or any MGDA-based training loop as a way to interpolate between objectives without training separate models.","Because the proof assumes homogeneous clients but the experiments show stability under non-IID data and heterogeneous reward models, a plausible (unproven) generalization is that disagreement drift is dominated by stochastic sampling noise rather than statistical heterogeneity — and should remain controllable by the same regularizer in fully heterogeneous federations.","The looseness of the constant in Lemma 8 (the proof's f_G uses β||λ||² while the algorithm uses (β/2)||λ||²) may mean the 4RM/β bound is off by a constant factor; this does not change the convergence rate but would affect quantitative predictions about how much regularization is needed."],"forward_implications":["Federated multi-objective alignment can be done with O(Cd) communication per round — one parameter vector per client — instead of O(CMd) gradients or the extra round trips required by compression-based server-centric designs.","The convergence guarantee decomposes the error into interpretable parts: optimization error O(log T/T), variance O(1/(CB)) showing linear speedup in clients×batch, bias from approximation and critic error, classical local-update drift O(α²K²), and the new disagreement drift O(√(M³)/(β√B) αK) that is explicitly controlled by tuning β and B.","The regularizer doubles as a preference-encoding mechanism: replacing the uniform β/2 I with Diag(p⁻¹) biases the local subproblem toward desired objectives, allowing practitioners to sweep a smooth Pareto trade-off frontier without changing the communication scheme.","The paper's diagnosis implies that unregularized local MGDA is fundamentally unstable in federated settings; its β=0 baseline exhibits erratic λ trajectories and degraded rewards, consistent with the claim that disagreement drift, not just data heterogeneity, is the core obstacle.","The asymptotic rate matches centralized multi-objective actor-critic, so the federated protocol does not pay a penalty in convergence order while gaining privacy and communication efficiency."],"fun_headline_variants":["A single ℓ2 penalty fixes federated alignment drift","One update per client, provable Pareto convergence","In-client regularization ends multi-gradient bottleneck","Federated alignment without gradient flooding","Local regularizer, global Pareto, one transmission"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the convergence proof, carried out for a simplified theoretical variant with linear value-function approximation and identical clients, genuinely describes the dynamics of the deployed algorithm (policy-gradient updates on low-rank adapters with non-identical client data and reward models); the proof's key bound also uses β||λ||² while the algorithm uses (β/2)||λ||², so the constant 4RM/β is not rigorously earned as written.","fun_headline_variants_meta":{"raw":{"variants":["A single ℓ2 penalty fixes federated alignment drift","One update per client, provable Pareto convergence","In-client regularization ends multi-gradient bottleneck","Federated alignment without gradient flooding","Local regularizer, global Pareto, one transmission"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1216,"prompt_tokens":779,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":523,"tokens_out":437,"duration_ms":5131,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:00:37.348115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the β=0 baseline with the regularized version on the same task: if the unregularized client λ vectors are already smooth and tightly clustered (no larger pairwise distance than the regularized run), the claimed disagreement drift and the necessity of the ℓ2 term would not hold. A sharper numeric check: with two clients on the same model and known gradient noise, solve the regularized MGDA for many batches and test whether ∥λ^c − λ^{c'}∥₂ ≤ (4RM/β) max_j ∥g_j^c − g_j^{c'}∥₂ always holds; any violation would falsify Lemma 8, the key bound.","supporting_citations":[],"review_version":1}