{"id":"3f09ed64-f169-4dcb-9439-6d067dccdf3b","arxiv_id":"2505.01075","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FedOA regularizes personalized adapters toward the global model in feature space to improve OOD generalization in federated foundation models, but the proof is incomplete and the empirical gains are modest.","lead":"A new federated fine-tuning method, FedOA, adds a feature-distance penalty between each client's personalized adapter and the shared global adapter, aiming to improve out-of-distribution generalization for foundation models. The paper claims theoretical guarantees, but a key proof step is invalid for the distance used, and the reported accuracy gains are small and overlap with baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.2's proof asserts an inequality that fails for the L2 distance used; without it, the regularizer's OOD guarantee is unproven.","rationale":"The reader's verdict is REJECT with moderate confidence, and the strongest claim includes Theorem 3.2's assertion that the feature-distance regularizer provably improves OOD generalization. My stress-test pass locates a concrete, internal flaw in that theorem: the proof's step (b) uses an inequality that does not follow from the stated assumptions and is in fact false for the L2 distance used by the algorithm. This is more directly load-bearing than the head-optimality assumption, because it breaks the paper's central theoretical result even if the head-optimality premise were granted. The head-optimality issue independently undermines Theorem 3.1, but the proof error in Theorem 3.2 is the sharper objection. The empirical section is too weak to rescue the claim: the gains over FedIT are small, only one round of hyperparameters is reported, and no code or full training details are provided. Thus the rejection is appropriate; my concern does not change the reader's verdict, but it does sharpen the basis for it. I mark agreement as partial because the reader's weakest_assumption names head optimality, while my focus is the inequality error in the proof of Theorem 3.2, which the reader also mentions in the rationale but not as the primary weakest assumption.","tokens_in":22306,"tokens_out":6002,"duration_ms":62895,"concrete_test":"Formalize Appendix D and check step (b) with D=L2. Specifically, verify the inequality |E[g(Φe(X))]−E[g(Φg(X))]| ≤ M E[||Φe(X)−Φg(X)||] for all |g|≤M; the counterexample above (Φe(X)=X, Φg(X)=X+ε/2, g(z)=M·1_{⌊2z/ε⌋ even}) shows it fails. If the authors instead intend a Lipschitz condition on g, add it explicitly, restate Theorem 3.2, and re-derive Eq. (12) under that condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix D's proof of Theorem 3.2 is invalid at step (b). It claims |E_{z∼P(Φe)}[g(z)] − E_{z∼P(Φg)}[g(z)]| ≤ M·E_X[D(Φe(X),Φg(X))] from |g|≤M, for the distance D actually used (L2 in Section 4). Boundedness of g alone does not imply continuity, and for arbitrary bounded g the inequality is false. With D=L2, take Φe(X)=X, Φg(X)=X+ε/2 on X∼U[0,1], and g(z)=M·1_{⌊2z/ε⌋ even}; then |E[g(Φe(X))−g(Φg(X))]|=M while M·E[D]=Mε/2, so the asserted bound fails for ε<2. The subsequent claim that the regularizer in Eq. (3) 'provably improves OOD generalization' therefore rests on an unproved inequality. A Lipschitz assumption on ℓ∘w would repair the argument, but none is stated. Separately, Theorem 3.1 depends on the premise that the frozen pretrained head is simultaneously optimal for all e∈E_all (Section 3.1); no evidence supports this, and without it the global model's objective does not reduce to invariant-feature learning, so the OOD guarantee for the global model also lacks support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedOA, a federated adapter method for foundation models that adds a per-client personalized adapter and a feature-distance regularizer pulling personalized features toward the globally aggregated model. It claims three theoretical results: that a conventional aggregated global model in FedFM inherently retains OOD generalization ability, that the personalized model's OOD risk is bounded by the feature distance between personalized and global encoders, and that FedOA converges at rate O(1/T) under non-convex assumptions. Experiments on four NLP tasks derived from Flan under a leave-one-task-out protocol, together with ablations, a scalability study, and a ViT/OfficeHome extension, report that FedOA outperforms the personalized baselines on average.","tokens_in":22628,"tokens_out":10756,"duration_ms":105484,"significance":"If the theoretical claims were correct, FedOA would be a useful contribution to federated foundation-model personalization with OOD guarantees. The empirical evaluation is reasonably broad: four NLP tasks, a leave-one-task-out evaluation, ablations over lambda and the distance function, convergence curves, a 30-client scalability experiment, and an OfficeHome/ViT extension. However, the advertised guarantees rest on an invalid proof step in Theorem 3.2 and on an unvalidated optimal-head premise in Theorem 3.1, so the central theoretical contribution is not established. The paper also provides no code release or machine-checked proofs, which limits independent verification of the empirical claims.","major_comments":[{"comment":"Step (b) of the proof of Theorem 3.2 asserts |E[g(z)] - E[g(z')]| <= M * E_X[D(Phi_e(X), Phi_g(X))] from the boundedness condition |g| <= M for a generic distance D. With D chosen as L2 distance in Section 4, this is false without a Lipschitz assumption on g. For example, with X ~ U[0,1], Phi_e(X)=X, Phi_g(X)=X+eps/2, and g(z)=M * 1_{floor(2z/eps) even}, the left-hand side equals M while M * E[D] = M*eps/2, so the inequality fails for eps < 2. A Lipschitz condition on the composition loss o head would repair the step, but none is stated. Consequently, Eq. (12) and the claim that the regularizer in Eq. (3) provably improves OOD generalization are unsupported.","section":"Appendix D, Eq. (14)"},{"comment":"The proof assumes that a single frozen pre-trained head w satisfies w in argmin_w R_e(w, Phi_g) for all e in E_all and justifies this by the scale of pre-training. This is an unvalidated premise; the paper offers no evidence that one fixed head is simultaneously optimal for all downstream environments. In addition, Assumption 2.1 only equates conditional means E[Y^e | Phi(X^e)=z], which is insufficient to make the discrepancy d_F(P_e,P_e') vanish for general losses. For instance, under squared loss, equal conditional means with different conditional variances still produce a nonzero discrepancy. Theorem 3.1's conclusion therefore lacks support.","section":"Section 3.1 / Appendix D (Theorem 3.1 proof)"},{"comment":"Even if the proof step in Eq. (14) were repaired, the bound in Eq. (12) is stated for R_e(f_e) with the distance E_{X~P_e}[D(Phi_e(X), Phi_g(X))] evaluated on the client's own distribution. This is not an out-of-distribution bound for an unseen environment e'; the regularizer in Eq. (3) directly minimizes this same training-distribution distance, so the claimed OOD guarantee is partly a statement about the training objective itself. To support the OOD claim, the paper would need a bound involving E_{X~P_e'}[D(Phi_e(X), Phi_g(X))] for the target environment.","section":"Theorem 3.2 / Appendix D, Eq. (12)"},{"comment":"The convergence statement does not match the objective being optimized. The client update minimizes R_e(phi) + lambda * D(Phi(phi), Phi(phi_g)), but Theorem 4.4 bounds (1/T) * sum_t E||nabla R_e(phi_e^{t-1})||^2, which omits the lambda * nabla_phi D term; stationarity of the regularized objective would require the full gradient to vanish. Moreover, the step-size choices eta_l = O(1/(T K L sigma)) and eta_g = O(1/(T L)) do not satisfy the theorem's own conditions eta_l <= 1/(8 sqrt(3(1+3T)) T (1+2K) K lambda sigma L) and eta_g <= 1/(2 sqrt(6(1+3T)) T L), and the rates in Eq. (8) are dimensionally inconsistent with the substitutions that produce them. In addition, Lemma E.4 uses ||Phi(phi_1)-Phi(phi_2)|| <= L ||phi_1-phi_2||, which follows from Assumption 4.1 only if Phi is L-Lipschitz, not merely L-smooth. These issues leave the advertised O(1/T) convergence claim unsubstantiated.","section":"Section 4, Theorem 4.4 / Appendix E.2"}],"minor_comments":[{"comment":"The word 'turning' appears where 'tuning' is intended, for example in Section 3.1 ('during the turning') and in the discussion of the fixed head.","section":"Throughout"},{"comment":"The description of FedSDR says 'we adapt pFedMe to the FedFM setting' but the surrounding text refers to FedSDR; this appears to be a copy-paste error.","section":"Appendix C.2"},{"comment":"The metric is written 'ROGUE-1' but should be 'ROUGE-1'.","section":"Page 8, Section 5.1"},{"comment":"The statement of Theorem 3.2 in the main text uses E[D(Phi_e(X_{e'}), Phi_g(X_{e'}))], while the appendix proof and Eq. (12) use E_{X~P_e}; the notation should be aligned.","section":"Section 3.1 / Appendix D"},{"comment":"The variable z is used as the label in the empirical risk R_e(Phi) but is not defined before the equation, making the feature-space loss ambiguous.","section":"Eq. (2)"},{"comment":"The symbol D is used both for the distance function and for a dataset in Table 1; consider renaming one of them to avoid confusion.","section":"Algorithm 1 and Eq. (3)"}],"recommendation":"reject","confidential_remarks":"The manuscript is a preprint under review. The issues are technical rather than behavioral: the core theoretical claims are not established as stated, while the empirical comparison is reasonable. If the authors reposition the paper as an empirical framework without the advertised OOD and convergence guarantees, a future submission could be viable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on 2505.01075. The paper has a real empirical core: FedOA trains a personalized adapter alongside the global one and adds feature-distance regularization, tested on leave-one-task-out NLP benchmarks with LLaMA-7B. The experiments are reasonable for a paper of this scope—several baselines, ablations on λ and distance function, scalability to 30 clients, and an extension to ViT. If you want to know whether this style of regularization helps in federated foundation models, the numbers are mildly encouraging: FedOA edges out FedIT on average (59.05 vs 58.36) and beats the personalized baselines, though the gaps are small and it loses on Entailment.\n\nThe problem is the theory, and the theory is the headline. Theorem 3.2's proof in Appendix D steps from |g|≤M to |E[g(z)] - E[g(z')]| ≤ M·E[D(Φe(X), Φg(X))]. That is not true for the L2 distance they use in the experiments; a bounded function can oscillate arbitrarily fast, and the gap between expectations can be M while the expected distance is arbitrarily small. You need a Lipschitz condition on ℓ∘w (or on g) to get that kind of bound. None is stated. So the central claim—that the feature-distance regularizer provably improves OOD generalization of the personalized model—is unproven as it stands.\n\nTheorem 3.1 is shakier in a different way. It assumes the frozen pretrained head is simultaneously optimal for all environments, \"since foundation models are pre-trained with massive data.\" That is not something you can just assert; fine-tuning heads per task is common practice, and the paper gives no evidence. Without that premise, the global model's objective does not reduce to invariant feature learning and the OOD guarantee for the global model doesn't follow.\n\nThe convergence analysis (Theorem 4.4) looks more solid—standard non-convex FL machinery with an extra drift term from the regularizer. I didn't find an obvious error there, though I didn't check every line. But convergence of the optimizer is not a substitute for the generalization claim.\n\nThe upshot: the paper proposes a plausible and reasonably evaluated approach, but the advertised theoretical guarantee doesn't hold in its present form. This is fixable in principle—add Lipschitz assumptions, prove the bound properly, and either justify the head-optimality premise or weaken the claim. As it stands, I would not accept it; I would encourage the authors to repair the theory and resubmit. For an editor: yes, send it out. It deserves a careful referee, and the issues are specific and addressable.","headline":"Plausible empirical recipe for personalized federated adapters, but the advertised OOD generalization proof breaks on an invalid inequality and an unverified head-optimality assumption.","tokens_in":23141,"tokens_out":4082,"would_cite":false,"duration_ms":37530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the aggregated global model in federated foundation-model fine-tuning already carries out-of-distribution robustness, and that a feature-distance regularizer that pulls each client's personalized encoder toward the…","keywords":["federated learning","foundation models","out-of-distribution generalization","parameter-efficient fine-tuning","adapters","personalization","invariant feature learning","convergence analysis"],"falsifier":"Take a foundation model whose pre-training corpus excludes the federated clients' task family, train FedOA with a learnable head instead of a frozen one, and measure leave-one-task-out accuracy: if the feature-distance regularizer no longer improves OOD accuracy or the aggregated global model degrades, the fixed-head optimality premise is violated.","tokens_in":22111,"feed_emoji":"🤖","tokens_out":6113,"duration_ms":57783,"temperature":0.7,"pith_summary":"The paper asks how out-of-distribution (OOD) generalization can survive in federated fine-tuning of foundation models, where models are huge, clients see different tasks or domains, and only a small adapter is trained and communicated. It argues that the conventional aggregated global model already has OOD robustness: because the frozen pre-trained head is assumed optimal for every task, the global objective reduces to learning invariant features. It then shows that a personalized model's OOD error is controlled by the expected distance between that client's features and the global model's features. On that ground it proposes FedOA, which keeps a personal adapter per client and regularizes it with $D(\\Phi_e(X), \\Phi_g(X))$, and proves convergence to a stationary point at rate $O(1/T)$ under non-convex assumptions. The empirical portion reports that this regularizer outperforms parameter-regularization and conventional-FL OOD baselines on leave-one-task-out NLP benchmarks.\n","feed_headline":"One regularizer restores OOD robustness in federated foundation models","feed_subtitle":"A feature-distance penalty ties each client's personalized adapter to the aggregated model, with proved convergence.","key_machinery":"The central object is the feature-distance regularizer $D(\\Phi_e(X), \\Phi_g(X))$, applied in representation space rather than parameter space. It appears in the optimization objective as $\\min_{\\Phi_e} R_e(\\Phi_e) + \\lambda D(\\Phi_e(X_e), \\Phi_g^*(X_e))$ with the global encoder $\\Phi_g^*$ solving the aggregated objective. The regularizer is what converts the Theorem 3.2 generalization bound into a training signal; it also sidesteps the structural heterogeneity of parameter-efficient modules such as low-rank adapters, because it shapes adapter parameters indirectly through their output features. In the convergence analysis it is instantiated as the squared L2 distance, and the proof tracks how this term bounds client drift between the personalized and global adapters.\n","core_discovery":"The central claim is Theorem 3.2: for a personalized hypothesis $(w, \\Phi_e)$ in a federated foundation model with fixed head $w$, the excess OOD risk is bounded by the invariant-feature distance $M \\cdot E_{X \\sim P_e}[D(\\Phi_e(X), \\Phi_g(X))]$ plus the global model's own bound. Since the global model's bound is tight when the frozen head is optimal across environments (Theorem 3.1), minimizing the feature distance is a principled way to bring personalized models' OOD guarantees close to the global model's. The proposed FedOA implements this by training two adapters per client: a personalized adapter updated with the regularizer and a global adapter that is aggregated by the server; only the global adapter is communicated. The paper also proves that under smoothness, bounded-gradient, and bounded-diversity assumptions, the personalized objectives reach a stationary point at rate $O(1/T)$.\n","pith_inferences":["The paper's argument suggests that if the head is no longer frozen, the same feature-distance logic would have to be extended to head parameters; the experiments do not test head tuning, so that extension remains an open prediction rather than a demonstrated result.","A sharper distance function than L2, for example a distributional discrepancy between feature batches, might capture the Theorem 3.2 term more directly; the paper only compares L2, cosine, and Pearson, so this is a testable direction.","The worst-case guarantee in the paper is about reaching a stationary point for each client's personalized objective, not about minimizing the worst environment risk; in strongly heterogeneous federations the regularizer may still leave large inter-client gaps.","One could test the fixed-head premise directly by measuring whether the pre-trained head is near-optimal for held-out environments; a positive measurement would isolate when the OOD gain of the regularizer should appear."],"forward_implications":["Conventional federated averaging over adapters already provides OOD robustness, so even without personalization the global model can be deployed on unseen clients or tasks.","Personalized adapters can be made OOD-robust without modifying the aggregated global model, so the method preserves communication efficiency.","The same regularization also reduces feature drift between clients and the global model, which the experiments tie to flatter loss surfaces and more stable generalization.","The convergence guarantee holds for non-convex objectives, so the method's extra regularizer does not change the asymptotic communication-computation trade-off."],"supporting_citations":[{"why":"Supplies the premise that a frozen pre-trained head is fixed and optimal across tasks, on which the reduction to invariant feature learning rests.","marker":"[18]"},{"why":"Provides the discrepancy-based generalization bound for the aggregated global model that Theorem 3.1 extends.","marker":"[22]"},{"why":"Introduces the invariant feature learning assumption used to show the discrepancy term vanishes.","marker":"[2]"},{"why":"Serves as a conventional-FL OOD baseline whose poor FedFM performance motivates the adapter-based regularizer.","marker":"[46]"},{"why":"Serves as a parameter-regularization personalization baseline that the feature-space regularizer is designed to beat.","marker":"[51]"},{"why":"Supplies the instruction-tuning benchmark from which the four leave-one-task-out federated datasets are built.","marker":"[50]"},{"why":"Provides the federated instruction-tuning global model baseline compared in the experiments.","marker":"[59]"},{"why":"Provides the federated averaging aggregation used in the method.","marker":"[34]"}],"fun_headline_variants":["Feature-distance penalty proves OOD robustness for each FL client","Align features, get global OOD bounds in federated foundation models","Personalized adapters with distance regulation earn the global model's OOD edge","One regularizer aligns client features to global OOD guarantee","Feature-distance regularizer guarantees OOD robustness for federated clients"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the claim that the frozen pre-trained head is already optimal for every possible environment, so the global model's objective collapses to invariant feature learning; if some environment needs a different head, the global model's OOD guarantee and the regularizer's rationale weaken.","fun_headline_variants_meta":{"raw":{"variants":["Feature-distance penalty proves OOD robustness for each FL client","Align features, get global OOD bounds in federated foundation models","Personalized adapters with distance regulation earn the global model's OOD edge","One regularizer aligns client features to global OOD guarantee","Feature-distance regularizer guarantees OOD robustness for federated clients"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2883,"prompt_tokens":942,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":1853}},"tokens_in":558,"tokens_out":1941,"duration_ms":14634,"temperature":1.0,"reasoning_tokens":1853,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:27:21.279829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a foundation model whose pre-training corpus excludes the federated clients' task family, train FedOA with a learnable head instead of a frozen one, and measure leave-one-task-out accuracy: if the feature-distance regularizer no longer improves OOD accuracy or the aggregated global model degrades, the fixed-head optimality premise is violated.","supporting_citations":[{"cited_title":"Robust learning from untrusted sources","cited_arxiv_id":null,"evidence_quote":"Provides the discrepancy-based generalization bound for the aggregated global model that Theorem 3.1 extends."},{"cited_title":"Learning personalized causally invariant representations for heterogeneous federated clients","cited_arxiv_id":null,"evidence_quote":"Serves as a conventional-FL OOD baseline whose poor FedFM performance motivates the adapter-based regularizer."},{"cited_title":"Perada: Parameter-efficient federated learning personalization with generalization guarantees","cited_arxiv_id":null,"evidence_quote":"Serves as a parameter-regularization personalization baseline that the feature-space regularizer is designed to beat."}],"review_version":1}