{"id":"304bb549-40d4-41a3-a5e3-102b20cfbc69","arxiv_id":"2608.10144","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A federated fine-tuning method that aggregates clients' sketched LoRA updates linearly, removing the bilinear mismatch and handling heterogeneous ranks without full-model computation.","lead":"This paper introduces a new way to fine-tune large AI models together across many devices, when different devices can only handle different-sized low-rank updates. It uses matrix sketching so the server can combine these updates without a common mathematical error, which should make collaborative fine-tuning faster, cheaper, and easier to protect with privacy techniques.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The advertised O(1/T) convergence rate for SeFoRA-Ho is not what Theorem 1 and Corollary 1 actually prove; the bound is O(1/sqrt(T) + sqrt(epsilon)).","rationale":"The reader's weakest assumption focused on the spectral-tail condition (Assumption 4), which is a genuine and important limitation: the sketching error floor controls the neighborhood size, and the paper gives no bound for the rank-heterogeneous algorithm where it matters most. I agree with that concern. However, the sharper issue that I found is internal to the rank-homogeneous theorem: even when Assumption 4 holds, the stated rate O(1/T) is contradicted by Corollary 1's own bound. The proof of Theorem 1 appears algebraically coherent, and the methodological contribution is plausible; this is not grounds for rejection. But the central convergence claim should be restated as O(1/sqrt(T) + sqrt(epsilon)) with an explicit dependence on the tail energy, or the authors need to add an analysis mechanism that genuinely yields O(1/T). This correction is compatible with a conditional acceptance: the algorithm can still be sound and useful, and the missing rank-heterogeneous analysis and the omitted HetLoRA baseline should also be addressed. My read therefore keeps the conditional verdict while adding a concrete, checkable correction to the theory.","tokens_in":35982,"tokens_out":10576,"duration_ms":112450,"concrete_test":"Re-derive the bound in Corollary 1 by substituting eta = gamma_T/E into Theorem 1 and keeping only leading terms. Set epsilon=0 (Assumption 4 with tau=0), so gamma_T = sqrt(Delta_0/(T*(chi*C3+2*C4))). The dominant term is 4*sqrt(Delta_0*(chi*C3+2*C4)/T) + O(1/T), which is Theta(1/sqrt(T)). Separately, minimize the two dominant learning-rate terms A/(eta*T) + B*eta over constant eta; the minimum is 2*sqrt(A*B/T), again Theta(1/sqrt(T)). If the authors maintain that the rate is O(1/T), they must exhibit a variance-reduced estimator or a non-constant step-size schedule that avoids this calculation; no such mechanism appears in Section B.2.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and Contribution 2 claim that SeFoRA-Ho converges to a neighborhood of a first-order stationary point at rate O(1/T), and the reader's strongest claim repeats this as O(1/T+c). The proof does not deliver this. In Corollary 1, the learning rate is chosen as eta = gamma_T/E with gamma_T = sqrt((Delta_0/T + chi*epsilon + 2*mu*epsilon^2)/(chi*C3 + 2*C4)). Substituting into the Theorem 1 bound gives average squared gradient norm bounded by 4*gamma_T*(chi*C3+2*C4) + O(gamma_T^2 + gamma_T^3). Even in the best case of zero sketching error (epsilon=0), gamma_T = Theta(1/sqrt(T)), so the bound is Theta(1/sqrt(T)), not Theta(1/T). With epsilon>0, the bound tends to a positive constant of order sqrt(epsilon) as T grows, so the error floor is sqrt(epsilon), not 'c' in an additive O(1/T+c) sense. This is not a stylistic mismatch: the rate is a headline theoretical contribution, and the stated theorem, as proven, supports only convergence to an O(sqrt(epsilon))-neighborhood at rate O(1/sqrt(T)). The Assumption 4 spectral-tail floor is indeed decisive, but even granting Assumption 4, the advertised convergence rate does not follow from the proof.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SeFoRA, a federated low-rank adaptation method in which clients transmit linear sketches of their local LoRA updates and the federator aggregates these sketches directly, thereby avoiding the bilinear mismatch and full-space aggregation while supporting heterogeneous client ranks. A rank-homogeneous variant, SeFoRA-Ho, is introduced and analyzed. The authors prove convergence of SeFoRA-Ho to a neighborhood of a first-order stationary point under smoothness, bounded-moment, and bounded-factor assumptions, and they bound the error floor through an assumed bound on the spectral tail energy of the aggregated client updates. Experiments on RoBERTa-Large over four GLUE tasks compare SeFoRA-Ho with FedIT and FFA-LoRA, and SeFoRA with FlexLoRA, FLoRA, and FSLoRA, reporting competitive accuracy and reduced server-side computation.","tokens_in":36298,"tokens_out":4505,"duration_ms":49406,"significance":"If the claims are correct, this is a meaningful step for federated LoRA: the sketching-based aggregation is elegant, enables linear aggregation, and avoids the bilinear mismatch without communicating full-model updates. The proof is detailed and builds on an external, well-established sketching bound rather than on a circular argument. The method is also practically plausible, with experiments covering rank-homogeneous and rank-heterogeneous settings. The main theoretical claim, however, is overstated: the stated convergence rate O(1/T) is not what the proof delivers, and this is a load-bearing issue that affects the abstract, the contributions list, and Table 1. The experimental comparison also omits HetLoRA, one of the most directly relevant baselines, so the empirical claim of outperforming the state of the art needs qualification.","major_comments":[{"comment":"The advertised convergence rate O(1/T) is not supported by the proof. In Corollary 1, the learning rate is chosen as eta = gamma_T/E with gamma_T = sqrt((Delta_0/T + chi*epsilon + 2*mu*epsilon^2)/(chi*C3 + 2*C4)). Substituting this into the bound gives an average squared gradient norm bounded by 4*gamma_T*(chi*C3 + 2*C4) plus O(gamma_T^2 + gamma_T^3). Even when the sketching error is exactly zero (epsilon = 0), gamma_T = Theta(1/sqrt(T)), so the bound is Theta(1/sqrt(T)), not Theta(1/T). For epsilon > 0, the bound tends to a positive constant of order sqrt(epsilon) as T grows, so the floor is O(sqrt(epsilon)), not an additive c in an O(1/T + c) statement. This is not a stylistic mismatch: the rate is a headline theoretical contribution. The abstract, Contribution 2, and Table 1 should be revised to state convergence to an O(sqrt(epsilon))-neighborhood at rate O(1/sqrt(T)), or the proof must be modified to yield the claimed rate.","section":"Section 4, Corollary 1; Abstract; Contribution 2 (Section 3); Table 1"},{"comment":"The experimental section omits HetLoRA, which the paper itself identifies as one of the most relevant rank-heterogeneous federated LoRA baselines and which appears in the comparison Table 1. Without this baseline, the claim that SeFoRA 'outperforms the state-of-the-art' is not fully substantiated for the rank-heterogeneous setting. The reported results also show SeFoRA-Ho performing comparably to FedIT, not clearly better; the text should be adjusted to reflect this accurately.","section":"Section 5, Rank-heterogeneous results (Fig. 1, Table 3)"},{"comment":"The convergence guarantee in Corollary 1 is conditioned on the aggregated client update matrix having small spectral tail energy beyond rank r. This assumption is central: if the tail energy is not small, the sketching floor dominates the optimization error. The paper does not empirically measure this tail energy or verify Assumption 4 in the experiments, despite using k = r + 2 in several settings. The authors should either provide measurements of the spectral tail energy of the aggregated updates or explicitly state that the practical regime of validity of the convergence guarantee is unverified.","section":"Assumption 4, Section 4"}],"minor_comments":[{"comment":"The sentence 'Combining LoRA with FL raises several challenges faces challenges that do not exist' contains a duplicated phrase and should be rewritten.","section":"Section 6, paragraph 'Combining PEFT with FL'"},{"comment":"Reference [5] contains the typo 'Antrhopic' and reference [48] has a spacing issue in 'V olkan'; these should be corrected.","section":"References"},{"comment":"The caption of Figure 1 reports accuracy curves but does not state the random seed count or error bars; adding standard-deviation shading or a note that the same three-seed protocol as Table 2 is used would improve clarity.","section":"Figure 1 and Section 5"},{"comment":"The pseudocode says the federator 'send the shared seed' while the text says 'communicates them to the clients via a shared random seed'; the wording should be unified for consistency.","section":"Algorithm 2, Step 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid algorithmic idea and a detailed proof, but the overstated convergence rate is a central claim that must be corrected before publication. The missing HetLoRA baseline should also be addressed. I do not see grounds for rejection, since the proof itself appears methodologically sound and the rate claim can be fixed by restating the result as O(1/sqrt(T) + sqrt(epsilon))."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is genuinely good. SeFoRA has each client send Y_i = ΔW_i Ω and Z_i = Ψ ΔW_i (linear sketches of its full LoRA update), and because sketching is linear the server can sum sketches to get a sketch of the aggregate update, then unsketch to a rank-r adapter. That sidesteps both the bilinear mismatch and rank heterogeneity without reconstructing full updates in d×s space. As far as I know, that's new relative to FSLoRA and friends. The authors also prove convergence for the rank-homogeneous variant (SeFoRA-Ho) under standard smoothness and bounded-moment assumptions, and they correctly identify the error floor as spectral-tail energy. Experiments on RoBERTa-Large over four GLUE tasks are reasonable and show SeFoRA matches or beats FlexLoRA/FLoRA/FSLoRA.\n\nBut there's a real problem with the headline theoretical claim. The abstract and Contributions say SeFoRA-Ho converges at O(1/T), and Table 1 lists O(1/T+c). That is not what Corollary 1 proves. With γ_T chosen as they do, the bound is 4γ_T(χC3+2C4) + O(γ_T^2 + γ_T^3), so even with zero sketching error the rate is Θ(1/√T), and with ε>0 the floor is ~√ε, not an additive c. The stated theorem supports O(1/√T + √ε), not O(1/T + c). This is a load-bearing misstatement, not a typo; it should be fixed in revision.\n\nTwo more soft spots, in proportion. The rank-heterogeneous SeFoRA — the headline algorithm — has no convergence guarantee; the paper only proves SeFoRA-Ho. That is clearly stated, but it is a big gap in a paper whose title emphasizes heterogeneous ranks. And the experiments omit HetLoRA, a closely related rank-heterogeneous baseline the authors discuss at length; its absence weakens the empirical claim. Both are addressable.\n\nWho should read this: anyone working on federated LoRA aggregation or privacy-preserving FL with adapters. The linear aggregation property is valuable for secure aggregation and DP, even if the rate claim is inflated. It deserves serious peer review, but I would push for correction of the theoretical rate before acceptance.","headline":"Novel sketching trick for federated LoRA that cleanly solves rank heterogeneity and the bilinear mismatch, but the advertised O(1/T) convergence rate is not supported by the proof (it's O(1/sqrt(T)) plus a floor).","tokens_in":36805,"tokens_out":3563,"would_cite":true,"duration_ms":34576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated LoRA's rank clash and bilinear mismatch both dissolve when clients send fixed-size linear sketches of their updates instead of factor matrices, with an $\\mathcal{O}(1/T)$ convergence rate up to a spectral-tail error floor.","keywords":["federated learning","low-rank adaptation","LoRA","matrix sketching","rank heterogeneity","bilinear mismatch","parameter-efficient fine-tuning","convergence analysis"],"falsifier":"Measure the average spectral tail energy $\\tau^2_{\\varrho+1}(\\sum_i p_i B_i A_i)$ of the aggregated client updates during a real federated fine-tuning run: if it is large relative to the early-round gradient norms, Corollary 1 predicts a stagnation plateau in the gradient norm that should be visible as training proceeds. Conversely, run the rank-heterogeneous SeFoRA on a synthetic task where clients' updates are constructed to occupy disjoint rank-$r$ subspaces; the paper gives no convergence bound for this setting, so an experiment showing the gradient norm stalling or diverging there would delimit the method's reach.","tokens_in":35793,"feed_emoji":"🧩","tokens_out":16763,"duration_ms":132172,"temperature":0.7,"pith_summary":"Federated fine-tuning of large models with LoRA runs into two problems that neither setting has alone: clients may pick different adapter ranks, making their factor matrices impossible to average directly, and even with matching ranks, averaging the factors first and multiplying afterwards gives a different matrix from averaging the full updates — the bilinear mismatch. SeFoRA sidesteps both by having each client send two fixed-size linear sketches of its local update $\\Delta W_i = B_i A_i$ rather than the factors themselves. Because sketching is linear, the federator's average of the sketches is exactly the sketch of the averaged update, so rank heterogeneity stops mattering to the aggregation and the mismatch is gone. For the rank-homogeneous version SeFoRA-Ho the paper proves convergence to a neighborhood of a first-order stationary point at rate $\\mathcal{O}(1/T)$, with the neighborhood size bounded by the spectral tail energy of the aggregated client updates. Experiments fine-tuning RoBERTa-Large on four GLUE tasks show SeFoRA matching or beating state-of-the-art rank-heterogeneous baselines at lower server-side computation.","feed_headline":"Aggregate sketches, not factors: federated LoRA converges","feed_subtitle":"Clients send fixed-size sketches of local updates; rank-heterogeneous adapters aggregate with no bilinear mismatch.","key_machinery":"The load-bearing object is the two-sided random sketching operator of [48]. For a target matrix $W$ and independent standard Gaussian matrices $\\Omega\\in\\mathbb{R}^{s\\times r}$ and $\\Psi\\in\\mathbb{R}^{k\\times d}$ with $k>r+1$, the pair of sketches $Y=W\\Omega$ and $Z=\\Psi W$ determines a rank-$r$ approximation $\\hat{W}=\\hat{B}\\hat{A}$ whose expected squared error is at most $(1 + \\frac{r}{k-r-1})(1 + \\frac{\\varrho}{r-\\varrho-1})\\tau^2_{\\varrho+1}(W)$, where $\\tau^2_{\\varrho+1}(W)$ is the spectral tail energy of $W$ beyond rank $\\varrho+1$ (Lemma 2). The property that does the work is linearity: averaging clients' sketch pairs gives the sketch of the averaged update, so the federator aggregates in a small subspace of size $dr + ks$ instead of the full model space. The inverse map (Algorithm 1) is a short QR-based factorization that runs $Y = \\hat{B}R$, factors $(\\Psi\\hat{B}) = UT$, and returns $\\hat{A} = T^{-1}U^\\top Z$.","core_discovery":"The central claim is that moving federated LoRA aggregation into a sketched subspace removes both the bilinear mismatch and rank heterogeneity at once. Each client sketches its local update $\\Delta W_i$ into $Y_i = \\Delta W_i\\Omega \\in \\mathbb{R}^{d\\times r}$ and $Z_i = \\Psi\\Delta W_i \\in \\mathbb{R}^{k\\times s}$ using shared Gaussian matrices $\\Omega$ and $\\Psi$ with $k > r+1$; the federator averages the sketches, and the sketch-inversion procedure of [48] recovers a rank-$r$ approximation of the aggregated update $\\sum_i p_i\\Delta W_i$. Linearity of the sketch operator makes the average of the sketches exactly the sketch of the average, eliminating the factor-level mismatch that plagues FedAvg-style LoRA, and the fixed sketch dimensions make client ranks irrelevant to the aggregation. The paper further proves (Theorem 1, Corollary 1, Section 4) that the rank-homogeneous variant SeFoRA-Ho converges to a neighborhood of a first-order stationary point of the original LoRA objective at rate $\\mathcal{O}(1/T + c)$, where the floor $c$ is bounded by the spectral tail energy of the aggregated client updates (Assumption 4); for the rank-heterogeneous version the claim is supported by experiments rather than a convergence theorem.","pith_inferences":["A consequence the paper leaves implicit: the federator can monitor the reconstruction error $\\|\\hat{W} - \\sum_i p_i\\Delta W_i\\|_F$ each round and grow the sketch width $k$ when it rises, trading communication for a smaller floor $c$ — a practical rule that follows directly from Corollary 1 but is not stated.","The convergence proof covers only the rank-homogeneous variant; whether the rank-truncation and rank-adaptation steps in the heterogeneous algorithm keep the tail energy small is untested, so the natural next theorem is a tail-energy bound for SeFoRA with adaptive client ranks.","The experiments show client ranks shrinking as training progresses, which suggests the global rank $r$ itself could decay over time; whether the guarantee degrades gracefully under such rank decay is an open question the paper does not address.","The similar accuracy of FlexLoRA and SeFoRA in several settings hints that the deciding factor between full-space and sketched aggregation is the tail energy of the aggregate rather than the aggregation space itself — a hypothesis testable by measuring tail energy directly on the same runs."],"forward_implications":["SeFoRA-Ho carries a non-convex convergence rate on the original LoRA objective ($\\mathcal{O}(1/T)$ up to the sketching floor) while neither aggregating in the full model space nor freezing one factor — a combination the cited baselines do not achieve.","Server-side cost per layer drops from full-space reconstruction (such as $O(d^2s)$ SVD or $O(Ndrs)$ update reconstruction) to $O((N+r)r(d+s))$, making large-model federation cheaper at the federator.","Clients can change their local ranks during training, growing or shrinking them, because the sketch dimensions stay fixed; adaptive rank allocation therefore combines directly with aggregation.","Because aggregation is linear, secure aggregation and lower-noise differential privacy for federated LoRA become reachable, which the paper names as future work.","The error floor $c$ is a real accuracy ceiling set by how well the aggregated update is captured by a rank-$r$ matrix, so tasks with high-rank shared updates pay a visible price."],"supporting_citations":[{"why":"Supplies the two-sided Gaussian sketching operator and the QR-based sketch-inversion procedure (Algorithm 1) whose linearity and error bound (Lemma 2) carry the whole aggregation scheme.","marker":"[48]"},{"why":"Defines the LoRA parametrization $\\Delta W = BA$ and the fine-tuning objective on which the federated problem is built.","marker":"[17]"},{"why":"FedIT, the naive FedAvg-style LoRA baseline in the rank-homogeneous comparison that exhibits the bilinear mismatch SeFoRA-Ho removes.","marker":"[38]"},{"why":"HetLoRA, the zero-padding rank-heterogeneous baseline that keeps the bilinear mismatch; the main method SeFoRA is designed to surpass.","marker":"[41]"},{"why":"FLoRA, the stacking baseline with exact aggregation in an enlarged space; its strongly-convex-only analysis motivates the non-convex guarantee pursued here.","marker":"[45]"},{"why":"FlexLoRA, the full-model-space SVD baseline whose server-side $O(d^2s)$ cost SeFoRA avoids while matching its accuracy.","marker":"[46]"},{"why":"FSLoRA, the concurrent sketching baseline with a modified training objective; SeFoRA's contribution is to prove convergence on the original LoRA objective instead.","marker":"[47]"},{"why":"One source of the bounded-factor assumption (Assumption 3) that the convergence proof relies on.","marker":"[43]"},{"why":"Second source of the bounded-factor assumption (Assumption 3) used in the convergence proof.","marker":"[54]"}],"fun_headline_variants":["Sketch it and share: federated LoRA without rank clashes","Federated LoRA: sketch updates to bypass rank mismatch","SeFoRA: sketch-aggregated LoRA for heterogeneous clients","One sketch fits all ranks: federated LoRA that converges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee and the practical accuracy both rest on the aggregated client update matrix being well approximated by a rank-$r$ matrix: if a meaningful share of its energy sits beyond rank $r$, the sketching error floor $c$ dominates the $\\mathcal{O}(1/T)$ term and the final model can sit far from a stationary point, and no bound is proven for the rank-heterogeneous algorithm where that assumption is most strained.","fun_headline_variants_meta":{"raw":{"variants":["Sketch it and share: federated LoRA without rank clashes","Federated LoRA: sketch updates to bypass rank mismatch","SeFoRA: sketch-aggregated LoRA for heterogeneous clients","One sketch fits all ranks: federated LoRA that converges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1279,"prompt_tokens":1018,"completion_tokens":261,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":188}},"tokens_in":634,"tokens_out":261,"duration_ms":2940,"temperature":1.0,"reasoning_tokens":188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:39.397259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the average spectral tail energy $\\tau^2_{\\varrho+1}(\\sum_i p_i B_i A_i)$ of the aggregated client updates during a real federated fine-tuning run: if it is large relative to the early-round gradient norms, Corollary 1 predicts a stagnation plateau in the gradient norm that should be visible as training proceeds. Conversely, run the rank-heterogeneous SeFoRA on a synthetic task where clients' updates are constructed to occupy disjoint rank-$r$ subspaces; the paper gives no convergence bound for this setting, so an experiment showing the gradient norm stalling or diverging there would delimit the method's reach.","supporting_citations":[{"cited_title":"Practical sketching algorithms for low-rank matrix approximation.SIAM Journal on Matrix Analysis and Applications, 38(4):1454–1485, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the two-sided Gaussian sketching operator and the QR-based sketch-inversion procedure (Algorithm 1) whose linearity and error bound (Lemma 2) carry the whole aggregation scheme."},{"cited_title":"Towards building the federated GPT: Federated instruction tuning","cited_arxiv_id":null,"evidence_quote":"FedIT, the naive FedAvg-style LoRA baseline in the rank-homogeneous comparison that exhibits the bilinear mismatch SeFoRA-Ho removes."},{"cited_title":"Heterogeneous LoRA for federated fine-tuning of on-device foundation models","cited_arxiv_id":null,"evidence_quote":"HetLoRA, the zero-padding rank-heterogeneous baseline that keeps the bilinear mismatch; the main method SeFoRA is designed to surpass."},{"cited_title":"FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations.Advances in Neural Information Processing Systems, 37:22513–22533, 2024","cited_arxiv_id":null,"evidence_quote":"FLoRA, the stacking baseline with exact aggregation in an enlarged space; its strongly-convex-only analysis motivates the non-convex guarantee pursued here."},{"cited_title":"Federated sketching LoRA: On-device collaborative fine-tuning of large language models","cited_arxiv_id":null,"evidence_quote":"FSLoRA, the concurrent sketching baseline with a modified training objective; SeFoRA's contribution is to prove convergence on the original LoRA objective instead."},{"cited_title":"Selective aggregation for low-rank adaptation in federated learning","cited_arxiv_id":null,"evidence_quote":"One source of the bounded-factor assumption (Assumption 3) that the convergence proof relies on."},{"cited_title":"Robust federated finetuning of LLMs via alternating optimization of LoRA.Advances in Neural Information Processing Systems, 38:120038–120090, 2026","cited_arxiv_id":null,"evidence_quote":"Second source of the bounded-factor assumption (Assumption 3) used in the convergence proof."}],"review_version":1}