Pith. sign in

REVIEW 3 major objections 4 minor 28 references

In federated LoRA fine-tuning, the optimal depth of parameter sharing is a function of client similarity, and a tree-structured aggregation that shares shallow layers broadly while specializing deep layers outperforms flat-model methods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:01 UTC pith:C3PXWT7K

load-bearing objection A strong empirical paper on layer-wise tree aggregation for federated LoRA, but the warm-up stability evidence in Appendix C.5 is a duplicated table and should be fixed before it can be trusted. the 3 major comments →

arxiv 2603.13282 v2 pith:C3PXWT7K submitted 2026-02-27 cs.LG cs.AI

FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Jieming Bian , Lei Wang , Letian Zhang , Jie Xu This is my paper
classification cs.LG cs.AI
keywords federated learningLoRAfine-tuningpersonalizationheterogeneitytree-structured aggregationhierarchical clusteringlarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Federated fine-tuning of large language models with LoRA faces two kinds of heterogeneity: clients hold different data distributions (statistical, horizontal), and transformer layers perform different functional roles (vertical). Existing personalized methods, whether they use a single global model, dual global/local modules, or flat client clustering, treat the LoRA module as one monolithic block. The paper argues these two dimensions are coupled: the safe depth at which a group of clients can keep sharing parameters is determined by how similar those clients are. FedTreeLoRA realizes this by building a global client-similarity tree from warmup LoRA updates, choosing per layer how many clusters to use, and synthesizing each client's parameters by mixing its peer-cluster expert with a frozen external expert. Across GLUE and FLAN benchmarks, the paper reports consistent gains over both global and flat-clustering personalized baselines with negligible added parameters.

Core claim

At the center of the paper is the claim that there is no single answer to 'how much should federated clients share?' The correct answer is a function of two coupled variables: the statistical distance between clients and the depth of the transformer layer. The two dimensions are orthogonal in source—one comes from external data distributions, the other from the internal architecture—but coupled in interaction, so the optimal aggregation boundary shifts shallower as client distributions diverge. Motivational experiments show that aggregating deep layers under heterogeneous clients can be worse than training locally, and that the optimal sharing boundary moves from all 24 layers under homogene

What carries the argument

The central object is the global merge tree built by agglomerative hierarchical clustering on a cross-client distance matrix of warmup LoRA B matrices; it provides a single topological skeleton in which every layer's partition is a valid cut, preventing contradictory reshuffling between adjacent layers. Layer-wise alignment then picks a cluster count per layer via Silhouette score (with a threshold tau for the single-cluster case) under a monotonicity constraint, so specialization only increases with depth. The parameter-synthesis mechanism is the Cluster-External expert: for each client and layer, the peer cluster's averaged LoRA parameters are combined with the averaged 'rest of world' par

Load-bearing premise

The whole structure rests on the assumption that a single tree computed once from warmup LoRA B-matrix distances captures the true, stable similarity topology among clients; if warmup updates are noisy or client relationships shift during training, the layer-wise sharing boundaries will be misaligned with what the data actually supports.

What would settle it

Run FedTreeLoRA with a deliberately misleading warmup—e.g., have clients train on a subset that does not reflect their full data distribution—and compare the resulting fixed tree against a tree recomputed after a few federated rounds. If the recomputed tree changes the layer-wise partition and improves accuracy, the claim that warmup topology is sufficient and stable fails; if performance is unchanged, the fixed-tree assumption is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Under non-IID GLUE (RoBERTa-Large, 20 clients, rank 4), the method reaches an average accuracy of 92.36, beating the best flat-clustering baseline by about 1.2 points and the global baseline by about 5.2 points, with nearly no added trainable parameters.
  • The layer-wise adaptive cluster count beats every uniform-depth setting (global, coarse, fine-grained), so decoupling aggregation depth from model depth is itself the source of most of the gain.
  • A single global tree outperforms independent per-layer clustering by about 2.9 points average, indicating that topological coherence across adjacent layers matters for stable fine-tuning.
  • Gains transfer from NLU to NLG on LLaMA-2-7B and BLOOM-7B under task-heterogeneous client partitions, so the mechanism is not tied to one architecture.
  • The analysis yields an O(1/sqrt(T)) convergence rate under standard smooth non-convex assumptions, matching ordinary FedAvg.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the tree is fixed after warmup, the framework implicitly assumes client similarity is stable across training. A natural extension is to refresh or grow the tree when new clients join or when drift is detected; the paper's dynamic-participation experiment only tests a single insertion at a fixed round.
  • The scalar mixing coefficient lambda gives each client a soft interpolation between peer consensus and global average; interpreted as a hierarchical prior, this suggests the method should be most useful when client groups are unbalanced in size or when task similarity has a nested structure, which the paper does not directly stress-test.
  • The silhouette-plus-threshold rule for choosing cluster counts is a heuristic with a reported plateau rather than a principled model-selection criterion; a validation-based or Bayesian alternative could be tested against it on the same benchmarks.
  • If layered specialization follows the same depth-similarity coupling in other fine-tuning paradigms (e.g., adapters other than LoRA), the tree skeleton may transfer as a general design, but the paper only demonstrates LoRA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. FedTreeLoRA targets personalized federated fine-tuning of LLMs with LoRA under the claim that existing methods rely on a 'flat-model assumption.' The paper proposes a two-step framework: (1) after a warmup phase, construct a global client-merging tree via agglomerative hierarchical clustering on averaged Frobenius distances of LoRA B matrices (Eq. 3); (2) for each Transformer layer, select a cluster count c*_l under a monotonicity constraint using a Silhouette score with a hand-set threshold tau and search window K (Eq. 5-6). Parameter synthesis is done via a Cluster Expert and a frozen External Expert mixed by a learnable scalar (Eq. 7-8). The paper reports consistent gains over FedIT, FFA-LoRA, FedSA, FedDPA, FedALT, and FedLEASE on GLUE with RoBERTa-Large and FLAN with LLaMA-2-7B, plus ablations, sensitivity analyses, an alternative LLM family, and a convergence theorem. The central empirical claim is that gains come from layer-wise tree-structured aggregation rather than from additional capacity.

Significance. If the empirical results hold, the paper makes a useful conceptual contribution: separating horizontal (statistical) from vertical (functional) heterogeneity and showing that the desirable depth of parameter sharing depends on client similarity. The paper ships a public code repository, reports three-run averages, and includes extensive ablations (interaction mechanisms, fixed-depth vs. adaptive depth, global tree vs. independent clustering) and robustness checks over N, r, E, alpha, distance metrics, and an alternative LLM family. These are genuine strengths. However, the paper's only direct evidence for the fixed-tree-after-warmup assumption is invalid as reported (Table 15 vs. Table 13), and the two framework hyperparameters tau and K appear to be selected on the same benchmarks used for evaluation. The convergence theorem has a proof gap. The conceptual claim and the main trend are plausible, but the supporting evidence needs correction before the claims can be accepted at face value.

major comments (3)
  1. [Appendix C.5, Table 15 (vs. Table 13)] The warm-up sensitivity table does not measure what it claims. The '5 epochs' row (MNLI 82.94, QNLI 89.31, SST2 94.19, QQP 84.75, avg 87.80) is identical to FedTreeLoRA's row in Table 13, which is the N=16 task-heterogeneity experiment using cosine distance. The default GLUE setting used for the main results is N=20, Dir(0.5), Frobenius distance, where FedTreeLoRA averages 92.36. Thus Table 15 is mislabeled or copied, and the paper's only direct support for the fixed-tree-after-warmup assumption is invalid. Please rerun this sensitivity analysis under the default GLUE setting and report the resulting topology stability.
  2. [Sec. C.1 / Fig. 6] The two framework hyperparameters tau and K are selected by sweeping on the same GLUE benchmark tasks used for the headline results, and tau is calibrated from the observed silhouette scale of those tasks ('layer-wise Silhouette statistics ... naturally lie in the order of 10^-2'). As reported, this makes the 'significantly outperforms' claim partially dependent on test-set performance. Please state the default values of tau and K in the main text or algorithm and describe a selection rule (e.g., a validation split or a pre-specified pilot study) that is independent of the evaluation sets.
  3. [Appendix A.1-A.2, Theorem 5.1] Assumption A.3 is stated for 'client-level LoRA factors' A(t)_l,k and B(t)_l,k, but the proof applies the alignment inequalities (10)-(11) to A(t,0)_l,k and B(t,0)_l,k, which are Cluster Experts averaged over clients in S_k. The statement that averaged experts 'inherit' the alignment property with client k's gradient does not follow from the norm bounds and needs a proof (or the theorem should be stated as conditional on alignment for the aggregated experts). As written, the convergence theorem is not fully established from the stated assumptions.
minor comments (4)
  1. [Sec. 5.2, Eq. (5) vs. Algorithm 1 line 14] The search space in Eq. (5) is c* <= c < min(N, c*+K), which excludes c=N, while Algorithm 1 line 14 uses c <= min(N, c*+K-1), which allows c=N when c*+K-1 >= N. Clarify which definition is intended, since this affects whether full personalization is reachable.
  2. [Sec. 5.3, Eq. (7)] When S_k contains all clients, R_k is empty and the External Expert formula divides by zero. The text says the External Expert is 'zeroed to avoid redundancy,' but this case should be stated explicitly in Eq. (7) rather than only in prose.
  3. [Algorithm 1, line 1] The 'Require' line is malformed: it mixes the input list with a stray brace and an unfinished notation '{A clus ...}'. Please rewrite the input/output specification cleanly.
  4. [Table 17] The last row of Table 17 is labeled 'V AScalar-Mixed'; this appears to be a typo for 'Scalar-Mixed (Ours)'. Please correct.

Circularity Check

0 steps flagged

No significant circularity: the central claims are supported by independent experiments and external baselines; no prediction reduces to a fitted input.

full rationale

FedTreeLoRA's derivation chain is not circular. The central hypothesis (optimal sharing depth depends on client similarity) is established by the motivational studies in Sec. 4 and Appendix B, which sweep aggregation scope (8/16/24 layers) and measure downstream accuracy under controlled label-skew regimes; those studies are independent of the method's own equations. The method subsequently uses warmup LoRA B matrices (Eq. 3) to construct a global AHC tree and silhouette scores (Eq. 6) to choose per-layer cluster counts, but the resulting partitions are validated against fixed-depth ablations and external baselines (FedIT, FFA-LoRA, FedSA, FedDPA, FedALT, FedLEASE) on GLUE and FLAN. The final accuracy results are therefore not fitted to the same objective that defines the method. The threshold tau is a hyperparameter; Appendix C.1 sweeps it over a range centered on the observed silhouette scale and demonstrates a robustness plateau, which is standard sensitivity analysis rather than circular calibration. Self-citations (Bian et al. survey, FedALT baseline) are not load-bearing: they are background or baseline comparisons, and no uniqueness theorem is invoked from the authors' own prior work. One appendix passage (C.5, Table 15) appears to duplicate Table 13's task-heterogeneity numbers instead of providing fresh warm-up-duration results in the default GLUE setting; this is a reproducibility/validity defect that should be corrected, but it does not make the central derivation circular.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 2 invented entities

The central method depends on a handful of modeling choices: the hierarchical-layer assumption, B-matrix similarity, a fixed tree, and two tuned hyperparameters. The convergence proof additionally relies on an alignment condition (A.3) that is assumed rather than verified. These are not disqualifying for an empirical ML paper, but they are exactly the load-bearing assumptions a replicator must test.

free parameters (3)
  • Heterogeneity threshold tau = Not explicitly stated; swept over [-0.02, 0.10], with a claimed robust plateau in [0, 0.06]
    Controls when the algorithm splits from a global cluster; used in the scoring function Eq. (6) and tuned by sensitivity analysis on the same GLUE benchmarks.
  • Search window K = Not explicitly stated; claimed robust plateau for K in {3,4,5}
    Limits how quickly cluster count can grow across layers (Eq. 5); chosen by sensitivity sweep on the main benchmarks.
  • Warm-up epochs E_warm = 10 epochs (NLU, 5 rounds); 2 epochs (NLG, 1 round)
    Determines the warmup LoRA matrices used to build the tree; stability is checked in Appendix C.5 but the choice itself is a design hyperparameter.
axioms (6)
  • domain assumption Transformer layers are functionally hierarchical: shallow layers encode general linguistic features, deep layers encode task-specific semantics.
    Invoked throughout (Sec. 4, Fig. 1) to justify layer-wise aggregation and the monotonicity constraint. The paper's own motivational studies provide the only federated-specific evidence; broader support comes from cited centralized analyses.
  • domain assumption LoRA B matrices carry task-specific semantic variation while A matrices capture shared representations, so pairwise distance between B updates is a valid client-similarity measure.
    Used in Eq. (3) and justified empirically in Appendix G, but it is a modeling choice about which low-rank factors encode client identity.
  • domain assumption Warmup-trained LoRA B matrices provide a stable client topology, and a single fixed tree remains valid for all subsequent optimization rounds.
    Assumed in Sec. 5.1 and defended only indirectly in Appendix C.5. If topology drifts, the method has no mechanism to adapt its tree.
  • domain assumption Silhouette coefficient with a hand-set threshold tau selects the correct cluster count at each layer, and monotonicity preserves semantic continuity.
    Core to Eq. (6) and the adaptive depth alignment; the choice of tau is calibrated empirically rather than derived.
  • ad hoc to paper Assumption A.3: LoRA matrices are bounded and satisfy a gradient-alignment condition with constants mu_A, mu_B > 0.
    Introduced specifically to make the convergence theorem tractable; the constants are not estimated or verified in experiments.
  • standard math Standard smoothness and bounded-variance stochastic gradient assumptions (A.1, A.2).
    Common in non-convex federated optimization; used as background for Theorem 5.1.
invented entities (2)
  • Global dependency tree T with layer-wise cuts no independent evidence
    purpose: Constrains per-layer client partitions and defines the topology of parameter sharing.
    An algorithmic abstraction with no falsifiable handle outside the reported benchmarks and ablations; its validity is inferred from performance gains.
  • Cluster Expert and External Expert per client-layer no independent evidence
    purpose: Blend peer-cluster consensus with a frozen global signal during forward passes (Eq. 8).
    Algorithmic constructs, not physical entities; the only evidence is the empirical ablation table.

pith-pipeline@v1.3.0-alltime-deepseek · 34202 in / 9789 out tokens · 101713 ms · 2026-08-02T20:01:02.597026+00:00 · methodology

0 comments
read the original abstract

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side \textit{statistical heterogeneity} but treated the model as a monolithic block, ignoring the \textit{functional heterogeneity} across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are \textit{orthogonal in source yet coupled in interaction}, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose \textbf{FedTreeLoRA}, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow `trunks' while progressively specializing on deep `branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

Figures

Figures reproduced from arXiv: 2603.13282 by Jieming Bian, Jie Xu, Lei Wang, Letian Zhang.

Figure 1
Figure 1. Figure 1: Vertical Heterogeneity. Aggregating only shallow layers significantly outperforms aggregating deep layers. We posit that treating all LoRA layers uniformly, as done in current Flat-Model approaches, leads to inefficient param￾eter sharing. In this section, we conduct two motivational studies to quantify how vertical heterogeneity (layer depth) interacts with horizontal heterogeneity (client data distribu￾t… view at source ↗
Figure 2
Figure 2. Figure 2: The Coupling Effect of Dual Heterogeneity. As client distributions diverge (from Homogeneous to Heterogeneous), the optimal sharing boundary shifts from deep to shallow layers. based on the severity of data heterogeneity. In the Ho￾mogeneous setting, performance improves monotonically as aggregation extends to deeper layers, benefiting from maximum knowledge sharing. However, as heterogeneity increases, th… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of FedTreeLoRA. (1) Global Topological Structure Modeling: A hierarchy tree is built via AHC on client LoRA B matrices during warmup to capture cross-client relationships. (2) Adaptive Layer-wise Alignment: For each layer l, the optimal cluster count c ∗ l is dynamically selected under a monotonicity constraint. (3) Cluster-External Expert Mechanism: Each client synthesizes parameters by mixing a … view at source ↗
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Extended Motivational Studies. (a) Substantiates Observation 1 (Vertical Heterogeneity) on SST2 and QQP datasets. (b)–(d) Substantiate Observation 2 (Coupling Effect) across different tasks, confirming that the optimal sharing boundary consistently shifts towards shallower layers as data heterogeneity increases. randomly similar, separating them at shallow layers would fail to capture the true performance … view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity analysis of hyperparameters τ and K. FedTreeLoRA demonstrates a clear robustness plateau within τ ∈ [0, 0.06] and K ∈ {3, 4, 5}, while still outperforming all baselines even in suboptimal settings. Impact of Threshold τ . Recall that τ acts as the baseline score for the single-cluster case (c = 1), and thus determines how much statistical evidence must accumulate before a layer is allowed to br… view at source ↗
Figure 7
Figure 7. Figure 7: Layer-wise client clustering under task heterogeneity. Clients assigned to the same GLUE task form stable branches at deeper layers, while all tasks share common representations in shallow layers, illustrating the hierarchical specialization discovered by FedTreeLoRA. distance focuses on the directional orientation of the LoRA updates. As summarized in [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Layer-wise average pairwise Frobenius distance between two clients on QNLI under IID (identical label distributions) and Non-IID ([0.8, 0.2] vs. [0.2, 0.8]) settings. The upper-left and upper-right panels (Proj A at Query and Value) show nearly overlapping IID and Non-IID curves across all layers, indicating that A captures distribution-invariant shared representations. The lower-left and lower-right panel… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 1 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [6]

    BERT: Pre-training of deep bidirectional transformers for lan- guage understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.),Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Shor...

    2019

  3. [7]

    doi: 10.18653/v1/N19-1423

    Association for Compu- tational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/. Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning: A meta-learning approach.arXiv preprint arXiv:2002.07948,

    Pith/arXiv arXiv 2002

  4. [10]

    Per- sonalized federated fine-tuning for heterogeneous data: An automatic rank learning approach via two-level lora

    Hao, J., Wu, Y ., Payani, A., Lee, M., and Liu, M. Per- sonalized federated fine-tuning for heterogeneous data: An automatic rank learning approach via two-level lora. arXiv preprint arXiv:2503.03920,

    Pith/arXiv arXiv

  5. [11]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

    Pith/arXiv arXiv

  6. [12]

    doi: 10.18653/v1/P19-1356

    Association for Computa- tional Linguistics. doi: 10.18653/v1/P19-1356. URL https://aclanthology.org/P19-1356/. Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm.Neural computation, 6(2): 181–214,

    doi:10.18653/v1/p19-1356

  7. [13]

    P., Liu, T., Ziyin, L., Allen, N

    Liang, P. P., Liu, T., Ziyin, L., Allen, N. B., Auerbach, R. P., Brent, D., Salakhutdinov, R., and Morency, L.-P. Think locally, act globally: Federated learning with local and global representations.arXiv preprint arXiv:2001.01523,

    Pith/arXiv arXiv 2001

  8. [14]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V . Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

    Pith/arXiv arXiv 1907

  9. [16]

    Qi, J., Luan, Z., Huang, S., Fung, C., Yang, H., and Qian, D

    doi: 10.1109/ICASSP48485.2024.10448255. Qi, J., Luan, Z., Huang, S., Fung, C., Yang, H., and Qian, D. Fdlora: Personalized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925,

    arXiv 2024

  10. [17]

    Treelora: Efficient continual learning via layer- wise loras guided by a hierarchical gradient-similarity tree.arXiv preprint arXiv:2506.10355,

    Qian, Y .-Y ., Xu, Y .-Z., Zhang, Z.-Y ., Zhao, P., and Zhou, Z.-H. Treelora: Efficient continual learning via layer- wise loras guided by a hierarchical gradient-similarity tree.arXiv preprint arXiv:2506.10355,

    Pith/arXiv arXiv

  11. [18]

    Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313,

    Sun, Y ., Li, Z., Li, Y ., and Ding, B. Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313,

    Pith/arXiv arXiv

  12. [19]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Pith/arXiv arXiv

  13. [20]

    Flat-Model Assumption

    For each setting, we randomly sample a different client to join the system in order to avoid bias toward a specific client distribution. We report: (1) the performance of the original clients without client joining, (2) the performance of existing clients after the new client joins, (3) the performance of the new client after joining the hierarchy, and (4...

    arXiv 2091

  14. [21]

    Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023a

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S...

    Pith/arXiv arXiv

  15. [23]

    Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations.arXiv preprint arXiv:2409.05976, 2024b

    Wang, Z., Shen, Z., He, Y ., Sun, G., Wang, H., Lyu, L., and Li, A. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations.arXiv preprint arXiv:2409.05976, 2024b. Yang, Y ., Long, G., Shen, T., Jiang, J., Blumenstein, M., et al. Dual-personalizing adapter for federated foundation models.Advances in Neural Information Pr...

    Pith/arXiv arXiv

  16. [24]

    Towards building the fed- eratedgpt: Federated instruction tuning

    Zhang, J., Vahidian, S., Kuo, M., Li, C., Zhang, R., Yu, T., Wang, G., and Chen, Y . Towards building the fed- eratedgpt: Federated instruction tuning. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6915–6919. IEEE,

    2024

  17. [25]

    one-round descent + telescoping

    Zhang, L., Chen, B., Bian, J., Wang, L., and Xu, J. Fedel: Federated elastic learning for heterogeneous devices. In Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., and Chen, N. (eds.),Advances in Neural Information Processing Systems, volume 38, pp. 35665– 35696. Curran Associates, Inc., 2025a. Zhang, Z., Liu, P., Xu, J., and Hu,...

    2025

  18. [26]

    We simulate a federated system with N= 16 clients, partitioned into four groups of four label IID clients each, corresponding to four GLUE tasks: MNLI, QNLI, SST-2, and QQP

    Task Heterogeneity.Following the protocol of FedLEASE (Wang et al., 2025), we evaluate a task-heterogeneous setting where client diversity arises from performing different NLP tasks rather than label skew. We simulate a federated system with N= 16 clients, partitioned into four groups of four label IID clients each, corresponding to four GLUE tasks: MNLI,...

    2025

  19. [28]

    negative transfer

    87.07±0.45 52.22±0.39 52.12±0.15 73.75±0.90 66.79 FedTreeLoRA (Ours) 88.84±0.34 55.20±1.01 52.85±0.45 74.23±0.68 67.78 G. Justification for Similarity Measurement via LoRABMatrices In FedTreeLoRA, we utilize the distance between clients’ LoRA B matrices to construct the topological tree. This design choice is grounded in the asymmetric roles of adapter ma...

    2025

  20. [2001]

    G., Aggarwal, V ., Singh, A

    Arivazhagan, M. G., Aggarwal, V ., Singh, A. K., and Choud- hary, S. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818,

    Pith/arXiv arXiv 1912

  21. [2017]

    Fedmm: Federated multi- modal learning with modality heterogeneity in compu- tational pathology

    Peng, Y ., Bian, J., and Xu, J. Fedmm: Federated multi- modal learning with modality heterogeneity in compu- tational pathology. InICASSP 2024 - 2024 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 1696–1700,

    2024

  22. [2018]

    Taming cross-domain representation variance in federated proto- type learning with heterogeneous data domains.arXiv preprint arXiv:2403.09048, 2024a

    Wang, L., Bian, J., Zhang, L., Chen, C., and Xu, J. Taming cross-domain representation variance in federated proto- type learning with heterogeneous data domains.arXiv preprint arXiv:2403.09048, 2024a. Wang, L., Bian, J., Zhang, L., and Xu, J. Adaptive lora experts allocation and selection for federated fine-tuning. In Belgrave, D., Zhang, C., Lin, H., Pa...

    Pith/arXiv arXiv

  23. [2019]

    R., Ezzeldin, Y

    Babakniya, S., Elkordy, A. R., Ezzeldin, Y . H., Liu, Q., Song, K.-B., El-Khamy, M., and Avestimehr, S. Slora: Federated parameter efficient fine-tuning of language models.arXiv preprint arXiv:2308.06522,

    Pith/arXiv arXiv

  24. [2020]

    Higher layers need more lora experts.arXiv preprint arXiv:2402.08562,

    Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y ., Guo, X., Yang, J., and Subrahmanian, V . Higher layers need more lora experts.arXiv preprint arXiv:2402.08562,

    Pith/arXiv arXiv

  25. [2023]

    Hydralora: An asymmetric lora architecture for efficient fine-tuning

    10 FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning Tian, C., Shi, Z., Guo, Z., Li, L., and Xu, C. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245,

    Pith/arXiv arXiv

  26. [2024]

    Bian, J., Peng, Y ., Wang, L., Huang, Y ., and Xu, J

    doi: 10.1109/TSP.2024.3408631. Bian, J., Peng, Y ., Wang, L., Huang, Y ., and Xu, J. A survey on parameter-efficient fine-tuning for foundation models in federated learning.arXiv preprint arXiv:2504.21099, 2025a. Bian, J., Wang, L., Zhang, L., and Xu, J. Lora-fair: Feder- ated lora fine-tuning with aggregation and initialization re- finement. InProceeding...

    arXiv 2024

  27. [2025]

    9 FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A com- prehensive survey.arXiv preprint arXiv:2403.14608,

    Pith/arXiv arXiv

  28. [2026]

    doi: 10.1609/aaai.v40i24. 39054. URL https://ojs.aaai.org/index. php/AAAI/article/view/39054. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y ., Fe- dus, W., Li, Y ., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53,

    work page doi:10.1609/aaai.v40i24