Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Heterogeneous LoRA allocation, chosen by layer importance, lets memory-limited clients fine-tune foundation models efficiently.

desk verdict Useful extension of FedRA with genuinely new importance-guided allocation strategies, but the headline gains are weakened by test-set proxy usage and post-hoc pattern selection. read the letter →

arxiv 2506.12213 v1 pith:C6MK23LC submitted 2025-06-13 cs.LG cs.DC

Zikai Zhang , Ping Liu , Jiahao Xu , Rui Hu This is my paper
classification cs.LGcs.DC
keywords FederatedLearningFoundationModelFine-TuningLow-RankAdaptationLoRAResourceHeterogeneityHeterogeneousAllocationFisherInformationMatrixParameter-Efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated fine-tuning of foundation models usually assumes every client can train the same LoRA adapters, but real clients have very different memory budgets. Fed-HeLLo instead assigns each client a different subset of LoRA layers, matched to its resources, and decides which layers each client should train. The paper argues that allocation should follow layer importance: Fisher Information scores capture importance that shifts during training, while geometric allocation shapes capture the stable roles of shallow and deep layers. Across five datasets and three data-distribution settings, Fed-HeLLo reports higher accuracy than random layer allocation and rank-based alternatives, with lower compute, memory, and communication costs. The practical stake is that low-memory devices can contribute data to foundation-model fine-tuning without being excluded or forcing everyone to the weakest client.

What carries the argument

The load-bearing mechanism is the allocation map m_i in {0,1}^l, which marks which LoRA layers a client trains, together with two importance signals that set the probabilities for sampling these maps. FIM-HLA converts per-layer gradient-norm scores into an allocation probability distribution, so layers with higher Fisher Information are trained by more clients. GD-HLA and RGD-HLA instead impose a global geometric prior, Triangle, Inverted Triangle, Bottleneck, or Uniform, on the collective allocation, stabilizing early rounds before FIM scores become informative. The co-design runs RGD-HLA for a warm-start phase and then refreshes the FIM-based probabilities every T_FIM rounds, so the final allocation blends dynamic and intrinsic layer importance.

What would settle it

Run Fed-HeLLo with FIM scores computed on a proxy dataset deliberately chosen to be unrepresentative of client domains, for example ImageNet images for clients holding medical text, holding everything else fixed, and compare final accuracy against FedRA; the central claim is falsified if Fed-HeLLo's advantage over random allocation disappears or reverses in that setting.

Watch

Extended reading notes

Core claim

Fed-HeLLo's central claim is that heterogeneous LoRA allocation can be both memory-efficient and more accurate than uniform or random allocation, provided each client's trainable LoRA layers are chosen by layer importance rather than at random. The framework lets every client train only c_i of the l LoRA layers, and the server aggregates updates only over the layers that client actually trained. To choose subsets, the server computes Fisher Information Matrix (FIM) scores from a small proxy dataset and samples allocation maps from a probability distribution weighted by those scores, giving FIM-HLA. Because FIM scores are noisy early in training, the first rounds instead use Randomized Geometrically-Defined HLA (RGD-HLA), which biases the collective allocation toward geometric patterns, Triangle, Inverted Triangle, Bottleneck, or Uniform, reflecting whether shallow, deep, or both groups of layers carry the task. In the reported experiments, Bottleneck is the strongest fixed pattern, and the full Fed-HeLLo pipeline, a warm-start with RGD-HLA followed by periodic FIM-HLA, outperforms random allocation FedRA and rank-based HETLoRA, FlexLoRA, and FLoRA across all five datasets, with the largest margins under Non-IID data, while cutting backward computation, memory, and communication cost.

Load-bearing premise

The gains depend on the server's small proxy dataset and early-round gradient norms actually revealing which LoRA layers matter to clients whose data can be extremely non-IID; if the proxy does not resemble client data, allocation drifts toward random and the advantage over FedRA shrinks.

Editorial extensions

If this is right

  • Under the tested settings, Fed-HeLLo improves average accuracy, Macro-F1, or Rouge-L over FedRA on all five datasets, with the largest gains in the most extreme Non-IID cases, such as +5.33 on LEDGAR 10/1.0.
  • Because each client trains only c_i of l LoRA layers, backward computational cost and communication cost scale down roughly with the allocated fraction, while rank-based alternatives reduce only optimizer-state memory and show large communication or convergence penalties.
  • The ablation results imply that the geometric pattern matters: Bottleneck allocation consistently beats Triangle, Inverted Triangle, and Uniform, especially under Non-IID data, suggesting that both shallow and deep layers should be kept trainable.
  • The co-design of RGD-HLA and FIM-HLA beats either component alone, indicating that a warm-start phase protects early training while FIM-based allocation improves later rounds.
  • Fed-HeLLo degrades gracefully when the proxy dataset is drawn from a different domain, as the Wikipedia-proxy experiment shows, but still outperforms FedRA and HETLoRA in that test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If FIM layer-importance rankings are stable across training rounds, one could refresh allocation probabilities less frequently than every T_FIM rounds or only when the client population changes, reducing server-side gradient computation further.
  • The reliance on a server-held labeled proxy is the main deployment constraint; a natural extension is to derive importance signals from aggregated client updates instead of server gradients, removing the proxy requirement entirely.
  • The Bottleneck pattern's dominance suggests a broader design principle for heterogeneous federated PEFT: preserve extreme layers, shallow and deep, while allowing the middle to be shared more sparsely; this may transfer to prompt tuning or adapter-based methods.
  • Within a single client, layer subsetting and rank adjustment target different memory components, so the framework could be extended to jointly allocate both the subset of LoRA layers and their ranks, rather than keeping rank fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. Fed-HeLLo proposes a federated fine-tuning framework in which clients train different subsets of LoRA layers according to their resource capabilities, coordinated by server-side heterogeneous LoRA allocation (HLA) strategies. The framework combines two allocation principles: FIM-HLA, which computes Fisher Information Matrix-based layer importance scores on a server-held proxy dataset and converts them into allocation probabilities, and GD-HLA/RGD-HLA, which imposes geometric patterns (Triangle, Inverted Triangle, Bottleneck, Uniform) on the collective distribution of trainable layers. The authors evaluate Fed-HeLLo on five datasets (CIFAR-100, LEDGAR, Natural Instruction, Dolly-15K, DomainNet-121) under IID and Non-IID data settings, reporting accuracy/F1/Rouge-L gains over FedRA and rank-based baselines, along with reduced backward computation, memory, communication, and training time. They also provide ablation studies for the GD-HLA patterns, the FIM-HLA proxy, and cross-silo settings.

Significance. If the reported gains are robust, Fed-HeLLo is a useful contribution to federated fine-tuning under resource heterogeneity, because it demonstrates that importance-guided, rather than purely random, allocation of trainable LoRA layers can improve global model quality while cutting per-client memory and communication cost. The paper covers a wider range of tasks, models, and non-IID settings than many prior works, and it ships code. However, the central claim depends on two practices that give the method access to test-distribution information: the FIM proxy is sampled from the test set (Section IV-A), and the GD-HLA pattern used in the main runs appears to be selected via test accuracy (Section IV-E2). The efficiency claim is also weakened by an incorrect complexity analysis (Section III-C). These issues do not invalidate the framework's potential, but they must be addressed before the reported margins over FedRA can be taken at face value.

major comments (4)
  1. [Section IV-A, Eq. (4)] FIM-HLA uses a proxy dataset sampled from the server's test set to compute layer importance scores, and these scores determine the allocation probabilities each round. Because the same test distribution is used for evaluation, the allocation strategy is informed by the evaluation target, whereas the baselines (FedRA, HETLoRA, etc.) receive no such information. The Wikipedia-proxy experiment (Table X) is a step in the right direction, but it covers only DomainNet-121 and one client setting, and the margin over FedRA shrinks from +1.83 to +1.37 in that setting. To support the central effectiveness claim, the paper should use proxy data that is not drawn from the test distribution for all five datasets, or explicitly restrict the claim to settings where the server holds test-distribution-like public data.
  2. [Section IV-E2, Table VIII] The GD-HLA pattern used in the main experiments is not identified in Sections IV-A through IV-C, but Table VIII evaluates four patterns on test accuracy and Bottleneck is the best. If Bottleneck is the pattern used in the main runs, then Fed-HeLLo's comparison with FedRA includes a post-hoc model-selection step on the evaluation set that FedRA does not enjoy. The paper should either state the pattern-selection procedure clearly, perform pattern selection on a validation split that is not used for test evaluation, or report results for all four patterns in the main tables.
  3. [Section III-C] The claimed computational complexity reduction is overstated. The paper gives FedAvg complexity O(τ l(d+R)^2 N s) and Fed-HeLLo complexity O(τ c(d+R)^2 N s / l), which implies a c/l reduction in forward+backward compute. This is not correct: forward passes must still go through all l transformer layers, and backpropagation to a trainable layer still requires passing gradients through the intervening frozen layers. What is reduced is the cost of computing and storing parameter gradients and optimizer states for the frozen layers, not the total forward+backward FLOPs. The empirical TFLOPs in Tables II-VI actually show Fed-HeLLo and FedRA have identical backward cost, which is consistent with the number of trainable layers but not with a c/l factor. Please revise the complexity analysis to distinguish parameter-update cost from inference/backprop through the full network.
  4. [Tables II-VI] No error bars or multiple-seed results are reported. Several margins over FedRA are small (e.g., +0.31 accuracy on CIFAR-100 IID 6:3:1, +0.05 on CIFAR-100 IID 1:1:1, +0.59 Rouge-L on Dolly-15K 6:3:1 average). Since client sampling, local data partitioning, and allocation are stochastic, these differences could be within run-to-run noise. The paper should report mean and standard deviation over at least 3-5 seeds, and ideally a paired significance test, for the comparisons against FedRA and the other baselines.
minor comments (6)
  1. [Abstract / Introduction] The contribution statement 'first work to design a federated LoRA-based fine-tuning framework with heterogeneous LoRA allocation' is contradicted by FedRA [45], which is a heterogeneous LoRA layer allocation method (randomly). The novelty should be framed as importance-guided or resource-aware allocation, not heterogeneous allocation per se.
  2. [Section III-A] The definition of trainable parameters θ is malformed: '{θ(1), · · ·, A(l), B(l)}' should be '{θ(1), ..., θ(l)}' (or include A(j), B(j) consistently).
  3. [Algorithm 1] In line 14, the aggregation uses δ^t but the local updates are indexed δ^{t+1}_i in lines 12 and the text. Please align the notation.
  4. [Section IV-C, DomainNet-121 paragraph] Typo: 'continures' should be 'continues'.
  5. [Table VIII heading] The table heading has 'CiFAR-100' with inconsistent capitalization; please use 'CIFAR-100'.
  6. [Section IV-E4] The descriptions of 'Proxy Data-1' and 'Proxy Data-2' are confusing: the names suggest the proxy data alone is the method, but the experiments actually use the proxy data for server-side fine-tuning. Please rename or clarify the experimental conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-level circularity: the reported gains are empirical and externally benchmarked; the FIM test-set proxy and post-hoc pattern selection are evaluation-leakage concerns, not equation-level reductions.

full rationale

No circular step reaches the quoted-evidence bar. Fed-HeLLo's local objective (Eq. 1), local updates (Algorithm 1 lines 11-12), and aggregation (Eq. 3) are standard FL operations; the HLA allocation maps are generated by Eqs. (5)-(7) from client resource levels and gradient-norm/FIM statistics, not from the final accuracy value. The effectiveness claim is an empirical comparison against external baselines such as FedRA, HETLoRA, FlexLoRA, and FLoRA on five datasets, so the central result has independent content. Two evaluation-integrity concerns are real but distinct from circularity. First, Section IV-A samples the FIM proxy D_FIM from the server's test set, so the FIM-HLA allocation probabilities in Eq. (5) encode test-distribution information; this can inflate the margin over FedRA, but it is data leakage rather than a self-definitional reduction, and Table X's Wikipedia-proxy experiment shows the method still beats FedRA without test-set samples. Second, Section IV-E2 selects the Bottleneck GD pattern by comparing four patterns on test accuracy, and the main runs do not state which GD prior RGD-HLA uses; if Bottleneck is carried into the main results, the configuration is post-hoc selected on the evaluation target, which weakens the comparison but does not make the reported accuracy equal to the selection criterion by construction. The only self-citations, Fed-Pilot [66] and Flowertune [20], appear in related-work lists and are not load-bearing for any derived claim; the novelty claim of being the first work on heterogeneous LoRA allocation also sits awkwardly with the authors' own Fed-Pilot [66], but that is a priority or correctness issue rather than a circularity. Under the hard rule that circularity requires a quoted equation-to-equation reduction or a fitted parameter renamed as prediction, the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger shows the method depends on several hand-chosen hyperparameters (TRGD, TFIM, proxy size, pattern choice) and on domain assumptions about layer importance and proxy-data representativeness. No new physical or conceptual entities are introduced.

free parameters (5)
  • TRGD (number of initial RGD-HLA rounds) = 50 for classification, 1 for instruction fine-tuning
    Chosen by the authors; no sensitivity analysis provided (Section IV-A).
  • TFIM (FIM-HLA update interval) = 50 for classification, 2 for instruction fine-tuning
    Chosen by the authors; no sensitivity analysis provided (Section IV-A).
  • Proxy dataset size |DFIM| = 100 (CIFAR-100), 50 (LEDGAR), 1000 (DomainNet-121), 50 (Natural Instruction), 50 (Dolly-15K)
    Manually selected; the proxy is sampled from the test set (Section IV-A).
  • GD-HLA geometric pattern = Bottleneck
    Bottleneck is selected after comparing four patterns on test accuracy (Table VIII, Section IV-E2).
  • LoRA rank r = 16 for ViT-base/BERT-base, 8 for DataJuicer-1B/OPT-1.3B
    Standard LoRA hyperparameter; affects memory footprint but not the core allocation mechanism.
assumptions (4)
  • domain assumption The FIM score, defined as the average squared gradient norm, is a reliable measure of LoRA layer importance for allocation.
    Invoked in Eq. (4) and used to construct allocation probabilities in Eq. (5); no theoretical or empirical justification is given.
  • domain assumption Layer importance has an intrinsic, task-independent structure expressible as fixed geometric patterns (shallow-priority, deep-priority, bottleneck, uniform).
    Section III-B2; the patterns are heuristics and the choice of Bottleneck is empirical on the test set.
  • domain assumption A server-held proxy dataset drawn from the test distribution is representative of client data without exposing privacy.
    Section IV-A; using test samples for allocation leaks evaluation information and may not generalize to unseen distributions.
  • ad hoc to paper Training only a subset c of l LoRA layers reduces backward compute by a factor c/l.
    Section III-C; in transformer backpropagation, gradients must flow through all layers, so the claimed O(c/l) speedup likely overstates the savings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation." pith.science (2026). https://pith.science/paper/C6MK23LC

@misc{pith2026250612213,
  author       = {Pith},
  title        = {Pith review of: Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6MK23LC}},
  note         = {Machine review of arXiv:2506.12213}
}
read the original abstract

Federated Learning has recently been utilized to collaboratively fine-tune foundation models across multiple clients. Notably, federated low-rank adaptation LoRA-based fine-tuning methods have recently gained attention, which allows clients to fine-tune FMs with a small portion of trainable parameters locally. However, most existing methods do not account for the heterogeneous resources of clients or lack an effective local training strategy to maximize global fine-tuning performance under limited resources. In this work, we propose Fed-HeLLo, a novel federated LoRA-based fine-tuning framework that enables clients to collaboratively fine-tune an FM with different local trainable LoRA layers. To ensure its effectiveness, we develop several heterogeneous LoRA allocation (HLA) strategies that adaptively allocate local trainable LoRA layers based on clients' resource capabilities and the layer importance. Specifically, based on the dynamic layer importance, we design a Fisher Information Matrix score-based HLA that leverages dynamic gradient norm information. To better stabilize the training process, we consider the intrinsic importance of LoRA layers and design a Geometrically-Defined HLA strategy. It shapes the collective distribution of trainable LoRA layers into specific geometric patterns, such as Triangle, Inverted Triangle, Bottleneck, and Uniform. Moreover, we extend GD-HLA into a randomized version, named Randomized Geometrically-Defined HLA, for enhanced model accuracy with randomness. By co-designing the proposed HLA strategies, we incorporate both the dynamic and intrinsic layer importance into the design of our HLA strategy. We evaluate our approach on five datasets under diverse federated LoRA fine-tuning settings, covering three levels of data distribution from IID to extreme Non-IID. Results show that Fed-HeLLo with HLA strategies is both effective and efficient.

Figures

Figures reproduced from arXiv: 2506.12213 by the authors.

Figure 1
Figure 1. (a) GPU memory usage for the ViT-base model, which includes a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Fed-HeLLo framework. Φ = {θ, Φ0} for a data point x sampled from Di . To solve this problem, one can follow the classic FL algorithm FedAvg [38]. Specifically, before training, the server broadcasts the initial global model Φ0 g = {θ 0 g , Φ0} to all clients, where θ 0 g repre￾sents the initialized global trainable parameters. Clients then save this model as their initial local models Φ0 i , i ∈ [n].… view at source ↗
Figure 3
Figure 3. The convergence of fine-tuning different LoRA layers on ViT-base [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The convergence of FedRA and FIM-HLA on LEDGAR dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The accuracy and training loss with respect to training rounds on CIFAR-100 dataset, across different data and device settings. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The radar chart presents the results of the ablation study focused on [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Heterogeneity-Oblivious Robust Federated Learning

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Horus makes federated learning robust to poisoning under extreme client heterogeneity by aggregating only LoRA adapters and detecting attackers from the spectral structure of the LoRA-A component.

Reference graph

Works this paper leans on

72 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    Slimfit: Memory-efficient fine-tuning of transformer-based models using training dynamics

    Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin Iancu, and Koushik Sen. Slimfit: Memory-efficient fine-tuning of transformer-based models using training dynamics. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Lo...

    work page 2024

  2. [2]

    Slora: Federated parameter efficient fine-tuning of language models

    Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06522, 2023

    arXiv 2023

  3. [3]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  4. [4]

    Strong baselines for parameter-efficient few-shot fine-tuning

    Samyadeep Basu, Shell Hu, Daniela Massiceti, and Soheil Feizi. Strong baselines for parameter-efficient few-shot fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 11024–11031, 2024

    work page 2024

  5. [5]

    Practical secure aggregation for federated learning on user- held data

    Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user- held data. arXiv preprint arXiv:1611.04482 , 2016

    arXiv 2016

  6. [6]

    Fltrust: Byzantine- robust federated learning via trust bootstrapping

    Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Gong. Fltrust: Byzantine- robust federated learning via trust bootstrapping. In Proceedings of NDSS, 2021

    work page 2021

  7. [7]

    Lexglue: A benchmark dataset for legal language understanding in english

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dubln, Ireland, 2022

    work page 2022

  8. [8]

    Data-juicer: A one-stop data processing system for large language mod- els

    Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, et al. Data-juicer: A one-stop data processing system for large language mod- els. In Companion of the 2024 International Conference on Management of Data, pages 120–134, 2024

    work page 2024

Show all 72 references
  1. [9]

    Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks

    Yixiong Chen, Alan Yuille, and Zongwei Zhou. Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks. In The Eleventh International Conference on Learning Representations, 2022

    2022

  2. [10]

    Heterogeneous lora for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 12903–12913, 2024

    2024

  3. [11]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606–3613, 2014

    2014

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009

    2009

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language un- derstanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2019

  6. [14]

    Heterofl: Computation and communication efficient federated learning for heterogeneous clients

    Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations , 2020

    2020

  7. [15]

    Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models. arXiv preprint arXiv:2203.06904 , 2022

    2022 arXiv

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International C...

    2020

  9. [17]

    Differential privacy

    Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming , pages 1–12. Springer, 2006

    2006

  10. [18]

    Model inversion attacks that exploit confidence information and basic countermeasures

    Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322–1333, 2015

    2015

  11. [19]

    Higher layers need more lora experts

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Sub- rahmanian. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562, 2024

    2024 arXiv

  12. [20]

    Flowertune: A cross-domain benchmark for federated fine-tuning of large language models

    Yan Gao, Massimo Roberto Scamarcia, Javier Fernandez-Marques, Mohammad Naseri, Chong Shen Ng, Dimitris Stripelis, Zexi Li, Tao Shen, Jiamu Bai, Daoyuan Chen, Zikai Zhang, et al. Flowertune: A cross-domain benchmark for federated fine-tuning of large language models. arXiv prep...

    2025

  13. [21]

    Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model

    Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing , 2023

    2023

  14. [22]

    Algorithm as 136: A k-means clustering algorithm

    John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics), 28(1):100–108, 1979

    1979

  15. [23]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2021

    2021

  16. [24]

    Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout

    Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems , 34:12876–12889, 2021

    2021

  17. [25]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021

    2021

  18. [26]

    Depthfl: Depthwise federated learning for heterogeneous clients

    Minjae Kim, Sangyoon Yu, Suhyun Kim, and Soo-Mook Moon. Depthfl: Depthwise federated learning for heterogeneous clients. In The Eleventh International Conference on Learning Representations , 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, JUNE 2025 14

    2022

  19. [27]

    Federated learning: Strategies for improving communication efficiency

    Jakub Konecn `y, H Brendan McMahan, Felix X Yu, Peter Richt ´arik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 8, 2016

    2016 arXiv

  20. [28]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. In Toronto, ON, Canada, 2009

    2009

  21. [29]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045–3059, 2021

    2021

  22. [30]

    Fedmd: Heterogenous federated learning via model distillation

    Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 , 2019

    1910 arXiv

  23. [31]

    Fedtp: Federated learning by transformer personalization

    Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi. Fedtp: Federated learning by transformer personalization. IEEE Transactions on neural networks and learning systems, 2023

    2023

  24. [32]

    Rouge: A package for automatic evaluation of sum- maries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of sum- maries. In Text summarization branches out , pages 74–81, 2004

    2004

  25. [33]

    No one left behind: Inclusive federated learning over heterogeneous devices

    Ruixuan Liu, Fangzhao Wu, Chuhan Wu, Yanlin Wang, Lingjuan Lyu, Hong Chen, and Xing Xie. No one left behind: Inclusive federated learning over heterogeneous devices. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3398–3406, 2022

    2022

  26. [34]

    Differentially private low-rank adaptation of large language model using federated learning

    Xiao-Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, Matt White, and Meikang Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Transactions on Management Information Systems , 2023

    2023

  27. [35]

    On surgical fine-tuning for language encoders

    Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, and Soundarara- jan Srinivasan. On surgical fine-tuning for language encoders. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages ...

    2023

  28. [36]

    A study of the attention abnormality in trojaned berts

    Weimin Lyu, Songzhu Zheng, Tengfei Ma, and Chao Chen. A study of the attention abnormality in trojaned berts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4727–4741, 2022

    2022

  29. [37]

    Attention-enhancing backdoor attacks against bert-based models

    Weimin Lyu, Songzhu Zheng, Lu Pang, Haibin Ling, and Chao Chen. Attention-enhancing backdoor attacks against bert-based models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10672–10690, 2023

    2023

  30. [38]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  31. [39]

    Asynchrony begets momentum, with an application to deep learning

    Ioannis Mitliagkas, Ce Zhang, Stefan Hadjis, and Christopher R ´e. Asynchrony begets momentum, with an application to deep learning. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 997–1004. IEEE, 2016

    2016

  32. [40]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27...

    2022

  33. [41]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. arXiv preprint arXiv:2403.17919 , 2024

    2024 arXiv

  34. [42]

    Sage- flow: Robust federated learning against both stragglers and adversaries

    Jungwuk Park, Dong-Jun Han, Minseok Choi, and Jaekyun Moon. Sage- flow: Robust federated learning against both stragglers and adversaries. Advances in neural information processing systems , 34:840–851, 2021

    2021

  35. [43]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision , pages 1406–1415, 2019

    2019

  36. [44]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021

    2021

  37. [45]

    Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients

    Shangchao Su, Bin Li, and Xiangyang Xue. Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients. In European Conference on Computer Vision , pages 342–358. Springer, 2025

    2025

  38. [46]

    Improving loRA in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations , 2024

    2024

  39. [47]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL...

    2018

  40. [48]

    {InfiniCache}: exploiting ephemeral serverless functions to build a {cost-effective} memory cache

    Ao Wang, Jingyuan Zhang, Xiaolong Ma, Ali Anwar, Lukas Rup- precht, Dimitrios Skourtis, Vasily Tarasov, Feng Yan, and Yue Cheng. {InfiniCache}: exploiting ephemeral serverless functions to build a {cost-effective} memory cache. In 18th USENIX conference on file and storage tec...

    2020

  41. [49]

    Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Sel- van Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Pro...

    2022

  42. [50]

    Flora: Federated fine-tuning large lan- guage models with heterogeneous low-rank adaptations

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large lan- guage models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976, 2024

    2024 arXiv

  43. [51]

    Application of computerized adaptive testing to educational problems

    David J Weiss and G Gage Kingsbury. Application of computerized adaptive testing to educational problems. Journal of educational mea- surement, 21(4):361–375, 1984

    1984

  44. [52]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 , 2023

    2023 arXiv

  45. [53]

    Fedbiot: Llm local fine-tuning in federated learning without full model

    Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3345–3355, 2024

    2024

  46. [54]

    Fast-convergent federated learning with adaptive weighting

    Hongda Wu and Ping Wang. Fast-convergent federated learning with adaptive weighting. IEEE Transactions on Cognitive Communications and Networking, 7(4):1078–1088, 2021

    2021

  47. [55]

    Node selection toward faster convergence for federated learning on non-iid data

    Hongda Wu and Ping Wang. Node selection toward faster convergence for federated learning on non-iid data. IEEE Transactions on Network Science and Engineering , 9(5):3099–3111, 2022

    2022

  48. [56]

    Fedfmsl: Federated learning of foundations models with sparsely activated lora

    Panlong Wu, Kangshuo Li, Ting Wang, Yanjie Dong, Victor CM Leung, and Fangxin Wang. Fedfmsl: Federated learning of foundations models with sparsely activated lora. IEEE Transactions on Mobile Computing , 2024

    2024

  49. [57]

    Fed2: Feature-aligned federated learning

    Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Fed2: Feature-aligned federated learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2066–2074, 2021

    2021

  50. [58]

    Federated foundation models: Privacy-preserving and collaborative learning for large models

    Sixing Yu, J Pablo Mu ˜noz, and Ali Jannesari. Federated foundation models: Privacy-preserving and collaborative learning for large models. arXiv preprint arXiv:2305.11414 , 2023

    2023 arXiv

  51. [59]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1–9, 2022

    2022

  52. [60]

    Fedcust: Offloading hyperparameter cus- tomization for federated learning

    Syed Zawad, Xiaolong Ma, Jun Yi, Cheng Li, Minjia Zhang, Lei Yang, Feng Yan, and Yuxiong He. Fedcust: Offloading hyperparameter cus- tomization for federated learning. Performance Evaluation, 167:102450, 2025

    2025

  53. [61]

    Visualizing and understanding con- volutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding con- volutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer, 2014

    2014

  54. [62]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIP...

    2023

  55. [63]

    Memory-adaptive depth-wise heterogenous federated learning

    Kai Zhang, Yutong Dai, Hongyi Wang, Eric Xing, Xun Chen, and Lichao Sun. Memory-adaptive depth-wise heterogenous federated learning. In Federated Learning Systems (FLSys) Workshop @ MLSys 2023 , 2023

    2023

  56. [64]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  57. [65]

    Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models

    Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational Linguistics 2023 , JOURNA...

    2023

  58. [66]

    Fed-pilot: Optimizing lora assignment for efficient federated foundation model fine-tuning

    Zikai Zhang, Jiahao Xu, Ping Liu, and Rui Hu. Fed-pilot: Optimizing lora assignment for efficient federated foundation model fine-tuning. arXiv preprint arXiv:2410.10200 , 2024

    2024 arXiv

  59. [67]

    Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning

    Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  60. [68]

    A comprehensive survey on pretrained foundation models: A history from bert to chatgpt

    Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023

    2023 arXiv

  61. [69]

    Autopeft: Automatic configuration search for parameter-efficient fine-tuning

    Han Zhou, Xingchen Wan, Ivan Vuli ´c, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. arXiv preprint arXiv:2301.12132, 2023

    2023 arXiv

  62. [70]

    Exact penalty method for federated learning

    Shenglong Zhou et al. Exact penalty method for federated learning. arXiv preprint arXiv:2208.11231 , 2022

    2022 arXiv

  63. [71]

    To prune, or not to prune: explor- ing the efficacy of pruning for model compression

    Michael Zhu and Suyog Gupta. To prune, or not to prune: explor- ing the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017

    2017 arXiv

  64. [72]

    When foundation model meets federated learning: Motivations, challenges, and future directions

    Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306.15546 , 2023

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.