Pith. sign in

REVIEW 5 major objections 5 minor 67 references

Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated PEFT can prune 90% of attention heads per client, cutting communication 1.8x and training compute 3.9x while staying within 2% of dense FedAvg accuracy.

desk verdict The integration is real, but the headline 3.9x training-OPs reduction is not supported by the paper's own algorithm—worth a serious look, but needs a major fix. read the letter →

arxiv 2506.00743 v1 pith:HQTZ3TUK submitted 2025-05-31 cs.CL cs.AIcs.DC

Yeshwanth Venkatesha , Souvik Kundu , Priyadarshini Panda This is my paper · ORCID
classification cs.CLcs.AIcs.DC
keywords federatedlearningparameter-efficientfine-tuningattentionheadpruningLoRAclientselectionnon-IIDdatalargelanguagemodelscommunicationefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that federated parameter-efficient fine-tuning (PEFT) of transformer language models can be made dramatically cheaper by treating attention heads as per-client assets rather than as a single global structure. The proposed recipe has three parts: each client prunes the 90% of attention heads it scores least confident, the server aggregates the surviving head updates with weights proportional to those confidence scores, and the server selects clients by the gap between their last local loss and the global model loss. If the recipe is right, resource-constrained devices can run federated fine-tuning with roughly 1.8x less communication and 3.9x fewer training operations than dense FedAvg while staying within 2% of its accuracy on the MultiNLI benchmark, and in several measured configurations the sparse model actually finishes ahead. The wider stakes are that PEFT methods, not just LoRA, could be federated efficiently whenever their trainable parameters can be assigned to attention heads.

What carries the argument

The machinery is a per-client, per-head importance score, $\alpha_{hc}=\frac{1}{|D_c|}\sum_{i\in D_c}\max(q_h^{(i)}\cdot k_h^{(i)})$, the average over the client's data of the maximum query-key dot product for head $h$. This score is the paper's proxy for a head's confidence and is used three ways: it decides which heads to prune (lowest scores set to zero), it weights each surviving head's update in the server's aggregation step, and it implicitly fingerprints the client's data distribution. The aggregation rule replaces uniform FedAvg averaging with an importance-weighted average per head, and the client-selection rule picks the clients whose last reported local loss is furthest above the global model's loss. All three share one design idea: the global model should listen most to the heads and clients that are most confident about their own data.

What would settle it

Take the paper's exact pipeline but replace the importance-based head masks with random masks at the same 90% sparsity, holding the loss-based client selection fixed, and compare final accuracy across MultiNLI, 20 Newsgroups, XL-Sum, and E2E NLG; the paper's central claim predicts importance masks should win by a clear margin, yet its own MultiNLI ablation already shows only a 0.07-point gap over random masks once client selection is held fixed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that head-level sparsity is compatible with federated aggregation, provided the right heads are kept and the right clients are heard. Using T5-small with LoRA on MultiNLI, the method reaches 90% head sparsity and measures a 1.8x communication advantage and a 3.9x reduction in training OPs relative to dense FedAvg, with accuracy dropping less than 2% and, in the 100-client/2-participant configuration, rising from 74.03% to 76.94%. The paper further claims the same machinery transfers to 20 Newsgroups, XL-Sum, and E2E NLG, to encoder-decoder, encoder-only, and decoder-only backbones, and to non-IID data with Dirichlet $\alpha=0.1$ without degradation. The mechanism is per-head importance pruning before transmission, importance-weighted aggregation on the server, and loss-gap client selection; the paper argues these three components complement each other, with pruning and weighting accelerating early convergence and client selection mattering in later rounds.

Load-bearing premise

The load-bearing premise is that a head's average maximum query-key attention score over a client's data reliably identifies which 90% of heads can be pruned without removing information the federated aggregation needs.

Editorial extensions

If this is right

  • At 90% sparsity only the LoRA $B$ matrices of surviving attention heads are transmitted, giving the paper's measured 1.8x communication reduction and 3.9x training-OP reduction on MultiNLI with T5-small.
  • On the same benchmark the sparse method reaches 76.94% accuracy in the 100/2 client setup against 74.03% for dense FedAvg with random client selection, so the claimed operating point is not just close to the dense baseline but sometimes above it.
  • Because the head-associated parameters are pruned before communication, any PEFT method whose trainable parameters map to attention heads can use the recipe; the paper demonstrates this on LoRA, full fine-tuning, IA3, prompt tuning, and p-tuning.
  • The viability boundary is sharp: accuracy holds up to about 90% sparsity and falls to random-guessing levels beyond 95%, so the practical gain is concentrated at the 90% operating point.
  • Importance-based pruning and weighted aggregation accelerate early-round convergence, while loss-based client selection provides later-round gains, so the three components target different phases of training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit is that the same per-head importance scores could be computed cheaply at server side from the transmitted update statistics, which would let the server re-mask or re-weight heads in later rounds without asking clients to recompute scores.
  • The paper's own ablation shows that with loss-based client selection the gap between importance-guided pruning and random pruning is only 0.07 percentage points on MultiNLI; read carefully, this suggests client selection may be carrying much of the accuracy benefit, a separation the paper does not quantify.
  • Because the score uses only one query-key vector pair per sample, a natural extension is to average over all token-pair attention weights or across layers, and to test whether the resulting masks track the functional head specialisation documented in the pruning literature; the paper leaves this untested.
  • The sharp collapse at 95% sparsity suggests a floor on the number of heads a task needs; an adaptive per-client sparsity schedule could be tested against the paper's fixed 90% threshold.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a federated PEFT pipeline for multi-head-attention language models that combines three components: (1) client-specific attention-head pruning guided by an importance score computed from attention activations, (2) head-specific weighted aggregation of updates, and (3) a client-selection strategy based on the loss gap between the global model and each client. Experiments on MultiNLI, 20 Newsgroups, XL-Sum, and E2E NLG with T5-small/BART and several PEFT methods report sparsity up to 90%, a 1.8x communication reduction, a 3.9x training-OPs reduction, and an accuracy drop under 2% relative to dense FedAvg with random client selection. Additional experiments study non-IID data, different backbone models, and different PEFT methods.

Significance. If the efficiency and accuracy claims were supported, the paper would contribute a practical combination of head pruning, weighted aggregation, and client selection for federated PEFT. The empirical scope is a clear strength: the evaluation spans four datasets, several transformer backbones, multiple PEFT methods, and non-IID settings, and the limitations section honestly notes the absence of a convergence analysis and real-device deployment. However, the central efficiency claim is not supported by the algorithm as currently described, and the main accuracy comparisons are made against a baseline that lacks the paper's own client-selection component. The underspecified importance-score formula and the absence of error bars or multiple seeds further reduce confidence. These issues are load-bearing and require revision.

major comments (5)
  1. [Section 3.1, Algorithm 1, Section 4.3, Appendix A] The claimed 3.9x training-OPs reduction is inconsistent with the method description. Section 3.1 states that the prunable set is P = {B}, i.e., only the LoRA B matrices, and Algorithm 1 performs local training (line 5) before computing importance scores (line 7) and then communicates the sparse parameters. Under this ordering, pruning cannot reduce local training computation, since the forward/backward pass over QKV, attention, and FC layers is performed in full. Yet Table 1 and Fig. 4 report OPs dropping from 33.07G to 8.47G, a 74% reduction, and the OPs accounting in Table A.6 counts QKV, attention, and FC operations in full. To support the 3.9x claim, the authors must either apply head masks before the local forward/backward pass and state this in Algorithm 1, or restrict the OPs claim to the pruned parameter set. As written, the abstract's central efficiency claim does not follow from the described method.
  2. [Section 3.1, Eq. (6)] The importance score in Eq. (6) is underspecified. Attention produces a matrix of query-key products over token positions, but the formula defines alpha_h^c as the average over data samples of max(q_h^(i) · k_h^(i)), with q and k appearing as single vectors per sample. It is not stated whether the max is over token positions, token pairs, or sequence positions, or how the q and k vectors are extracted. Since this score determines which 90% of heads are pruned, the experiments are not reproducible without a precise definition. The citation to [19] does not resolve the mismatch, because the prose describes 'maximum attention scores' while Eq. (6) uses a dot product of one query-key pair.
  3. [Table 2 and Section 4.2] The main results compare the full method (which includes loss-based client selection) against FedAvg with random client selection. This conflates the contribution of head pruning/weighted aggregation with the contribution of client selection. Table 1 shows that loss-based client selection alone improves FedAvg from 74.03% to 78.76% on MultiNLI, a 4.73% gain. Therefore, the favorable numbers in Table 2 could largely be due to client selection rather than to pruning or aggregation. The 'accuracy drop under 2%' claim should be evaluated against FedAvg with the same client-selection strategy for every dataset, not only for MultiNLI in Table 1.
  4. [Table 1 and general experimental setup] All reported accuracies come from a single run with no seeds, error bars, or statistical significance tests. The load-bearing comparison is especially fragile: with loss-based client selection, importance-based pruning achieves 76.94% versus random pruning's 76.87%, a gap of 0.07%. Without variance estimates, the paper's claim that importance-based pruning is better than random pruning, and the claim that the accuracy drop stays under 2%, are not statistically supported. The authors should report means and standard deviations over at least three seeds, or otherwise justify why single runs are sufficient.
  5. [Table 1 and Section 3.2] The paper lists head-specific weighted aggregation as a core contribution, but it is never isolated in the ablations. Table 1 varies head-pruning strategy (random vs. importance-based) and client-selection strategy, but it does not include a condition with importance pruning and client selection but without weighted aggregation, nor a condition with random pruning plus weighted aggregation. Consequently, the effect of Eq. (8) is not identified separately from the other two mechanisms. An ablation that turns the weighted-aggregation term on and off is needed to support the claim that this component 'mitigates the performance drop caused by sparsity.'
minor comments (5)
  1. [Section 2.1] The heading contains a typo: 'Muti-Head Attention' should be 'Multi-Head Attention.'
  2. [Algorithm 1, line 8] The line 'Communicates sparse parameters Pc and head importance scores αc' is not a grammatically complete imperative; it should be rephrased, for example, 'Communicate sparse parameters Pc and head importance scores αc to the server.'
  3. [Section 3.2, Eq. (8)] The value of the numerical-stability constant ε is never reported. Please provide the value used in the experiments, or state that it is a fixed constant and give its value.
  4. [Table 4] The text says the table reports 'maximum sparsity achieved before the model diverges,' but no sparsity column appears in the table. Either add the column or revise the caption and surrounding text to match the actual table contents.
  5. [Appendix A, Table A.6] In the LoRA backward row, the term 'T d3' appears in the OPs expression. LoRA backward computations involve rank-r matrices and should scale with r, not with d^3; this term looks like an error and may inflate the OPs totals. Please verify and correct the formula.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the importance-score pruning, weighted aggregation, and client selection are empirical heuristics evaluated on held-out data, not predictions that reduce to their own inputs by construction.

full rationale

This is an empirical engineering paper rather than a derivation, and no load-bearing step reduces to its own input. The head importance score (Eq. 6) is computed from local attention query-key products over client data; it is a heuristic fingerprint used to prune heads (Eq. 7) and to weight updates (Eq. 8). It is not defined in terms of the target accuracy or the final reported results, and the validation accuracy is measured on held-out data, so the evaluation is not self-validating. The 90% sparsity level is selected from a sweep over sparsity values (Fig. 4), which is a hyperparameter choice, not a fitted parameter renamed as a prediction. The communication reduction is an accounting consequence of not transmitting pruned B matrices, and the accuracy comparisons are against standard FedAvg baselines on public benchmarks. The one self-citation involving a co-author ([18]) is contextual background on federated learning and is not used to justify the method's core claims. A separate concern is that the claimed 3.9x OPs reduction appears inconsistent with the method description, because Algorithm 1 computes importance scores only after full local training and Section 3.1 restricts prunable parameters to P = {B}; however, that is an internal consistency or correctness issue, not circularity, and no equation in the paper reduces a claimed result to its own input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three engineering assumptions rather than on derived mathematics: that LoRA B matrices are head-separable, that attention confidence scores are a reliable pruning signal under non-IID data, and that loss gaps are comparable across clients. The 90% sparsity level is chosen from a validation sweep (Fig. 4), making it a free parameter; epsilon in Eq. 8 is an unreported stability constant. No new theoretical entities are introduced.

free parameters (2)
  • Sparsity threshold (90% of attention heads) = 90% of heads pruned
    Chosen from the validation sweep in Fig. 4 to keep the accuracy drop under 2% while maximizing communication and compute savings. This is the main free knob that produces the reported 1.8x and 3.9x gains.
  • Epsilon in weighted aggregation (Eq. 8) = not reported
    A small constant added for numerical stability. Its value is not given, but it should not materially affect results.
assumptions (4)
  • domain assumption LoRA B matrices can be split per attention head and pruned independently while A and the task head remain shared
    Section 3.1 states P = {B}. The paper assumes that removing the B matrix of a pruned head removes that head's LoRA contribution without corrupting the shared A. This is load-bearing for both the communication and OPs claims.
  • domain assumption Average maximum attention score is a valid importance measure for pruning in federated settings
    Eq. 6 uses max(q_h dot k_h) per sample as a proxy for head confidence, borrowing from [19]. The paper does not validate this measure under heterogeneous client data or specify exactly which positions/layers are used. Both pruning and weighted aggregation depend on this score.
  • domain assumption Client losses from different clients are comparable and their gap to the global loss is a useful selection signal
    Eq. 9 ranks clients by (local loss minus global loss). This assumes loss scales and client data distributions are comparable enough that the difference reflects client divergence rather than noise or scaling artifacts.
  • standard math Standard attention, LoRA, and FedAvg update equations are correct and carry over unchanged
    Equations 1 through 8 use standard transformer attention, softmax, LoRA low-rank decomposition, and federated averaging. No new mathematical machinery is introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection." pith.science (2026). https://pith.science/paper/HQTZ3TUK

@misc{pith2026250600743,
  author       = {Pith},
  title        = {Pith review of: Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HQTZ3TUK}},
  note         = {Machine review of arXiv:2506.00743}
}
read the original abstract

Parameter Efficient Fine-Tuning (PEFT) has become the de-facto approach in adapting Large Language Models (LLMs) for downstream tasks in Natural Language Processing. However, its adoption in privacy-preserving distributed learning frameworks, such as Federated Learning (FL), remains relatively limited. This is mainly due to challenges specific to FL, such as resource-constrained devices and diverse data distributions among clients. In this paper, we propose an efficient method to perform PEFT within the FL framework for Multi-Head Attention (MHA) based language models. We address the challenges through head pruning, a novel head-specific weighted aggregation mechanism, and a client selection strategy. Head pruning minimizes training complexity within the clients, guided by the importance score computed based on the confidence of the attention head. Weighted aggregation of heads ensures the global model captures crucial updates from diverse clients complementing our client selection strategy. We show results on the MultiNLI benchmark along with 20 Newsgroups, XL-Sum, and E2E NLG datasets. We use the MultiNLI dataset and T5-small model with LoRA as our PEFT method, attaining sparsity levels of up to 90%, resulting in a communication advantage of up to 1.8x and a reduction in training OPs of 3.9x while maintaining the accuracy drop under 2%.

Figures

Figures reproduced from arXiv: 2506.00743 by the authors.

Figure 1
Figure 1. Within a federated environment, devices often face resource limitations and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of our strategy encompassing attention head pruning guided by [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Training curve over the first 30 rounds illustrating the impact of importance [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effect of pruning on accuracy, communication cost and OPs. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Assessment of head importance scores in the context of IID vs Non-IID scenarios. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 34 canonical work pages

  1. [19]

    Voita, D

    E. Voita, D. Talbot, F. Moiseev, R. Sennrich, I. Titov, Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned, in: A. Korhonen, D. Traum, L. M` arquez (Eds.), Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, ...

    doi:10.18653/v1/p19-1580 2019

  2. [1]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901

    2020

  3. [2]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Research 21 (1) (2020) 5485–5551

    work page 2020

  4. [3]

    Lewis, Y

    M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019)

    arXiv 2019

  5. [4]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018)

    arXiv 2018

  6. [5]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). 19

    work page 2017

  7. [6]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020)

    arXiv 2020

  8. [7]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, High- resolution image synthesis with latent diffusion models, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695

    2022

Show all 67 references
  1. [8]

    Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, M. Zhou, Layoutlm: Pre-training of text and layout for document image understanding, in: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 1192–1200

    2020

  2. [9]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, Robust speech recognition via large-scale weak supervision, in: Interna- tional Conference on Machine Learning, PMLR, 2023, pp. 28492–28518

    2023

  3. [10]

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, et al., Xls-r: Self-supervised cross-lingual speech representation learning at scale, arXiv preprint arXiv:2111.09296 (2021)

    2021 arXiv

  4. [11]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  5. [12]

    X. L. Li, P. Liang, Prefix-tuning: Optimizing continuous prompts for generation, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Natural Lang...

    2021 doi

  6. [13]

    X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, J. Tang, P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, arXiv preprint arXiv:2110.07602 (2021). 20

    2021 arXiv

  7. [14]

    McMahan, E

    B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, Communication-efficient learning of deep networks from decentralized data, in: Artificial intelligence and statistics, PMLR, 2017, pp. 1273– 1282

    2017

  8. [15]

    A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augen- stein, H. Eichner, C. Kiddon, D. Ramage, Federated learning for mobile keyboard prediction, arXiv preprint arXiv:1811.03604 (2018)

    2018 arXiv

  9. [16]

    M. Chen, A. T. Suresh, R. Mathews, A. Wong, C. Allauzen, F. Beaufays, M. Riley, Federated learning of n-gram language models, arXiv preprint arXiv:1910.03432 (2019)

    2019 arXiv

  10. [17]

    M. Liu, S. Ho, M. Wang, L. Gao, Y. Jin, H. Zhang, Federated learning meets natural language processing: A survey, arXiv preprint arXiv:2107.12603 (2021)

    2021 arXiv

  11. [18]

    Babakniya, S

    S. Babakniya, S. Kundu, S. Prakash, Y. Niu, S. Avestimehr, Revisiting sparsity hunting in federated learning: Why does sparsity consensus matter?, Transactions on Machine Learning Research (2023)

    2023

  12. [20]

    J. Li, R. Cotterell, M. Sachan, Differentiable subset pruning of trans- former heads, Transactions of the Association for Computational Lin- guistics 9 (2021) 1442–1459

    2021

  13. [21]

    Behnke, K

    M. Behnke, K. Heafield, Losing heads in the lottery: Pruning trans- former attention in neural machine translation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 2664–2674

    2020

  14. [22]

    K. Shim, I. Choi, W. Sung, J. Choi, Layer-wise pruning of transformer at- tention heads for efficient language modeling, in: 2021 18th International SoC Design Conference (ISOCC), IEEE, 2021, pp. 357–358. 21

    2021

  15. [23]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, S. Gelly, Parameter-efficient transfer learn- ing for nlp, in: International Conference on Machine Learning, PMLR, 2019, pp. 2790–2799

    2019

  16. [24]

    Zhang, M

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, T. Zhao, Adaptive budget allocation for parameter-efficient fine-tuning, arXiv preprint arXiv:2303.10512 (2023)

    2023 arXiv

  17. [25]

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, C. A. Raffel, Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, Advances in Neural Information Processing Systems 35 (2022) 1950–1965

    2022

  18. [26]

    E. B. Zaken, S. Ravfogel, Y. Goldberg, Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models, arXiv preprint arXiv:2106.10199 (2021)

    2021

  19. [27]

    Y.-L. Sung, J. Cho, M. Bansal, Lst: Ladder side-tuning for parameter and memory efficient transfer learning, Advances in Neural Information Processing Systems 35 (2022) 12991–13005

    2022

  20. [28]

    X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, J. Tang, Gpt understands, too, AI Open (2023)

    2023

  21. [29]

    Lester, R

    B. Lester, R. Al-Rfou, N. Constant, The power of scale for parameter- efficient prompt tuning, arXiv preprint arXiv:2104.08691 (2021)

    2021 arXiv

  22. [30]

    Hilmkil, S

    A. Hilmkil, S. Callh, M. Barbieri, L. R. S¨ utfeld, E. L. Zec, O. Mo- gren, Scaling federated learning for fine-tuning of large language models, in: International Conference on Applications of Natural Language to Information Systems, Springer, 2021, pp. 15–23

    2021

  23. [31]

    B. Y. Lin, C. He, Z. Zeng, H. Wang, Y. Huang, C. Dupuy, R. Gupta, M. Soltanolkotabi, X. Ren, S. Avestimehr, Fednlp: Benchmarking fed- erated learning methods for natural language processing tasks, arXiv preprint arXiv:2104.08815 (2021)

    2021 arXiv

  24. [32]

    H. Li, Z. Cai, J. Wang, J. Tang, W. Ding, C.-T. Lin, Y. Shi, Fedtp: Federated learning by transformer personalization, IEEE Transactions on Neural Networks and Learning Systems (2023). 22

    2023

  25. [33]

    G. Sun, M. Mendieta, J. Luo, S. Wu, C. Chen, Fedperfix: Towards partial model personalization of vision transformers in federated learning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4988–4998

    2023

  26. [34]

    Y. Kim, J. Kim, W.-L. Mok, J.-H. Park, S. Lee, Client-customized adaptation for parameter-efficient federated learning, in: Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 1159–1172

    2023

  27. [35]

    L. Qu, Y. Zhou, P. P. Liang, Y. Xia, F. Wang, E. Adeli, L. Fei-Fei, D. Rubin, Rethinking architecture design for tackling data heterogeneity in federated learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10061–10071

    2022

  28. [36]

    J. Chen, W. Xu, S. Guo, J. Wang, J. Zhang, H. Wang, Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers, arXiv preprint arXiv:2211.08025 (2022)

    2022 arXiv

  29. [37]

    X. Zuo, Q. Zhang, R. Han, An empirical analysis of vision transformer and cnn in resource-constrained federated learning, in: Proceedings of the 2022 5th International Conference on Machine Learning and Machine Intelligence, 2022, pp. 8–13

    2022

  30. [38]

    Hyeon-Woo, M

    N. Hyeon-Woo, M. Ye-Bin, T.-H. Oh, Fedpara: Low-rank hadamard product for communication-efficient federated learning, arXiv preprint arXiv:2108.06098 (2021)

    2021 arXiv

  31. [39]

    Babakniya, A

    S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-B. Song, M. El- Khamy, S. Avestimehr, Slora: Federated parameter efficient fine-tuning of language models, arXiv preprint arXiv:2308.06522 (2023)

    2023 arXiv

  32. [40]

    H. Zhao, W. Du, F. Li, P. Li, G. Liu, Reduce communication costs and preserve privacy: Prompt tuning method in federated learning, arXiv preprint arXiv:2208.12268 (2022)

    2022 arXiv

  33. [41]

    Zhuang, C

    W. Zhuang, C. Chen, L. Lyu, When foundation model meets federated learning: Motivations, challenges, and future directions, arXiv preprint arXiv:2306.15546 (2023). 23

    2023 arXiv

  34. [42]

    Zhang, Y

    Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, Z. Xu, Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models, in: Annual Meeting of the Associa- tion of Computational Linguistics 2023, Association for Computational L...

    2023

  35. [43]

    G. Sun, M. Mendieta, T. Yang, C. Chen, Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning, arXiv preprint arXiv:2210.01708 (2022)

    2022 arXiv

  36. [44]

    Sietsma, Dow, Neural net pruning-why and how, in: IEEE 1988 interna- tional conference on neural networks, IEEE, 1988, pp. 325–333

    1988

  37. [45]

    LeCun, J

    Y. LeCun, J. Denker, S. Solla, Optimal brain damage, Advances in neural information processing systems 2 (1989)

    1989

  38. [46]

    S. Han, H. Mao, W. J. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510.00149 (2015)

    2015 arXiv

  39. [47]

    Blalock, J

    D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, J. Guttag, What is the state of neural network pruning?, Proceedings of machine learning and systems 2 (2020) 129–146

    2020

  40. [48]

    J. Mao, H. Yang, A. Li, H. Li, Y. Chen, Tprune: Efficient transformer pruning for mobile devices, ACM Transactions on Cyber-Physical Sys- tems 5 (3) (2021) 1–22

    2021

  41. [49]

    Lagunas, E

    F. Lagunas, E. Charlaix, V. Sanh, A. M. Rush, Block pruning for faster transformers, arXiv preprint arXiv:2109.04838 (2021)

    2021 arXiv

  42. [50]

    W. Held, D. Yang, Shapley head pruning: Identifying and removing interference in multilingual transformers, arXiv preprint arXiv:2210.05709 (2022)

    2022 arXiv

  43. [51]

    H. Wang, Z. Zhang, S. Han, Spatten: Efficient sparse attention architec- ture with cascade token and head pruning, in: 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), IEEE, 2021, pp. 97–110. 24

    2021

  44. [52]

    M. A. Gordon, K. Duh, N. Andrews, Compressing bert: Studying the effects of weight pruning on transfer learning, arXiv preprint arXiv:2002.08307 (2020)

    2020 arXiv

  45. [53]

    Y. J. Cho, J. Wang, G. Joshi, Towards understanding biased client selection in federated learning, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2022, pp. 10351–10375

    2022

  46. [54]

    Zhang, X

    W. Zhang, X. Wang, P. Zhou, W. Wu, X. Zhang, Client selection for federated learning with non-iid data in mobile edge computing, IEEE Access 9 (2021) 24462–24474

    2021

  47. [55]

    L. Fu, H. Zhang, G. Gao, M. Zhang, X. Liu, Client selection in federated learning: Principles, challenges, and opportunities, IEEE Internet of Things Journal (2023)

    2023

  48. [56]

    Nishio, R

    T. Nishio, R. Yonetani, Client selection for federated learning with hetero- geneous resources in mobile edge, in: ICC 2019-2019 IEEE international conference on communications (ICC), IEEE, 2019, pp. 1–7

    2019

  49. [57]

    J. Xu, H. Wang, Client selection and bandwidth allocation in wireless federated learning networks: A long-term perspective, IEEE Transactions on Wireless Communications 20 (2) (2020) 1188–1200

    2020

  50. [58]

    AbdulRahman, H

    S. AbdulRahman, H. Tout, A. Mourad, C. Talhi, Fedmccs: Multicriteria client selection model for optimal iot federated learning, IEEE Internet of Things Journal 8 (6) (2020) 4723–4735

    2020

  51. [59]

    Huang, W

    T. Huang, W. Lin, W. Wu, L. He, K. Li, A. Y. Zomaya, An efficiency- boosting client selection scheme for federated learning with fairness guarantee, IEEE Transactions on Parallel and Distributed Systems 32 (7) (2020) 1552–1564

    2020

  52. [60]

    Williams, N

    A. Williams, N. Nangia, S. Bowman, A broad-coverage challenge corpus for sentence understanding through inference, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long...

    2018

  53. [61]

    Lang, Newsweeder: Learning to filter netnews, in: A

    K. Lang, Newsweeder: Learning to filter netnews, in: A. Prieditis, S. Rus- sell (Eds.), Machine Learning Proceedings 1995, Morgan Kaufmann, San Francisco (CA), 1995, pp. 331–339. doi:https://doi.org/10.1016/B978-1- 55860-377-6.50048-7

    1995 doi

  54. [62]

    Hasan, A

    T. Hasan, A. Bhattacharjee, M. S. Islam, K. Mubasshir, Y.-F. Li, Y.-B. Kang, M. S. Rahman, R. Shahriyar, XL-sum: Large-scale multilingual abstractive summarization for 44 languages, in: Findings of the Associa- tion for Computational Linguistics: ACL-IJCNLP 2021, Association f...

    2021

  55. [63]

    Novikova, O

    J. Novikova, O. Duˇ sek, V. Rieser, The e2e dataset: New challenges for end-to-end generation, arXiv preprint arXiv:1706.09254 (2017)

    2017 arXiv

  56. [64]

    X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, J. Tang, P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 61–68

    2022

  57. [65]

    V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  58. [66]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pre- training approach, arXiv preprint arXiv:1907.11692 (2019)

    2019 arXiv

  59. [67]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (8) (2019) 9. Appendix A. MAC Operations The layerwise operations and their corresponding OPs count are detailed in Table A.6. The main com...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.