arxiv: 2604.21197 · v1 · submitted 2026-04-23 · 💻 cs.LG

Recognition: unknown

Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach

Guilin Deng , Silong Chen , Yuchuan Luo , Yi Liu , Songlei Wang , Zhiping Cai , Lin Liu , Xiaohua Jia

show 1 more author

Shaojing Fu

Authors on Pith no claims yet

Pith reviewed 2026-05-09 22:08 UTC · model grok-4.3

classification 💻 cs.LG

keywords membership inference attackfederated learninglarge language modelsprojection residualgradient subspaceprivacy vulnerabilitydifferential privacy

0 comments

The pith

Projection residuals of hidden embeddings onto gradient subspaces enable near-perfect membership inference on federated LLMs without auxiliary models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that in federated LLM training, where parties share gradients but keep data local, a passive attacker can still determine whether a given input was part of any party's training set. It does so by representing each sample via its hidden embedding vector and measuring how much of that vector lies outside the subspace spanned by the observed gradients. The resulting residual distance turns out to be systematically smaller for members than for non-members. Because the method needs only the current gradient and the target embedding, it avoids shadow models, historical updates, or extra data and stays effective even when differential privacy noise is added to the gradients.

Core claim

The projection residual of a sample's hidden embedding vector onto the gradient subspace directly encodes membership information in FedLLM updates; computing these residuals yields a simple threshold classifier that reaches near 100 percent accuracy across four benchmarks and four model families while outperforming prior MIAs by up to 75.75 percent and resisting strong differential privacy.

What carries the argument

The projection residual of a hidden embedding vector onto the low-dimensional subspace spanned by the federated gradient updates.

If this is right

Shared gradients in federated LLM training leak membership far more than previously assumed, even when models converge quickly and gradients are sparse.
Existing shadow-model and classifier-based MIAs become unnecessary for these settings; a direct geometric test suffices.
Differential privacy applied at typical noise levels does not eliminate the residual signal, so stronger or differently targeted defenses are required.
Security analyses of federated learning must now account for embedding-to-gradient projection links rather than treating gradients as opaque aggregates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same residual test may expose membership in other large-scale distributed training regimes where embeddings are computed locally but gradients are exchanged.
If residuals remain informative after convergence, periodic re-computation of the gradient subspace during training could serve as an ongoing privacy monitor.
Defenses might need to perturb embeddings themselves or deliberately orthogonalize them to gradients rather than adding noise only to parameter updates.

Load-bearing premise

That the magnitude of the residual after projecting an embedding onto the gradient subspace is consistently and detectably smaller for training inputs than for non-training inputs, even after rapid convergence and under privacy noise.

What would settle it

Running the ProjRes procedure on a federated LLM where the training set is known and measuring whether membership classification accuracy falls to the random-guess level of 50 percent.

Figures

Figures reproduced from arXiv: 2604.21197 by Guilin Deng, Lin Liu, Shaojing Fu, Silong Chen, Songlei Wang, Xiaohua Jia, Yi Liu, Yuchuan Luo, Zhiping Cai.

**Figure 1.** Figure 1: Overview of the feasibility study results. (a) A glance at the complexity of traditional Computer Vision (CV) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: Overview of the proposed ProjRes attack. Using the adapter as the trainable module, an honest-but-curious server conducts the ProjRes attack as follows: ① Perform a forward pass on a target sample x to obtain its hidden embedding f(x) at the adapter layer, where f(·) denotes the model segment preceding the adapter. ② Construct a linear subspace S spanned by the adapter downsampling-layer gradients uploaded… view at source ↗

**Figure 3.** Figure 3: Comparison of hidden embedding vector proper [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Comparison of attack performance across different models and datasets using ROC curves. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

**Figure 6.** Figure 6: The AUC scores of ProjRes against BERT-Base with a single adapter inserted at different Transformer layers on CoLA. 5 10 20 50 100 200 Number of clients 0.2 0.4 0.6 0.8 1.0 AUC Ours Fed-loss Cosine Gradient-diff Score-Diff Score-Ratio FTA FedMIA [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

**Figure 7.** Figure 7: AUC scores of MIAs against Adapter-based [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

**Figure 8.** Figure 8: An overview of three FedLLM fine-tuning strategies. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

**Figure 11.** Figure 11: The AUC scores of ProjRes against BERTBase with a single adapter inserted at different Transformer layers. TABLE 12: AUC scores of ProjRes against Adapter-based BERT-Base under various defenses on CoLA after 1 epoch. Defense Value Batch Size 1 2 4 8 16 DP (σ) 0.01 1.000 1.000 1.000 0.999 0.848 0.1 1.000 0.911 0.663 0.565 0.537 1 0.495 0.502 0.514 0.491 0.510 1.5 0.520 0.519 0.499 0.507 0.517 GP (β) 70% 1… view at source ↗

**Figure 10.** Figure 10: AUC scores of MIAs agaist LoRA-based BERT [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗

read the original abstract

Federated Large Language Models (FedLLMs) enable multiple parties to collaboratively fine-tune LLMs without sharing raw data, addressing challenges of limited resources and privacy concerns. Despite data localization, shared gradients can still expose sensitive information through membership inference attacks (MIAs). However, FedLLMs' unique properties, i.e. massive parameter scales, rapid convergence, and sparse, non-orthogonal gradients, render existing MIAs ineffective. To address this gap, we propose ProjRes, the first projection residuals-based passive MIA tailored for FedLLMs. ProjRes leverages hidden embedding vectors as sample representations and analyzes their projection residuals on the gradient subspace to uncover the intrinsic link between gradients and inputs. It requires no shadow models, auxiliary classifiers, or historical updates, ensuring efficiency and robustness. Experiments on four benchmarks and four LLMs show that ProjRes achieves near 100% accuracy, outperforming prior methods by up to 75.75%, and remains effective even under strong differential privacy defenses. Our findings reveal a previously overlooked privacy vulnerability in FedLLMs and call for a re-examination of their security assumptions. Our code and data are available at $\href{https://anonymous.4open.science/r/Passive-MIA-5268}{link}$.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ProjRes gives a practical passive MIA for federated LLMs that hits high accuracy without shadow models, but the core claim rests on an unproven link between projection residuals and membership.

read the letter

The main point is that this paper introduces ProjRes, a projection-residual attack on federated LLM gradients that works passively and claims near-100% accuracy across four benchmarks and four models while staying effective under differential privacy. It avoids the usual overhead of shadow models or historical updates, which matters for the scale and convergence speed of these systems. The approach uses hidden embeddings projected onto the gradient subspace to extract a membership signal, and the authors release code, which helps with checking the results. That combination of tailoring to sparse non-orthogonal gradients and strong reported numbers is the clearest advance over earlier MIAs that break down in this regime. The experiments appear to cover reasonable ground for an initial study. The soft spot is the missing explanation for why the residuals should separate members from non-members. The abstract talks about an intrinsic link, but there is no derivation, even simplified, showing how a sample's contribution to the gradient produces a distinguishable residual once projected. In high-dimensional sparse LLM gradients this step is not obvious, and without it the near-perfect results could be tied to the specific benchmarks or tuning rather than a reliable property. No ablations or statistical tests are summarized to rule that out. This work is for people studying privacy attacks in collaborative LLM training. A reader who needs concrete attack ideas or baselines for FedLLM defenses would get something usable from the method and the numbers. It deserves peer review because the setting is timely and the empirical performance is sharp enough to test, even though the theoretical gap needs addressing in revision.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes ProjRes, a passive membership inference attack for federated large language models that projects hidden embedding vectors onto the gradient subspace and uses the resulting residuals to infer sample membership. The approach requires no shadow models, auxiliary classifiers, or historical updates. Experiments across four benchmarks and four LLMs are reported to yield near-100% accuracy, outperforming prior MIAs by up to 75.75% while remaining effective under differential privacy.

Significance. If the reported results hold under scrutiny, the work identifies a concrete and efficient privacy leakage channel in FedLLMs that exploits the geometry between gradients and embeddings. The absence of auxiliary models and the public release of code and data are positive features that facilitate verification and extension. The findings challenge prevailing assumptions about gradient privacy in federated LLM training and could motivate new defense research.

major comments (2)

[Method (description of ProjRes and projection residual)] The central claim rests on the assertion of an 'intrinsic link' between gradients and input membership that is revealed by projection residuals. No derivation, even for a simplified low-dimensional case, is supplied to show why the residual after projection onto the gradient subspace systematically separates members from non-members, especially given the sparse, non-orthogonal, high-dimensional character of LLM gradients and the separate embedding space. This justification is load-bearing for interpreting the near-100% accuracies as evidence of a reliable geometric property rather than benchmark-specific behavior.
[Experimental evaluation and results] The experimental claims of near-100% accuracy and large margins over baselines are presented without reported statistical tests across random seeds, ablation studies on the precise definition or dimensionality of the gradient subspace, or controls for possible post-hoc selection of the four benchmarks and four LLMs. These omissions make it difficult to assess whether the performance generalizes or could be reproduced by an independent party.

minor comments (1)

[Abstract] The abstract contains a minor grammatical issue: 'i.e. massive parameter scales' should read 'i.e., massive parameter scales'.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback on our manuscript. The comments highlight important areas for strengthening the theoretical motivation and experimental rigor of ProjRes. We address each major comment point by point below, outlining the specific revisions we will incorporate in the next version of the paper.

read point-by-point responses

Referee: [Method (description of ProjRes and projection residual)] The central claim rests on the assertion of an 'intrinsic link' between gradients and input membership that is revealed by projection residuals. No derivation, even for a simplified low-dimensional case, is supplied to show why the residual after projection onto the gradient subspace systematically separates members from non-members, especially given the sparse, non-orthogonal, high-dimensional character of LLM gradients and the separate embedding space. This justification is load-bearing for interpreting the near-100% accuracies as evidence of a reliable geometric property rather than benchmark-specific behavior.

Authors: We agree that a formal derivation would strengthen the interpretation of the results. While the current manuscript motivates ProjRes through the geometric relationship between gradients (computed from embeddings) and membership, it relies primarily on empirical evidence. In the revised version, we will add a new subsection providing a simplified low-dimensional derivation. This will consider a linear model where gradients lie in a low-rank subspace spanned by member embeddings; we will show analytically that the projection residual norm is systematically smaller for members than non-members due to the alignment between the input and the gradient direction. We will then discuss how this intuition extends (with caveats) to the sparse, high-dimensional LLM setting, including why non-orthogonality does not invalidate the separation in practice. This addition will make the geometric property more explicit and address concerns about benchmark-specific behavior. revision: yes
Referee: [Experimental evaluation and results] The experimental claims of near-100% accuracy and large margins over baselines are presented without reported statistical tests across random seeds, ablation studies on the precise definition or dimensionality of the gradient subspace, or controls for possible post-hoc selection of the four benchmarks and four LLMs. These omissions make it difficult to assess whether the performance generalizes or could be reproduced by an independent party.

Authors: We acknowledge these omissions limit the ability to assess robustness and reproducibility. In the revision, we will add: (1) results averaged over 5 independent random seeds with standard deviations and t-test p-values comparing ProjRes to baselines; (2) ablation studies varying the gradient subspace dimensionality (e.g., using top-k singular vectors for k from 10 to 1000) and alternative subspace definitions (e.g., random vs. PCA-based); (3) explicit justification for benchmark and model selection (standard GLUE/SuperGLUE tasks and representative LLMs like LLaMA-7B, GPT-2, etc., chosen for diversity in scale and architecture) along with results on two additional held-out datasets to demonstrate generalization. These changes will be presented in expanded experimental sections and appendices, with code already public to support independent verification. revision: yes

Circularity Check

0 steps flagged

No circularity; method geometrically defined with empirical results

full rationale

The paper defines ProjRes directly via projection of hidden embedding vectors onto the gradient subspace to expose an intrinsic link, with no equations or steps that reduce the claimed attack accuracy to a fitted parameter, self-referential quantity, or input by construction. Effectiveness is presented as an experimental outcome on four benchmarks and four LLMs rather than a derived tautology. No load-bearing self-citations, uniqueness theorems, or smuggled ansatzes appear in the abstract or description for the core claim. The approach remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach rests on the domain assumption that hidden embeddings capture sufficient information about gradient-input relationships; no free parameters, new entities, or ad-hoc axioms are introduced in the abstract.

axioms (1)

domain assumption Hidden embedding vectors serve as effective sample representations whose projection residuals onto the gradient subspace reveal membership information.
Central to the ProjRes method as described.

pith-pipeline@v0.9.0 · 5547 in / 1206 out tokens · 31548 ms · 2026-05-09T22:08:25.438927+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

65 extracted references · 9 canonical work pages · 2 internal anchors

[1]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019
[2]

Bert: Pre- training of deep bidirectional transformers for language understand- ing,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language understand- ing,” inProc. of NAACL, 2019

2019
[3]

The llama 3 herd of models,

A. Dubey, Jauhriet al., “The llama 3 herd of models,” 2024

2024
[4]

Qwen2.5-Omni Technical Report

J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Danget al., “Qwen2. 5-omni technical report,”arXiv preprint arXiv:2503.20215, 2025

work page internal anchor Pith review arXiv 2025
[5]

Feddat: An approach for foundation model finetuning in multi-modal heteroge- neous federated learning,

H. Chen, Y . Zhang, D. Krompass, J. Gu, and V . Tresp, “Feddat: An approach for foundation model finetuning in multi-modal heteroge- neous federated learning,” inProc. of AAAI, 2024

2024
[6]

Efficient federated learning for modern nlp,

D. Cai, Y . Wu, S. Wang, F. X. Lin, and M. Xu, “Efficient federated learning for modern nlp,” inProc. of MobiCom, 2023

2023
[7]

Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,

W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y . Xie, Y . Li, B. Ding, and J. Zhou, “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” inProc. of KDD, 2024

2024
[8]

The impact of gdpr on global technology development,

H. Li, L. Yu, and W. He, “The impact of gdpr on global technology development,”Journal of Global Information Technology Manage- ment, vol. 22, no. 1, pp. 1–6, 2019

2019
[9]

Openfedllm: Training large language models on decentral- ized private data via federated learning,

R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y . Xu, Y . Du, Y . Wang, and S. Chen, “Openfedllm: Training large language models on decentral- ized private data via federated learning,” inProc. of KDD, 2024

2024
[10]

The future of large language model pre-training is federated,

L. Sani, A. Iacob, Z. Cao, B. Marino, Y . Gao, T. Paulik, W. Zhao, W. F. Shen, P. Aleksandrov, X. Qiuet al., “The future of large language model pre-training is federated,”arXiv preprint arXiv:2405.10853, 2024

work page arXiv 2024
[11]

Safely learning with private data: A federated learning framework for large language model,

J. Zheng, H. Zhang, L. Wang, W. Qiu, H. Zheng, and Z. Zheng, “Safely learning with private data: A federated learning framework for large language model,” inProc. of EMNLP, 2024

2024
[12]

Dual-personalizing adapter for federated foundation models,

G. Long, T. Shen, J. Jiang, M. Blumensteinet al., “Dual-personalizing adapter for federated foundation models,” inProc. of NeurIPS, 2024

2024
[13]

Adapter- fusion: Non-destructive task composition for transfer learning,

J. Pfeiffer, A. Kamath, A. R ¨uckl´e, K. Cho, and I. Gurevych, “Adapter- fusion: Non-destructive task composition for transfer learning,” in Proc. of EACL, 2021

2021
[14]

Dp-dylora: Fine-tuning transformer-based models on-device under differentially private federated learning using dynamic low-rank adaptation,

J. Xu, K. Saravanan, R. van Dalen, H. Mehmood, D. Tuckey, and M. Ozay, “Dp-dylora: Fine-tuning transformer-based models on- device under differentially private federated learning using dynamic low-rank adaptation,”arXiv preprint arXiv:2405.06368, 2024

work page arXiv 2024
[15]

Memetic federated learning for biomedical natural language processing,

X. Zhou, C. Tan, D. Jiang, B. Zhang, S. Li, Y . Xu, Q. Xu, and S. Gao, “Memetic federated learning for biomedical natural language processing,” inCCF International Conference on Natural Language Processing and Chinese Computing. Springer, 2021, pp. 43–55

2021
[16]

An in-depth evaluation of federated learning on biomedical natural lan- guage processing for information extraction,

L. Peng, G. Luo, S. Zhou, J. Chen, Z. Xu, J. Sun, and R. Zhang, “An in-depth evaluation of federated learning on biomedical natural lan- guage processing for information extraction,”NPJ Digital Medicine, vol. 7, no. 1, p. 127, 2024

2024
[17]

Fedlegal: The first real-world federated learning benchmark for legal nlp,

Z. Zhang, X. Hu, J. Zhang, Y . Zhang, H. Wang, L. Qu, and Z. Xu, “Fedlegal: The first real-world federated learning benchmark for legal nlp,” inProc. of ACL, 2023

2023
[18]

Prompt federated learning for weather forecasting: Toward foundation models on meteorological data,

S. Chen, G. Long, T. Shen, and J. Jiang, “Prompt federated learning for weather forecasting: Toward foundation models on meteorological data,” inProc. of IJCAI, 2023

2023
[19]

Flexible and secure code deployment in federated learning using large language models: Prompt engineer- ing to enhance malicious code detection,

J. Seo, N. Zhang, and C. Rong, “Flexible and secure code deployment in federated learning using large language models: Prompt engineer- ing to enhance malicious code detection,” inProc. of CloudCom, 2023

2023
[20]

F- codellm: A federated learning framework for adapting large language models to practical software development,

Z. Cai, J. Chen, W. Chen, W. Wang, X. Zhu, and A. Ouyang, “F- codellm: A federated learning framework for adapting large language models to practical software development,” inProc. of ICSE, 2024

2024
[21]

Code summarization without direct access to code-towards exploring federated llms for software engi- neering,

J. Kumar and S. Chimalakonda, “Code summarization without direct access to code-towards exploring federated llms for software engi- neering,” inProc. of EASE, 2024

2024
[22]

Recursive deep models for semantic compositionality over a sentiment treebank,

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” Seattle, Washington, USA, Oct. 2013, pp. 1631–1642

2013
[23]

Learning multiple layers of features from tiny images,

A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,”Handbook of Systemic Autoimmune Diseases, vol. 1, no. 4, 2009

2009
[24]

Accuracy-privacy trade-off in deep ensemble: A membership inference perspective,

S. Rezaei, Z. Shafiq, and X. Liu, “Accuracy-privacy trade-off in deep ensemble: A membership inference perspective,” inProc. of SP, 2023

2023
[25]

Learning-based difficulty cal- ibration for enhanced membership inference attacks,

H. Shi, T. Ouyang, and A. Wang, “Learning-based difficulty cal- ibration for enhanced membership inference attacks,” inProc. of EuroS&P, 2024

2024
[26]

Membership inference attacks by exploiting loss trajectory,

Y . Liu, Z. Zhao, M. Backes, and Y . Zhang, “Membership inference attacks by exploiting loss trajectory,” inProc. of CCS, 2022

2022
[27]

Com- parative analysis of membership inference attacks in federated and centralized learning,

A. Abbasi Tadi, S. Dayal, D. Alhadidi, and N. Mohammed, “Com- parative analysis of membership inference attacks in federated and centralized learning,”Information, vol. 14, no. 11, p. 620, 2023

2023
[28]

Efficient privacy auditing in federated learning,

H. Chang, B. Edwards, A. S. Paul, and R. Shokri, “Efficient privacy auditing in federated learning,” inProc. of USENIX Security, 2024

2024
[29]

Towards label-only membership inference attack against pre-trained large language models,

Y . He, B. Li, L. Liu, Z. Ba, W. Dong, Y . Li, Z. Qin, K. Ren, and C. Chen, “Towards label-only membership inference attack against pre-trained large language models,” inProc. of USENIX Security, 2025

2025
[30]

Towards sparsified federated neuroimaging models via weight pruning,

D. Stripelis, U. Gupta, N. Dhinagar, G. V . Steeg, P. M. Thompson, and J. L. Ambite, “Towards sparsified federated neuroimaging models via weight pruning,” inInternational Workshop on Distributed, Col- laborative, and Federated Learning. Springer, 2022, pp. 141–151

2022
[31]

Defending membership infer- ence attacks via privacy-aware sparsity tuning,

Q. Hu, H. Zhang, and H. Wei, “Defending membership infer- ence attacks via privacy-aware sparsity tuning,”arXiv preprint arXiv:2410.06814, 2024

work page arXiv 2024
[32]

Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,

M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” inProc. of SP, 2019

2019
[33]

Effective passive membership inference attacks in federated learning against overparameterized models,

J. Li, N. Li, and B. Ribeiro, “Effective passive membership inference attacks in federated learning against overparameterized models,” in Proc. of ICLR, 2023

2023
[34]

Neural network accept- ability judgments,

A. Warstadt, A. Singh, and S. R. Bowman, “Neural network accept- ability judgments,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 625–641, 2019

2019
[35]

Yelp dataset challenge: Review rating prediction,

N. Asghar, “Yelp dataset challenge: Review rating prediction,”arXiv preprint arXiv:1605.05362, 2016

work page arXiv 2016
[36]

Recursive deep models for semantic compositionality over a sentiment treebank,

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y . Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” inProc. of EMNLP, 2013

2013
[37]

Learning word vectors for sentiment analysis,

A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y . Ng, and C. Potts, “Learning word vectors for sentiment analysis,” inProc. of ACL, 2011

2011
[38]

Communication-efficient learning of deep networks from decentral- ized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentral- ized data,” inProc. of AISTATS, 2017

2017
[39]

Save it all: Enabling full parameter tuning for federated large language models via cycle block gradient descent,

L. Wang, Z. Wang, and X. Tang, “Save it all: Enabling full parameter tuning for federated large language models via cycle block gradient descent,” Jan 2024

2024
[40]

Online model compression for federated learning with large models,

T.-J. Yang, Y . Xiao, G. Motta, F. Beaufays, R. Mathews, and M. Chen, “Online model compression for federated learning with large models,” inProc. of ICASSP, 2023

2023
[41]

Parameter-efficient transfer learning for nlp,

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Larous- silhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inProc. of ICML, 2019

2019
[42]

Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization,

T. Che, J. Liu, Y . Zhou, J. Ren, J. Zhou, V . S. Sheng, H. Dai, and D. Dou, “Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization,” inProc. of EMNLP, 2023

2023
[43]

P- tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks,

X. Liu, K. Ji, Y . Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P- tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks,” inProc. of ACL, 2022

2022
[44]

How to combine membership-inference attacks on multiple updated models,

M. Jagielski, S. Wu, A. Oprea, J. Ullman, and R. Geambasu, “How to combine membership-inference attacks on multiple updated models,” ArXiv, vol. abs/2205.06369, 2022

work page arXiv 2022
[45]

Fedmia: An effective membership inference attack exploiting

G. Zhu, D. Li, H. Gu, Y . Yao, L. Fan, and Y . Han, “Fedmia: An effective membership inference attack exploiting ”all for one” principle in federated learning,” inProc. of CVPR, 2025

2025
[46]

Exploiting unintended feature leakage in collaborative learning,

L. Melis, C. Song, E. De Cristofaro, and V . Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” inProc. of SP, 2019

2019
[47]

Perfectly ac- curate membership inference by a dishonest central server in federated learning,

G. Pichler, M. Romanelli, L. R. Vega, and P. Piantanida, “Perfectly ac- curate membership inference by a dishonest central server in federated learning,”IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 4, pp. 4290–4296, 2023

2023
[48]

Analysis of privacy leakage in federated large language models,

M. Vu, T. Nguyen, M. T. Thaiet al., “Analysis of privacy leakage in federated large language models,” inProc. of AISTATS, 2024

2024
[49]

Dager: Exact gradient inversion for large language models,

I. Petrov, D. I. Dimitrov, M. Baader, M. N. M ¨uller, and M. Vechev, “Dager: Exact gradient inversion for large language models,” inProc. of NeurIPS, 2024

2024
[50]

Qwen2 Technical Report

Q. Team, “Qwen2 technical report,”arXiv preprint arXiv:2407.10671, 2024

work page internal anchor Pith review arXiv 2024
[51]

Towards practical few-shot federated nlp,

D. Cai, Y . Wu, H. Yuan, S. Wang, F. X. Lin, and M. Xu, “Towards practical few-shot federated nlp,” inProceedings of the 3rd Workshop on Machine Learning and Systems, 2023, pp. 42–48

2023
[52]

{FwdLLM}: Efficient federated finetuning of large language models with perturbed infer- ences,

M. Xu, D. Cai, Y . Wu, X. Li, and S. Wang, “{FwdLLM}: Efficient federated finetuning of large language models with perturbed infer- ences,” inProc. of USENIX ATC, 2024

2024
[53]

A framework for evaluating gradient leakage attacks in federated learning,

W. Wei, L. Liu, M. L. Loper, K.-H. Chow, M. E. Gursoy, S. Truex, and Y . Wu, “A framework for evaluating gradient leakage attacks in federated learning,”ArXiv, vol. abs/2004.10397, 2020

work page arXiv 2004
[54]

Deep leakage from gradients,

L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” inProc. of NeurIPS, 2019

2019
[55]

Fedbert: When federated learning meets pre-training,

Y . Tian, Y . Wan, L. Lyu, D. Yao, H. Jin, and L. Sun, “Fedbert: When federated learning meets pre-training,”ACM Transactions on Intelligent Systems and Technology (TIST), vol. 13, no. 4, pp. 1–26, 2022

2022
[56]

Qa-lora: Quantization-aware low-rank adap- tation of large language models,

Y . Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian, “Qa-lora: Quantization-aware low-rank adap- tation of large language models,” inProc. of ICLR, 2023

2023
[57]

Low-parameter federated learning with large language models,

J. Jiang, H. Jiang, Y . Ma, X. Liu, and C. Fan, “Low-parameter federated learning with large language models,” inInternational Conference on Web Information Systems and Applications. Springer, 2024, pp. 319–330

2024
[58]

Fdlora: Personalized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925, 2024

J. Qi, Z. Luan, S. Huang, C. Fung, H. Yang, and D. Qian, “Fdlora: Personalized federated learning of large language model via dual lora tuning,”arXiv preprint arXiv:2406.07925, 2024

work page arXiv 2024
[59]

Federated fine- tuning of large language models under heterogeneous tasks and client resources,

J. Bai, D. Chen, B. Qian, L. Yao, and Y . Li, “Federated fine- tuning of large language models under heterogeneous tasks and client resources,” inProc. of NeurIPS, 2024

2024
[60]

Fedrdma: Communication-efficient cross-silo federated LLM via chunked RDMA transmission,

Z. Zhang, D. Cai, Y . Zhang, M. Xu, S. Wang, and A. Zhou, “Fedrdma: Communication-efficient cross-silo federated LLM via chunked RDMA transmission,” inProc. of EuroMLSys, 2024

2024
[61]

Membership inference attacks and defenses in federated learning: A survey,

L. Bai, H. Hu, Q. Ye, H. Li, L. Wang, and J. Xu, “Membership inference attacks and defenses in federated learning: A survey,”ACM Computing Surveys, vol. 57, no. 4, pp. 1–35, 2024

2024
[62]

Agrevader: Poisoning membership inference against byzantine-robust federated learning,

Y . Zhang, G. Bai, M. A. P. Chamikara, M. Ma, L. Shen, J. Wang, S. Nepal, M. Xue, L. Wang, and J. Liu, “Agrevader: Poisoning membership inference against byzantine-robust federated learning,” inProc. of WWW, 2023

2023
[63]

Systematic evaluation of privacy risks of machine learning models,

L. Song and P. Mittal, “Systematic evaluation of privacy risks of machine learning models,” inProc. of USENIX Security, 2021

2021
[64]

Federated few-shot learning for mobile nlp,

D. Cai, S. Wang, Y . Wu, F. X. Lin, and M. Xu, “Federated few-shot learning for mobile nlp,” inProc. of MobiCom, 2023

2023
[65]

Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

E. B. Zaken, S. Ravfogel, and Y . Goldberg, “Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” inProc. of ACL, 2022. Ethics Considerations None. Appendix A. FedLLMs Fine-tuning Strategy As shown in Fig. 8, FedLLMs has three commonly used fine-tuning strategies, which are described in detail below: Partial Param...

2022