Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that federated PEFT turns small adapters into a jailbreak vector: a minority of malicious clients can raise attack success above 80 percent while the base model's weights stay frozen.

desk verdict A credible but measurement-sensitive empirical study that maps where FedPEFT jailbreak attacks work and where current defenses fail; the qualitative conclusions survive the wobbly API judge, the exact percentages do not. read the letter →

arxiv 2411.19335 v2 pith:CZMCPUI3 submitted 2024-11-28 cs.CR cs.AI

Shenghui Li , Edith C.-H. Ngai , Fanghua Ye , Thiemo Voigt This is my paper · ORCID
classification cs.CRcs.AI
keywords federatedlearningparameter-efficientfine-tuningLoRAjailbreakattacksafetyalignmentmodelpoisoningrobustaggregationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a warning about federated parameter-efficient fine-tuning (FedPEFT), the practice of adapting a shared pre-trained language model by having clients train only small adapter modules and aggregating those updates on a server. It claims that this paradigm contains a built-in attack surface: a handful of malicious clients that fine-tune adapters such as LoRA on harmful instruction-response pairs can drive the model's attack success rate from below 4 percent to over 80 percent, while frozen base-model weights and safety alignment remain untouched. The same mechanism works, less dramatically, with adapter methods that train only 0.001 percent of parameters. The paper also evaluates defenses and finds them wanting: robust aggregation schemes that are supposed to filter malicious updates collapse when benign clients hold heterogeneous data, and post-fine-tuning safety alignment restores safety only by paying a measurable accuracy cost. A sympathetic reader should come away believing that the safety of a federated fine-tuning system cannot be assumed from the safety of the base model or the small size of the adapters.

What carries the argument

The load-bearing object is the low-rank adapter update exchanged during FedPEFT. For LoRA, each weight is modified by a product of two small trainable matrices, $W_i \gets W_i + A_i B_i^\top$; for (IA)3 and LayerNorm, the updates are learned scaling vectors on activations or normalization gains. Because the federated server averages these compact updates with FedAvg, a small number of malicious clients can shift the aggregated adapter in a direction that maximises the likelihood of harmful completions while the frozen base weights remain superficially pristine. The paper's measurement machinery is the attack success rate computed on two harmful-prompt benchmarks, judged by an API-based safety classifier that the authors themselves describe as approximate and imperfect.

What would settle it

Re-run the LoRA attack on the same models, datasets, and client settings, then have human annotators score a random sample of the generated responses; if the human-judged attack success rate stays near the pre-attack baseline while the API judge reports around 80 percent, the central quantitative claim is an artifact of the evaluation oracle.

Watch

Extended reading notes

Core claim

The paper establishes 'PEFT-as-an-Attack' (PaaA): in a federated setup where all clients share a frozen, safety-aligned base model and exchange only the parameters of LoRA, (IA)3, or LayerNorm modules, a minority of compromised clients who follow the standard FedAvg protocol can train those small modules on harmful question-answer pairs and thereby jailbreak the global model. Across four recent instruction-tuned models, LoRA-based attacks reach an attack success rate above 80 percent within roughly twenty communication rounds, and even the sparsest methods reach 40 to 60 percent, after starting below 4 percent. The attack is not neutralized by Byzantine-robust aggregation: DnC and ClippedClustering hold in a single-domain IID setting but let the attack succeed when benign clients legitimately work on different domains. A post-fine-tuning safety-alignment step does bring the attack success rate below 10 percent, but at the cost of reduced accuracy on the target tasks, which the paper identifies as an 'alignment tax.' The claim is not that every federated fine-tuning system will be attacked, but that the standard protocol provides no protective barrier between small trainable adapters and the model's refusal behaviour.

Load-bearing premise

The whole attack-success-rate story rests on an automated API-based judge that the paper itself calls approximate and imperfect; if that judge is biased toward calling outputs harmful, the headline 80 percent and the sub-10 percent defense numbers could change substantially.

Editorial extensions

If this is right

  • Any organization deploying FedPEFT with standard FedAvg aggregation should assume that a single malicious participant among fifteen can degrade the global model's safety within a few communication rounds.
  • The safety of the base model is not a shield: freezing all pre-trained weights and tuning less than one percent of parameters does not prevent the jailbreak.
  • Robust aggregation is only as strong as its assumption about benign data; under non-IID client data, the leading aggregation defenses cannot distinguish the attack from legitimate variation.
  • Post-hoc safety alignment can repair the model, but the measured accuracy loss means a purely reactive defense is not free.
  • The threat is not limited to LoRA: all three tested PEFT families show elevated attack success, so restricting trainable parameters is not a sufficient mitigation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural editorial extension: because the mechanism is the low-rank adapter update, the same attack likely transfers to other adapter-based PEFT variants and to centralized adaptation pipelines that accept user-provided adapters, even if those adapters are not trained in a federated loop.
  • A testable extension would be to probe the adapter's low-rank subspace before aggregation and check whether harmful behaviour localises to a few singular directions; if it does, a projection-based defense could block the attack with less alignment tax than full re-alignment.
  • The imperfect API judge means the absolute numbers should be validated with human annotation before policy decisions are made; the qualitative direction of the result is more robust than the exact percentages.
  • The failure of DnC and ClippedClustering under heterogeneous data suggests that future defenses should model task and domain heterogeneity explicitly rather than treating all divergent updates as malicious.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether a minority of malicious clients in federated parameter-efficient fine-tuning can jailbreak safety-aligned large language models by locally fine-tuning PEFT modules (LoRA, (IA)3, LayerNorm) on harmful data. Across four instruction-tuned models and two QA tasks, it reports that LoRA reaches roughly 70-80% attack success rate on AdvBench/JailbreakBench with one or five malicious clients out of 15, while (IA)3 and LayerNorm reach 40-60%; it then evaluates robust aggregation schemes and a post-PEFT safety alignment defense, finding that DnC and ClippedClustering fail under heterogeneous mixed-domain data and that PPSA lowers ASR below 10% at the cost of task accuracy. The authors integrate the implementation into the Blades benchmark and frame the contribution as a new attack surface and a defense evaluation for FedPEFT.

Significance. The qualitative finding that fine-tuning a small PEFT module on harmful data can undo safety alignment is plausible and consistent with the existing non-federated fine-tuning attack literature, and the paper adds value by demonstrating it in a federated setting across several models and PEFT methods and by testing defenses. Strengths include the breadth of the experimental matrix, the use of public safety benchmarks, the explicit disclosure that the API judge is approximate, and the release of code through Blades. The main caveat is that all ASR numbers rest on a single unvalidated API judge, so the precise percentages should be treated as provisional until corroborated; nevertheless the directional conclusions are likely to survive better measurement.

major comments (3)
  1. [IV-C.1] The only safety metric is the OpenAI chat completion API judge, which the paper itself footnotes as "not perfect and provides only an approximate assessment of the potential harm." No validation against human labels, no alternative judge, and no repeated or seeded runs are reported. Since the headline claims (LoRA ASR exceeding 70-80%, DnC/ClippedClustering failing with ASR above 80% in the mixed-domain setting, and PPSA cutting ASR below 10%) are all ASR values from this proxy, the paper should report the judge's agreement with human annotations on a sample, or at least show that the ordinal conclusions are unchanged under a second judge or perturbed thresholds; otherwise the quantitative claims are not independently bounded.
  2. [V-C, Table II] The RAS evaluation is limited to Phi-3.5-Mini-Instruct with LoRA, 15 clients, exactly 3 malicious clients, and one particular construction of the mixed-domain setting. The conclusion that "all RASs fail" under heterogeneous distributions is therefore a statement about this configuration, not a general result. Moreover, the success/failure contrast between the MedQA-only and mixed-domain rows is drawn from single-run ASR values with no variance estimates; adding multiple seeds or clearly qualifying the scope of the conclusion would be needed to support the strength of the claim.
  3. [V-D, Figure 5] The PPSA experiment uses a single 14-round configuration with three alignment clients added only in the final four rounds, and the text does not report the alignment dataset size, the number of local steps, the learning rate, or the number of seeds. In addition, the largest utility drop cited for LLaMA-3.2-3B-Instruct is on MetaMathQA, a task the model was not being fine-tuned on during the MedQA runs, so it is unclear how much of the drop is alignment tax rather than domain mismatch. The authors should provide the missing training details and separate out the task-specific effect before concluding that PPSA "severely degrades" accuracy.
minor comments (5)
  1. [Abstract and Section I] The headline success rate is stated inconsistently: the abstract says "approximate 80%", the introduction says ">80%", and Section I contains a typo "(> 80%))"; please unify the numbers and fix the typo.
  2. [Table I] LLaMA-3.2-3B-Instruct is listed with 7.6B total parameters and a 0.53% trainable ratio for LoRA, but the released model has about 3.2B parameters; the table appears to contain a typo.
  3. [Equation (2)] The RMSNorm formula is garbled in the typeset text; please replace it with the standard form \hat{x}_i = x_i / \sqrt{(1/d)\sum_j x_j^2} \cdot g_i.
  4. [Figure 4] The caption of panel (b) says "Harmfulness Rate (%)" while the caption and text report ASR; use one consistent term throughout.
  5. [Section IV-A] The phrase "4-int quantized" should presumably be "4-bit quantized".

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claims are direct empirical measurements, with no fitted parameter, self-citation chain, or definitional reduction standing in for evidence.

full rationale

This paper is an empirical security evaluation, not a derivation. The central claims—that FedPEFT with LoRA, (IA)3, or LayerNorm raises attack success rates to roughly 40–80% with a small number of malicious clients, that DnC and ClippedClustering fail under heterogeneous data distributions, and that PPSA trades safety for downstream utility—are supported by direct measurements of model outputs against external safety benchmarks (AdvBench and JailbreakBench) judged by an external API. No equation is defined in terms of its target conclusion, no parameter is fitted and then reported as a prediction, and no uniqueness theorem is imported from prior work. The authors cite their own Blades benchmark and earlier federated-learning robustness studies, but those citations are tools and background rather than the evidence for the attack; the attack's existence rests on the measured outputs. The footnoted imperfection of the OpenAI API judge ('the API is not perfect and provides only an approximate assessment of the potential harm') is a measurement-validity limitation that could shift absolute ASR numbers, but it is not a circularity. The derivation chain is therefore self-contained with respect to circularity, and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to make the attack work; the attack is standard harmful fine-tuning and defense behavior is measured directly. The stated assumptions concern the threat model, data source, and evaluation judge, all acknowledged in the paper.

assumptions (4)
  • domain assumption The OpenAI API harmfulness judge is a valid proxy for actual harmful content generation.
    Footnote 6 in Section IV-C acknowledges the API is 'not perfect and provides only an approximate assessment of potential harm,' yet ASR is the primary safety metric without manual validation.
  • domain assumption Adversaries can only manipulate PEFT module updates and cannot modify the frozen base model parameters.
    Stated in Section III-B as part of adversary capabilities; the attack's efficiency claim depends on this restriction.
  • domain assumption Honest clients constitute the majority and contribute benign training data while following the standard FedPEFT protocol.
    Stated in Section III-B; RAS defense results assume a 12-benign to 3-malicious client split.
  • domain assumption BeaverTails unsafe samples are representative of harmful data that can jailbreak aligned models.
    Used in Section IV-A to construct malicious training sets; no demonstration that the specific sampled subset is optimal or typical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning." pith.science (2026). https://pith.science/paper/CZMCPUI3

@misc{pith2026241119335,
  author       = {Pith},
  title        = {Pith review of: PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZMCPUI3}},
  note         = {Machine review of arXiv:2411.19335}
}
read the original abstract

Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a promising paradigm for privacy-preserving and efficient adaptation of Pre-trained Language Models (PLMs) in Federated Learning (FL) settings. It preserves data privacy by keeping the data decentralized and training the model on local devices, ensuring that raw data never leaves the user's device. Moreover, the integration of PEFT methods such as LoRA significantly reduces the number of trainable parameters compared to fine-tuning the entire model, thereby minimizing communication costs and computational overhead. Despite its potential, the security implications of FedPEFT remain underexplored. This paper introduces a novel security threat to FedPEFT, termed PEFT-as-an-Attack (PaaA), which exposes how PEFT can be exploited as an attack vector to circumvent PLMs' safety alignment and generate harmful content in response to malicious prompts. Our evaluation of PaaA reveals that with less than 1% of the model's parameters set as trainable, and a small subset of clients acting maliciously, the attack achieves an approximate 80% attack success rate using representative PEFT methods such as LoRA. To mitigate this threat, we further investigate potential defense strategies, including Robust Aggregation Schemes (RASs) and Post-PEFT Safety Alignment (PPSA). However, our empirical analysis highlights the limitations of these defenses, i.e., even the most advanced RASs, such as DnC and ClippedClustering, struggle to defend against PaaA in scenarios with highly heterogeneous data distributions. Similarly, while PPSA can reduce attack success rates to below 10%, it severely degrades the model's accuracy on the target task. Our results underscore the urgent need for more effective defense mechanisms that simultaneously ensure security and maintain the performance of the FedPEFT paradigm.

Figures

Figures reproduced from arXiv: 2411.19335 by the authors.

Figure 1
Figure 1. Architectures of the three PEFT methods examined in this [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the system model. ① FedPEFT System: Multiple clients collaboratively fine-tune a PLM with small PEFT modules using their local datasets. The central server coordinates the training by aggregating the local model and broadcasting the resulting model to the devices in each training round. ② Threat Model: Compromised clients perform PEFT on malicious data while following the standard FedPEFT protocol. ③ Att… view at source ↗
Figure 3
Figure 3. Performance comparison of three FedPEFT methods across 25 communication rounds, fine-tuned on the MedQA dataset without [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: ASR comparison of FedPEFT methods under varying numbers of malicious clients (0, 1, and 5) across different PLMs over [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Evaluation of jailbreak attacks, normal fine-tuning, and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Graph representation learning plus iterative augmented Lagrangian optimization creates stronger, harder-to-detect model manipulation attacks on federated LLM fine-tuning, cutting global accuracy by up to 26%.

Reference graph

Works this paper leans on

87 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [40]

    Emerging safety attack and defense in federated instruction tuning of large language models,

    R. Ye, J. Chai, X. Liu, Y . Yang, Y . Wang, and S. Chen, “Emerging safety attack and defense in federated instruction tuning of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.10630

    arXiv 2024

  2. [1]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Minneapolis, Minnesota: Association for C...

    2019

  3. [2]

    Language models are few-shot learners,

    T. Brown et al. , “Language models are few-shot learners,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    2020

  4. [3]

    Chatgpt,

    OpenAI, “Chatgpt,” 2022. [Online]. Available: https://openai.com/blog/ chatgpt/

    2022

  5. [6]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288,

    arXiv

  6. [7]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,

    Meta , “Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,” 2024, llama 3.2 for Edge AI and Vision. [Online]. Available: https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices

    2024

  7. [8]

    Qwen technical report,

    J. B. et al., “Qwen technical report,” 2023

    2023

  8. [9]

    Available: https://doi.org/10.48550/arXiv.2307.09288

    [Online]. Available: https://doi.org/10.48550/arXiv.2307.09288

    arXiv doi:10.48550/arxiv.2307.09288

Show all 87 references
  1. [10]

    Qwen2.5: A party of foundation models,

    Qwen Team, “Qwen2.5: A party of foundation models,” September

  2. [11]

    On the client preference of llm fine-tuning in federated learning,

    F. Wu, X. Liu, H. Wang, X. Wang, and J. Gao, “On the client preference of llm fine-tuning in federated learning,” 2024. [Online]. Available: https://arxiv.org/abs/2407.03038

    2024 arXiv

  3. [12]

    Qwen2 technical report,

    A. Yaai, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al. , “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  4. [13]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Online and Punta Cana, Dominican Republic: Association for Computational Linguistics,...

    2021

  5. [14]

    Available: https://qwenlm.github.io/blog/qwen2.5/

    [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/

  6. [15]

    Analyzing and reducing catastrophic forgetting in parameter efficient tuning,

    W. Ren, X. Li, L. Wang, T. Zhao, and W. Qin, “Analyzing and reducing catastrophic forgetting in parameter efficient tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2402.18865

    2024 arXiv

  7. [16]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223 , 2023. [Online]. Available: https://arxiv.org/abs/2303.18223

    2023 arXiv

  8. [17]

    Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation,

    S.-Y . YEH, Y .-G. Hsieh, Z. Gao, B. B. W. Yang, G. Oh, and Y . Gong, “Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net...

    2024

  9. [18]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  10. [19]

    Regulation (eu) 2016/679 of the european parliament and of the council,

    GDPR, “Regulation (eu) 2016/679 of the european parliament and of the council,” 2016. [Online]. Available: http://data.europa.eu/eli/reg/ 2016/679/2016-05-04

    2016

  11. [20]

    VeRA: Vector- based random matrix adaptation,

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano, “VeRA: Vector- based random matrix adaptation,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=NjNfLdxr3A

    2024

  12. [21]

    Communication-Efficient Learning of Deep Networks from Decentralized Data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Re...

    2017

  13. [22]

    Lessons from archives: Strategies for collecting sociocultural data in machine learning,

    E. S. Jo and T. Gebru, “Lessons from archives: Strategies for collecting sociocultural data in machine learning,” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , ser. FAcctT ’20. New York, NY , USA: Association for Computing Machinery, 202...

    2020 doi

  14. [23]

    Synergizing foundation models and federated learning: A survey,

    S. Li, F. Ye, M. Fang, J. Zhao, Y .-H. Chan, E. C. H. Ngai, and T. V oigt, “Synergizing foundation models and federated learning: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12844

    2024

  15. [24]

    California consumer privacy act (ccpa),

    CCPA, “California consumer privacy act (ccpa),” 2023. [Online]. Available: https://oag.ca.gov/privacy/ccpa

    2023

  16. [25]

    Training language models to follow instructions with human feedback,

    L. e. a. Ouyang, “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 27 730–27 744. [On...

    2022

  17. [26]

    A survey on efficient federated learning methods for foundation model training,

    H. Woisetschläger, A. Isenko, S. Wang, R. Mayer, and H.-A. Jacobsen, “A survey on efficient federated learning methods for foundation model training,” in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-24 . International Joint ...

    2024 doi arXiv

  18. [27]

    RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback,

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. R. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogi, and S. Prakash, “RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback,” in Forty-first International Conference on Machine Learning , 20...

    2024

  19. [28]

    A comprehensive study of jailbreak attack versus defense for large language models,

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek, “A comprehensive study of jailbreak attack versus defense for large language models,” in Findings of the Association for Computational Linguistics ACL 2024 , L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thailand and virtu...

    2024

  20. [29]

    Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning,

    T. Huang, G. Bhattacharya, P. Joshi, J. Kimball, and L. Liu, “Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2408.09600

    2024 arXiv

  21. [30]

    An experimental study of byzantine- robust aggregation schemes in federated learning,

    S. Li, E. C.-H. Ngai, and T. V oigt, “An experimental study of byzantine- robust aggregation schemes in federated learning,” IEEE Transactions on Big Data , vol. 10, no. 6, pp. 975–988, 2024

    2024

  22. [31]

    Constitutional ai: Harmlessness from ai feedback,

    Y . B. et al., “Constitutional ai: Harmlessness from ai feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2212.08073

    2022 arXiv

  23. [32]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho,...

    2022

  24. [33]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openr...

    2024

  25. [34]

    Blades: A unified benchmark suite for byzantine attacks and defenses in federated learning,

    S. Li, E. C.-H. Ngai, F. Ye, L. Ju, T. Zhang, and T. V oigt, “Blades: A unified benchmark suite for byzantine attacks and defenses in federated learning,” in 2024 IEEE/ACM Ninth International Conference on Internet- of-Things Design and Implementation (IoTDI) , 2024, pp. 158–1...

    2024

  26. [35]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019. [Online]. Available: https://arxiv.org/abs/1907.11692

    1907 arXiv

  27. [36]

    Manipulating the byzantine: Opti- mizing model poisoning attacks and defenses for federated learning,

    V . Shejwalkar and A. Houmansadr, “Manipulating the byzantine: Opti- mizing model poisoning attacks and defenses for federated learning,” in NDSS, 2021

    2021

  28. [37]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” 2016. [Online]. Available: https://arxiv.org/abs/1607.06450

    2016 arXiv

  29. [38]

    Tuning layernorm in attention: Towards efficient multi-modal LLM finetuning,

    B. Zhao, H. Tu, C. Wei, J. Mei, and C. Xie, “Tuning layernorm in attention: Towards efficient multi-modal LLM finetuning,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=YR3ETaElNK

    2024

  30. [39]

    Stanford alpaca: An instruction-following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  31. [41]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...

    2017

  32. [42]

    Local model poisoning attacks to byzantine-robust federated learning,

    M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in29th USENIX Security Symposium (USENIX Security 20) , 2020, pp. 1605–1622. [Online]. Available: https://www.usenix.org/system/files/sec20summer_fang_prepub.pdf

    2020

  33. [43]

    Root mean square layer normalization,

    B. Zhang and R. Sennrich, “Root mean square layer normalization,” in Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https...

    2019

  34. [44]

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack,

    T. Huang, S. Hu, and L. Liu, “Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack,” in The Thirty- eighth Annual Conference on Neural Information Processing Systems ,

  35. [45]

    Phi-3 technical report: A highly capable language model locally on your phone,

    Microsoft, “Phi-3 technical report: A highly capable language model locally on your phone,” 2024. [Online]. Available: https: //arxiv.org/abs/2404.14219

    2024 arXiv

  36. [46]

    Robust aggregation for federated learning,

    K. Pillutla, S. M. Kakade, and Z. Harchaoui, “Robust aggregation for federated learning,” arXiv preprint arXiv:1912.13445 , 2019

    1912 arXiv

  37. [47]

    Metamath: Bootstrap your own mathematical questions for large language models,

    L. Yu, W. Jiang, H. Shi, J. YU, Z. Liu, Y . Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu, “Metamath: Bootstrap your own mathematical questions for large language models,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://o...

    2024

  38. [48]

    Byzantine-robust aggregation in federated learning empowered industrial iot,

    S. Li, E. Ngai, and T. V oigt, “Byzantine-robust aggregation in federated learning empowered industrial iot,” IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 1165–1175, 2023

    2023

  39. [49]

    Measuring mathematical problem solving with the MATH dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. [O...

    2021

  40. [50]

    Available: https://openreview.net/forum?id=lpXDZKiAnt

    [Online]. Available: https://openreview.net/forum?id=lpXDZKiAnt

  41. [51]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset,

    J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y . Wang, and Y . Yang, “Beavertails: Towards improved safety alignment of llm via a human-preference dataset,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko,...

    2023

  42. [52]

    Parameter-efficient fine-tuning of large-scale pre-trained language models,

    N. Ding, Y . Qin, G. Yang, F. Wei, Z. Yang, Y . Su, S. Hu, Y . Chen, C.-M. Chan, W. Chen et al. , “Parameter-efficient fine-tuning of large-scale pre-trained language models,” Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023

    2023

  43. [53]

    Distributed statistical machine learning in adversarial settings: Byzantine gradient descent,

    Y . Chen, L. Su, and J. Xu, “Distributed statistical machine learning in adversarial settings: Byzantine gradient descent,” Proceedings of the ACM on Measurement and Analysis of Computing Systems , 2017

    2017

  44. [54]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 2021

    2021

  45. [55]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations , 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  46. [56]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, vol. 11, no. 14,

  47. [57]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” 2023

    2023

  48. [58]

    Jailbreakbench: An open robustness bench- mark for jailbreaking large language models,

    P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V . Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramèr, H. Hassani, and E. Wong, “Jailbreakbench: An open robustness bench- mark for jailbreaking large language models,” in NeurIPS Datasets and Benchmarks...

    2024

  49. [59]

    Byzantine-robust distributed learning: Towards optimal statistical rates,

    D. Yin, Y . Chen, R. Kannan, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” in International Conference on Machine Learning . PMLR, 2018

    2018

  50. [60]

    Mitigating the alignment tax of RLHF,

    Y . Lin, H. Lin, W. Xiong, S. Diao, J. Liu, J. Zhang, R. Pan, H. Wang, W. Hu, H. Zhang, H. Dong, R. Pi, H. Zhao, N. Jiang, H. Ji, Y . Yao, and T. Zhang, “Mitigating the alignment tax of RLHF,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proce...

    2024

  51. [61]

    Constitutional ai recipe,

    S. Huang, L. Tunstall, E. Beeching, L. von Werra, O. Sanseviero, K. Rasul, and T. Wolf, “Constitutional ai recipe,” Hugging Face Blog,

  52. [62]

    Available: https://huggingface.co/blog/constitutional_ai

    [Online]. Available: https://huggingface.co/blog/constitutional_ai

  53. [63]

    Federated adaptation for foundation model-based recommendations,

    C. Zhang, G. Long, H. Guo, X. Fang, Y . Song, Z. Liu, G. Zhou, Z. Zhang, Y . Liu, and B. Yang, “Federated adaptation for foundation model-based recommendations,” in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-24 . Internati...

    2024 doi arXiv

  54. [64]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  55. [65]

    Personalized federated learning for text classification with gradient-free prompt tuning,

    W. Rui, Y . Tong, Z. Ruiyi, K. Sungchul, R. Ryan A., Z. Handong, W. Junda, M. Subrata, Y . Lina, and H. Ricardo, “Personalized federated learning for text classification with gradient-free prompt tuning,” in Proceedings of the 2024 Conference of the North American Chapter of t...

    2024

  56. [66]

    Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients,

    S. Su, B. Li, and X. Xue, “Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients,” arXiv preprint arXiv:2311.11227 , oct 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2311.11227

    2024 doi arXiv

  57. [67]

    What is in your safe data? identifying benign data that breaks safety,

    L. He, M. Xia, and P. Henderson, “What is in your safe data? identifying benign data that breaks safety,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id= Hi8jKh4HE9

    2024

  58. [68]

    Backdoor threats from compromised foundation models to federated learning,

    X. Li, S. Wang, C. Wu, H. Zhou, and J. Wang, “Backdoor threats from compromised foundation models to federated learning,” in International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023 , oct 2023. [Online]. Available: https://o...

    2023

  59. [69]

    Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning,

    S. Malaviya, M. Shukla, and S. Lodha, “Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning,” in Proceedings of The 2nd Conference on Lifelong Learning Agents , ser. Proceedings of Machine Learning Research,...

    2023

  60. [70]

    Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization,

    T. Che, J. Liu, Y . Zhou, J. Ren, J. Zhou, V . Sheng, H. Dai, and D. Dou, “Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ,...

    2023

  61. [71]

    Removing RLHF protections in GPT-4 via fine-tuning,

    Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. Hashimoto, and D. Kang, “Removing RLHF protections in GPT-4 via fine-tuning,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: S...

    2024

  62. [72]

    Fedbpt: Efficient federated black-box prompt tuning for large language models,

    J. Sun, Z. Xu, H. Yin, D. Yang, D. Xu, Y . Chen, and H. R. Roth, “Fedbpt: Efficient federated black-box prompt tuning for large language models,” in Proceedings of the 41th International Conference on Machine Learning , jul 2024. [Online]. Available: https://doi.org/10.48550/a...

    2024 doi arXiv

  63. [73]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications,

    B. Wei, K. Huang, Y . Huang, T. Xie, X. Qi, M. Xia, P. Mittal, M. Wang, and P. Henderson, “Assessing the brittleness of safety alignment via pruning and low-rank modifications,” in Forty-first International Conference on Machine Learning

  64. [74]

    Navigating the safety landscape: Measuring risks in finetuning large language models,

    S. Peng, P.-Y . Chen, M. Hull, and D. H. Chau, “Navigating the safety landscape: Measuring risks in finetuning large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2405.17374

    2024 arXiv

  65. [75]

    Federated fine-tuning of large language models under heterogeneous language tasks and client resources,

    J. Bai, D. Chen, B. Qian, L. Yao, and Y . Li, “Federated fine-tuning of large language models under heterogeneous language tasks and client resources,” arXiv preprint arXiv:2402.11505 , 2024. [Online]. Available: https://arxiv.org/abs/2402.11505

    2024 arXiv

  66. [76]

    Fine-tuning can cripple your foundation model; preserving features may be the solution,

    J. Mukhoti, Y . Gal, P. H. S. Torr, and P. K. Dokania, “Fine-tuning can cripple your foundation model; preserving features may be the solution,”

  67. [77]

    Securing federated learning against novel and classic backdoor threats during foundation model integration,

    X. Bi and X. Li, “Securing federated learning against novel and classic backdoor threats during foundation model integration,” 2024. [Online]. Available: https://arxiv.org/abs/2410.17573

    2024 arXiv

  68. [78]

    Shadow alignment: The ease of subverting safely-aligned language models,

    X. Yang, X. Wang, Q. Zhang, L. R. Petzold, W. Y . Wang, X. Zhao, and D. Lin, “Shadow alignment: The ease of subverting safely-aligned language models,” in ICLR 2024 Workshop on Secure and Trustworthy Large Language Models , 2024. [Online]. Available: https://openreview.net/for...

    2024

  69. [79]

    A safety realignment framework via subspace-oriented model fusion for large language models,

    X. Yi, S. Zheng, L. Wang, X. Wang, and L. He, “A safety realignment framework via subspace-oriented model fusion for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2405.09055

    2024 arXiv

  70. [80]

    No two devils alike: Unveiling distinct mechanisms of fine-tuning attacks,

    C. T. Leong, Y . Cheng, K. Xu, J. Wang, H. Wang, and W. Li, “No two devils alike: Unveiling distinct mechanisms of fine-tuning attacks,” 2024. [Online]. Available: https://arxiv.org/abs/2405.16229

    2024 arXiv

  71. [83]

    What makes and breaks safety fine-tuning? a mechanistic study,

    S. Jain, E. S. Lubana, K. Oksuz, T. Joy, P. H. S. Torr, A. Sanyal, and P. K. Dokania, “What makes and breaks safety fine-tuning? a mechanistic study,” 2024. [Online]. Available: https://arxiv.org/abs/2407.10264

    2024 arXiv

  72. [85]

    Available: https://arxiv.org/abs/2308.13320

    [Online]. Available: https://arxiv.org/abs/2308.13320

    arXiv

  73. [86]

    Safety fine- tuning at (almost) no cost: A baseline for vision large language models,

    Y . Zong, O. Bohdal, T. Yu, Y . Yang, and T. Hospedales, “Safety fine- tuning at (almost) no cost: A baseline for vision large language models,” in Forty-first International Conference on Machine Learning , 2024. [Online]. Available: https://openreview.net/forum?id=bWZKvF0g7G

    2024

  74. [87]

    Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack,

    T. Huang, S. Hu, F. Ilhan, S. F. Tekin, and L. Liu, “Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack,”

  75. [88]

    Available: https://arxiv.org/abs/2405.18641

    [Online]. Available: https://arxiv.org/abs/2405.18641

    arXiv

  76. [2021]

    Available: https://www.mdpi.com/2076-3417/11/14/6421

    [Online]. Available: https://www.mdpi.com/2076-3417/11/14/6421

  77. [2023]

    Available: https://arxiv.org/abs/2302.13971

    [Online]. Available: https://arxiv.org/abs/2302.13971

    arXiv

  78. [2024]

    Available: https://arxiv.org/abs/2303.08774

    [Online]. Available: https://arxiv.org/abs/2303.08774

    arXiv

  79. [7449]

    Available: https://aclanthology.org/2024.findings-acl.443

    [Online]. Available: https://aclanthology.org/2024.findings-acl.443

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.