Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Benign parameter-efficient fine-tuning can significantly shift LLM safety and fairness, with adapter methods preserving alignment and prompt-based methods degrading it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:50 UTC pith:H27RDAIW

load-bearing objection Useful first broad PEFT safety/fairness map, but the concentrated utility-outlier removal likely biases the 'LLaMA stable' result and the abstract's coding extension is missing from the body; needs re-analysis before I'd trust the specific rankings. the 4 major comments →

arxiv 2511.00382 v2 pith:H27RDAIW submitted 2025-11-01 cs.AI cs.LG

Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs

Mina Taraghi , Yann Pequignot , Amin Nikanjam , Mohamed Amine Merzouk , Foutse Khomh This is my paper
classification cs.AI cs.LG
keywords Large Language ModelsParameter-Efficient Fine-TuningSafety alignmentFairnessLoRAIA3Prompt-TuningBBQ
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether parameter-efficient fine-tuning (PEFT) on benign, everyday data can quietly change an LLM's safety and fairness. It fine-tunes four instruction-tuned 7-8B models (LLaMA, Qwen, Mistral, Gemma) with four PEFT methods -- LoRA, IA3, Prompt-Tuning, P-Tuning -- across six training settings, producing 235 usable variants evaluated on 11 safety hazard categories and 9 fairness dimensions. The central finding: adapter-based methods (LoRA, IA3) generally preserve or even improve safety and fairness, while prompt-based methods (Prompt-Tuning, P-Tuning) more often degrade both. The base model strongly moderates the effect: LLaMA stays stable, Qwen improves modestly, Gemma shows the steepest safety decline, and Mistral varies most. The authors conclude that benign intent does not guarantee safe behavior, and recommend starting from a well-aligned base model, favoring adapters, and auditing at category level.

Core claim

On the paper's own terms, the discovery is that even benign, task-appropriate fine-tuning is not alignment-neutral: the choice of PEFT method and base model can shift measured safety and fairness by large amounts. Across 235 fine-tuned variants, adapter-based methods (LoRA, IA3) tend to raise or preserve safety scores and keep fairness accuracy higher with lower bias, whereas prompt-based methods (Prompt-Tuning, P-Tuning) significantly reduce safety in most hazard categories and depress fairness accuracy, especially in ambiguous contexts. The base model is a strong moderator -- LLaMA is comparatively robust, Qwen shows modest gains, Gemma exhibits the steepest and most consistent safety decl

What carries the argument

The central instrument is a controlled experimental grid: four instruction-tuned 7-8B base models (LLaMA-3-8B, Qwen2.5-7B, Mistral-7B, Gemma-7B) are each fine-tuned with four PEFT methods (LoRA, IA3, Prompt-Tuning, P-Tuning) under six training settings (SFT/DPO x two datasets x one or five epochs x two learning rates), yielding 264 variants of which 235 pass validity filters. Safety is scored with an automated guard model (LLaMA-Guard-2) on the 330-prompt HEx-PHI benchmark across 11 hazard categories; fairness is scored with a manually cleaned version of the BBQ-Lite multiple-choice benchmark (15,876 questions after corrections) across 9 demographic categories, using accuracy and bias scores

Load-bearing premise

The rankings of methods and base models assume the automated safety judge (LLaMA-Guard-2) and the manually edited BBQ-Lite fairness benchmark measure safety and fairness equally across all four model families; if the judge is systematically biased toward or against one family, the central comparisons could be distorted.

What would settle it

Re-score a sample of the fine-tuned variants' prompt-response pairs with human annotators (or a second guard model from a different training lineage) and check whether the adapter-vs-prompt ranking and the base-model ordering (LLaMA stable, Gemma declining) reproduce; a systematic disagreement that varies by model family would indicate the reported differences are measurement artifacts rather than alignment shifts.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Favoring adapter-based PEFT (LoRA, IA3) over prompt-based methods should reduce the risk of benign fine-tuning harming safety or fairness.
  • Base-model selection is a risk decision: starting from a well-aligned model (e.g., Qwen for fairness, LLaMA for safety stability) is more decisive than hyperparameter tuning.
  • Safety and fairness do not move together; a configuration that improves one can worsen the other, so both must be audited separately at category level.
  • Specific categories are early-warning indicators: Child Abuse Content and Adult Content for safety, Sexual Orientation and Nationality for fairness -- aggregate scores should not be used alone.
  • Fine-tuning hyperparameters (learning rate, epochs, dataset, SFT vs DPO) have limited and sporadic effects on alignment compared to method and base model choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism (adapters leave core weights intact, prompt methods rewrite the input path) is causal, then any PEFT variant that modifies input embeddings or activation distributions -- such as other soft-prompt or prefix methods -- may carry similar alignment risk; this is a testable extension.
  • The steepest fairness drops occurring in categories with the highest base accuracy suggest a ceiling or regression-to-the-mean pattern; a direct test would compare fine-tuning effects on held-out category variants with matched base accuracy.
  • The findings imply an audit protocol: before deploying any PEFT-adapted model, rerun a category-level guard model and bias benchmark even when the tuning dataset was benign; this could become a standard pre-deployment check on model hubs.
  • Because the base model effect dominates the fine-tuning config effect, organizations could screen candidate base models once per use case and then fix a conservative adapter setting, substantially reducing the cost of per-configuration safety and fairness audits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a large-scale empirical study of how four PEFT methods (LoRA, IA3, Prompt-Tuning, P-Tuning) affect the safety and fairness of four instruction-tuned 7–8B LLMs. The authors fine-tune models on benign conversational data under SFT/DPO, varying dataset, learning rate, and epoch count; evaluate safety with HEx-PHI prompts scored by Llama-Guard-2, fairness with a manually modified BBQ-Lite, and utility with GPT-4o-based MT-Bench scoring. After removing inference failures and utility outliers, 235 models are analyzed with paired non-parametric tests. The central claim is that adapter-based methods generally preserve or improve safety/fairness, while prompt-based methods more often degrade them, with base-model choice as a strong moderator: LLaMA stable, Qwen modest gains, Gemma steepest safety decline, Mistral most variable.

Significance. If the central claim holds, this is a valuable and practically relevant contribution: it provides the first systematic, multi-model comparison of PEFT-specific alignment risks, and the practical guidance (prefer adapters, audit category-level metrics, start from a well-aligned base) is actionable. The study's strengths include the large experimental matrix (264 initial fine-tunes), the use of external benchmarks and an independent guard model for measurement, the detailed statistical appendix, and the provision of a replication package with the modified BBQ-Lite. The paper also transparently reports threats to validity. However, the robustness of the headline comparisons is currently undermined by a non-random filtering step and by some inconsistent reporting of sample sizes and significance.

major comments (4)
  1. [§IV (Data Filtering) and Appendix C] The removal of utility outliers is concentrated in one cell: 13 of the 19 removed models are Prompt-Tuning runs of LLaMA (Appendix C). Because §IV.C reports positive correlations between utility and safety/accuracy, removing these low-utility runs likely removes disproportionately low-safety, low-fairness runs. This directly inflates LLaMA's apparent safety stability and attenuates the prompt-vs-adapter contrast, both of which anchor the paper's central claim. The §VI.A caveat that 'worst-case degradation may be underestimated' does not address the cell-specific, non-random nature of the removal. Please reanalyze with the 19 outlier models included (treating severe utility collapse as an outcome, not a missing value) and/or provide a sensitivity analysis excluding each cell in turn.
  2. [§III.C and Abstract] The reported numbers do not reconcile. §III.C states '24 models were fine-tuned for each of LLaMA, Mistral, and Qwen, and 16 for Gemma, bringing the total number of fine-tuned models to 264'; 24+24+24+16=88, not 264. The abstract also mentions a coding-task extension with 96 additional fine-tuned models, but the main text contains no section describing or analyzing this extension. These inconsistencies matter because filtering fractions, cell sizes for paired tests, and the claimed scope of the study depend on the exact counts. Please correct the counts and either integrate the coding-extension results or remove that claim.
  3. [§IV.A.1 and Table X/Appendix E] The text says safety 'increases significantly with LoRA (p = 0.059)' while the stated significance level is α = 0.05. Appendix Table X lists the LoRA-vs-base comparison as p = 0.0587, which is not significant at the chosen threshold. This is not merely a typo: the findings summary in §IV.A claims 'Adapter-based techniques yield statistically significant safety gains,' which is supported only by IA3 under the stated α. Please reword the LoRA claim and adjust any summary statements that rely on it.
  4. [§III.D.2 / §VI.A] Safety scores are produced by Llama-Guard-2, an 8B model from the Llama-3 family, and are used to rank all fine-tuned models, including Llama-3-8B-Instruct itself. A systematic same-family or judge-model bias could distort the LLaMA-stable versus Gemma-declining comparison, which is the paper's second headline finding. The threat is acknowledged but not quantified. Please report agreement on a subsample with a second independent judge (e.g., a different guard model or human annotations), or at least discuss the direction of the likely bias and why the cross-model moderation finding survives it.
minor comments (4)
  1. [Affiliation block] Typographical errors: 'Poytechnique Montreal' should be 'Polytechnique Montreal'.
  2. [§III.D.3, Eq. (1)–(2)] The bias-score equations are correct but the explanation of n_biased_ans and n_non-UNKNOWN_outputs is spread across the main text and Appendix B; consider moving the full derivation into the main text for readability.
  3. [§IV.B] Table VI headers 'Accuracy AMB' etc. are clear, but the text switches between 'Bias AMB' and 'BiasScore AMB' without consistency; unify notation.
  4. [Appendix A] The manual modifications to BBQ-Lite are substantial (200 examples excluded, 32 corrected). The appendix is helpful, but the paper should state explicitly whether the modified benchmark is released and whether the base-model fairness scores in Table VI are computed on the original or modified set.

Circularity Check

0 steps flagged

No circular derivation: claims rest on external benchmarks, fresh fine-tuned runs, and independent judges; no parameter is fitted to the target outcomes.

full rationale

The paper's central claims are empirical comparisons of post-tuning safety and fairness scores. These scores are computed from external instruments: HEx-PHI prompts scored by LLaMA-Guard-2, BBQ-Lite (with the authors' manually documented corrections), and GPT-4o utility judgments. No equation in the paper fits a parameter to the target safety/fairness outcome, and no 'prediction' is derived from a fitted input. The only explicitly definitional relation is Bias_AMB = (1 - accuracy) * Bias_DIS, and the paper itself flags the resulting correlation as expected from the definition ('BiasAMB has a statistically significant negative correlation with Acc.AMB, which is expected because of how the bias score AMB is defined and calculated'), so it is not presented as an independent finding. There is no load-bearing self-citation: the authors' references are to independent prior work, and the one mechanistic explanation (Gemma safety-neuron placement) cites external work (Chen et al.). The acknowledged threats — LLaMA-Guard-2 being a Llama-3-8B model, and the Tukey-fences utility-outlier removal concentrated in LLaMA Prompt-Tuning (Appendix C) — are measurement/selection validity concerns raised in §VI.A ('worst-case degradation may be underestimated'), not circular reductions of the paper's derivation to its own inputs. Because the analysis is self-contained against external benchmarks and the conclusions are not forced by construction, no significant circularity is present.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The study is empirical; the ledger captures hand-chosen settings and benchmark assumptions that the conclusions rest on. No new entities are postulated.

free parameters (3)
  • LoRA hyperparameters (alpha, r, dropout) = alpha=16, r=4, dropout=0.1
    Taken from the most frequent values in mined HuggingFace adapter configs; not tuned for safety/fairness but affects update magnitude and could moderate alignment shifts.
  • Training sample fractions = 10% UltraChat (20,786) and 34% UltraFeedback (20,785)
    Chosen so both training sets have equal size under compute limits; results may not extend to full-scale training (acknowledged in Section VI.C).
  • Utility outlier removal threshold = Tukey's fences k=1.5 (19 models removed)
    A standard but hand-selected threshold; 13 of the removed models are Prompt-Tuning on Llama, changing the composition of the prompt-method group and potentially understating its average degradation.
axioms (4)
  • domain assumption HEx-PHI plus LLaMA-Guard-2 measures safety alignment.
    Sections III.D.1-III.D.2 adopt this benchmark and judge; if the judge mis-ranks model families, all safety comparisons shift.
  • domain assumption Modified BBQ-Lite measures social bias.
    Section III.D.3 and Appendix A: 200 items removed and 32 corrected; the edits are not externally validated.
  • domain assumption UltraChat and UltraFeedback are representative benign conversational datasets.
    Section III.B: chosen for HuggingFace popularity and prior use in Zephyr; residual biases may still affect fine-tuned behavior.
  • domain assumption Utility outliers can be excluded without biasing safety/fairness comparisons.
    Appendix C.A.2: the excluded models are mostly one method-model pair, so the remaining sample is not a random subset of all fine-tuned models.

pith-pipeline@v1.3.0-alltime-deepseek · 42324 in / 10212 out tokens · 92526 ms · 2026-08-04T06:50:25.492522+00:00 · methodology

0 comments
read the original abstract

Organizations increasingly adapt Large Language Models (LLMs) from public repositories such as HuggingFace to downstream tasks. Prior work shows that even fine-tuning on benign datasets can weaken safety alignment, raising a practical question: does benign parameter-efficient fine-tuning (PEFT) also affect safety and fairness? We present the first large-scale, systematic study showing that benign PEFT can significantly alter both. We fine-tune four instruction-tuned model families (Meta-Llama-3-8B, Qwen2.5-7B, Mistral-7B, and Gemma-7B) with four widely used PEFT methods: LoRA, IA3, Prompt-Tuning, and P-Tuning. In total, we evaluate 235 conversationally fine-tuned variants across eleven safety hazard categories and nine fairness dimensions. We assess generalization beyond conversational tuning by incorporating a compact extension focused on coding tasks, involving 96 additional fine-tuned models. Results show that benign PEFT can induce detrimental alignment shifts. Adapter-based methods (LoRA, IA3) are generally safer and less disruptive to fairness, whereas prompt-based methods more often reduce safety and worsen fairness accuracy. Base model choice strongly moderates these effects: LLaMA is comparatively stable, Qwen shows modest gains, Gemma exhibits the steepest safety decline, and Mistral is the most variable. The coding-task extension also produces alignment shifts relative to base models, but matched comparisons with the conversational task reveal limited task-level differences. Overall, safety improvements do not reliably transfer to fairness, and no single configuration optimizes every fairness metric. These findings support a practical guideline for safety-critical deployment: benign intent does not guarantee safe behaviour; start from a well-aligned base model, favour adapter-based PEFT, and audit safety and fairness at the category level.

Figures

Figures reproduced from arXiv: 2511.00382 by Amin Nikanjam, Foutse Khomh, Mina Taraghi, Mohamed Amine Merzouk, Yann Pequignot.

Figure 2
Figure 2. Figure 2: Distribution of safety changes for different PEFT methods [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Distribution of safety changes per peft method and base [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of safety changes across fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average change in safety per category for different fine-tuning variables [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Fairness scores of base models per category [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of changes in fairness metrics per each PEFT method (top) and base model (bottom) [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of fairness metrics for each PEFT method and each base model [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average change of fairness metrics per category per base model and per PEFT method [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Correlations for overall results, statistically significant [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

    cs.SE 2026-04 unverdicted novelty 5.0

    LLM judges for code tasks show high sensitivity to prompt biases that systematically favor certain options, changing accuracy and model rankings even when code is unchanged.

Reference graph

Works this paper leans on

114 extracted references · 25 linked inside Pith · cited by 1 Pith paper

  1. [1]

    V . B. Parthasarathy, A. Zafar, A. Khan, and A. Shahid, “The ultimate guide to fine-tuning LLMs from basics to breakthroughs: An exhaus- tive review of technologies, research, best practices, applied research challenges and opportunities,”arXiv preprint arXiv:2408.13296, Aug. 2024

    Pith/arXiv arXiv 2024

  2. [2]

    LLM agents for education: Advances and applications,

    Z. Chu, S. Wang, J. Xie, T. Zhu, Y . Yan, J. Ye, A. Zhong, X. Hu, J. Liang, P. S. Yuet al., “LLM agents for education: Advances and applications,”arXiv preprint arXiv:2503.11733, Mar. 2025

    arXiv 2025

  3. [3]

    Large Language Models for Chatbot Health Advice Studies: A Systematic Review,

    B. Huo, A. Boyle, N. Marfo, W. Tangamornsuksan, J. P. Steen, T. McKechnie, Y . Lee, J. Mayol, S. A. Antoniou, A. J. Thirunavukarasu et al., “Large Language Models for Chatbot Health Advice Studies: A Systematic Review,”JAMA Network Open, vol. 8, no. 2, p. e2457879, Feb. 2025

    2025

  4. [4]

    When scaling meets LLM finetuning: The effect of data, model and finetuning method,

    B. Zhang, Z. Liu, C. Cherry, and O. Firat, “When scaling meets LLM finetuning: The effect of data, model and finetuning method,” inProc. Int. Conf. Learn. Represent. (ICLR), Apr. 2024

    2024

  5. [5]

    Foundations of large language models,

    T. Xiao and J. Zhu, “Foundations of large language models,”arXiv preprint arXiv:2501.09223, Jan. 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Does fine-tuning LLMs on new knowledge encourage hallucinations?

    Z. Gekhman, G. Yona, R. Aharoni, M. Eyal, A. Feder, R. Reichart, and J. Herzig, “Does fine-tuning LLMs on new knowledge encourage hallucinations?” inProc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP), Nov. 2024, pp. 7765–7784

    2024

  7. [7]

    On the impact of fine-tuning on chain-of-thought reasoning,

    E. Lobo, C. Agarwal, and H. Lakkaraju, “On the impact of fine-tuning on chain-of-thought reasoning,”arXiv preprint arXiv:2411.15382, Nov. 2024

    Pith/arXiv arXiv 2024

  8. [8]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning,

    Y . Luo, Z. Yang, F. Meng, Y . Li, J. Zhou, and Y . Zhang, “An empirical study of catastrophic forgetting in large language models during continual fine-tuning,”arXiv preprint arXiv:2308.08747, Jan. 2025

    Pith/arXiv arXiv 2025

  9. [9]

    The janus interface: How fine-tuning in large language models amplifies privacy risks,

    X. Chen, S. Tang, R. Zhu, S. Yan, L. Jin, Z. Wang, L. Su, Z. Zhang, X. Wang, and H. Tang, “The janus interface: How fine-tuning in large language models amplifies privacy risks,” inProc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), Dec. 2024, pp. 1285–1299

    2024

  10. [10]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” inProc. Int. Conf. Learn. Represent. (ICLR), May 2024

    2024

  11. [11]

    Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models,

    F. Eiras, A. Petrov, P. H. S. Torr, M. P. Kumar, and A. Bibi, “Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models,” inProc. Int. Conf. Learn. Represent. (ICLR), Apr. 2025

    2025

  12. [12]

    LoRA fine-tuning efficiently undoes safety training in Llama 2-chat 70b,

    S. Lermen and C. Rogers-Smith, “LoRA fine-tuning efficiently undoes safety training in Llama 2-chat 70b,” inProc. ICLR Workshop on Secure and Trustworthy Large Language Models, Apr. 2024

    2024

  13. [13]

    Safe LoRA: The silver lining of reducing safety risks when fine-tuning large language models,

    C.-Y . Hsu, Y .-L. Tsai, C.-H. Lin, P.-Y . Chen, C.-M. Yu, and C.-Y . Huang, “Safe LoRA: The silver lining of reducing safety risks when fine-tuning large language models,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 37, Dec. 2024, pp. 65 072–65 094

    2024

  14. [14]

    LoRATK: LoRA once, backdoor everywhere in the share-and-play ecosystem,

    H. Liu, S. Zhong, X. Sun, M. Tian, M. Hariri, Z. Liu, R. Tang, Z. Jiang, J. Yuan, Y .-N. Chuanget al., “LoRATK: LoRA once, backdoor everywhere in the share-and-play ecosystem,”arXiv preprint arXiv:2403.00108, Mar. 2024

    Pith/arXiv arXiv 2024

  15. [15]

    On the vulnerability of safety alignment in open-access LLMs,

    J. Yi, R. Ye, Q. Chen, B. Zhu, S. Chen, D. Lian, G. Sun, X. Xie, and F. Wu, “On the vulnerability of safety alignment in open-access LLMs,” inFindings Assoc. Comput. Linguist. (ACL), Aug. 2024, pp. 9236–9260

    2024

  16. [16]

    huggingface (Hugging Face),

    HuggingFace, “huggingface (Hugging Face),” Jan. 2025. [Online]. Available: https://huggingface.co/huggingface

    2025

  17. [17]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inProc. Int. Conf. Learn. Represent. (ICLR), Apr. 2022

    2022

  18. [18]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, Dec. 2022, pp. 1950–1965

    2022

  19. [19]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” inProc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP), Nov. 2021, pp. 3045–3059

    2021

  20. [20]

    GPT understands, too,

    X. Liu, Y . Zheng, Z. Du, M. Ding, Y . Qian, Z. Yang, and J. Tang, “GPT understands, too,”AI Open, vol. 5, pp. 208–215, Jan. 2024

    2024

  21. [21]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, Jul. 2024

    Pith/arXiv arXiv 2024

  22. [22]

    meta-llama/Meta-Llama-3-8B-Instruct · Hugging Face — huggingface.co,

    Meta-Llama, “meta-llama/Meta-Llama-3-8B-Instruct · Hugging Face — huggingface.co,” https://huggingface.co/meta-llama/Meta-Llama-3 -8B-Instruct, [Accessed 24 February 2025]

    2025

  23. [23]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnieret al., “Mistral 7B,”arXiv preprint arXiv:2310.06825, Oct. 2023

    Pith/arXiv arXiv 2023

  24. [24]

    mistralai/Mistral-7B-Instruct-v0.3 · Hugging Face — hug- gingface.co,

    MistralAI, “mistralai/Mistral-7B-Instruct-v0.3 · Hugging Face — hug- gingface.co,” https://huggingface.co/mistralai/Mistral-7B-Instruct-v0. 3, [Accessed 24 February 2025]

    2025

  25. [25]

    Qwen2.5 technical report,

    Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huanget al., “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, Dec. 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Qwen/Qwen2.5-7B-Instruct · Hugging Face — hugging- face.co,

    Qwen, “Qwen/Qwen2.5-7B-Instruct · Hugging Face — hugging- face.co,” https://huggingface.co/Qwen/Qwen2.5- 7B- Instruct, [Accessed 24 February 2025]

    2025

  27. [27]

    Gemma: Open models based on Gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Loveet al., “Gemma: Open models based on Gemini research and technology,”arXiv preprint arXiv:2403.08295, Mar. 2024

    Pith/arXiv arXiv 2024

  28. [28]

    google/gemma-7b-it · Hugging Face — huggingface.co,

    Google, “google/gemma-7b-it · Hugging Face — huggingface.co,” http s://huggingface.co/google/gemma-7b-it, [Accessed 24 February 2025]

    2025

  29. [29]

    BBQ: A hand-built bias benchmark for question answering,

    A. Parrish, A. Chen, N. Nangia, V . Padmakumar, J. Phang, J. Thomp- son, P. M. Htut, and S. Bowman, “BBQ: A hand-built bias benchmark for question answering,” inFindings Assoc. Comput. Linguist. (ACL), May 2022, pp. 2086–2105

    2022

  30. [30]

    Universal language model fine-tuning for text classification,

    J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” inProc. 56th Annu. Meet. Assoc. Comput. Linguist. (ACL), Jul. 2018, pp. 328–339

    2018

  31. [31]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” inProc. Int. Conf. Learn. Represent. (ICLR), Apr. 2022

    2022

  32. [32]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, Dec. 2022, pp. 27 730–27 744

    2022

  33. [33]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousseet al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,”arXiv preprint arXiv:2209.07858, Sep. 2022

    Pith/arXiv arXiv 2022

  34. [34]

    Supervised Fine-Tuning - Hugging Face LLM Course

    HuggingFace, “Supervised Fine-Tuning - Hugging Face LLM Course.” [Online]. Available: https://huggingface.co/learn/llm-course/en/chapter 11/3

  35. [35]

    Training a helpful and harmless assistant with reinforcement learning from human feed- back,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighanet al., “Training a helpful and harmless assistant with reinforcement learning from human feed- back,”arXiv preprint arXiv:2204.05862, Apr. 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 36, Dec. 2023, pp. 53 728–53 741

    2023

  37. [37]

    Are smarter LLMs safer? exploring safety-reasoning trade-offs in prompting and fine-tuning,

    A. Li, Y . Mo, M. Li, Y . Wang, and Y . Wang, “Are smarter LLMs safer? exploring safety-reasoning trade-offs in prompting and fine-tuning,” arXiv preprint arXiv:2502.09673, Feb. 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Multitask-Bench: Unveiling and mitigating safety gaps in LLMs fine- tuning,

    E. Jan, N. Aldahoul, M. Ali, F. Ahmad, F. Zaffar, and Y . Zaki, “Multitask-Bench: Unveiling and mitigating safety gaps in LLMs fine- tuning,” inProc. Int. Conf. Comput. Linguist. (COLING), Jan. 2025, pp. 9025–9043

    2025

  39. [39]

    Parameter- Efficient Fine-Tuning for Large Models: A Comprehensive Survey,

    Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter- Efficient Fine-Tuning for Large Models: A Comprehensive Survey,” Transactions on Machine Learning Research, Jul. 2024

    2024

  40. [40]

    LLM Fine-Tuning: Concepts, Opportunities, and Challenges,

    X.-K. Wu, M. Chen, W. Li, R. Wang, L. Lu, J. Liu, K. Hwang, Y . Hao, Y . Pan, Q. Menget al., “LLM Fine-Tuning: Concepts, Opportunities, and Challenges,”Big Data and Cognitive Computing, vol. 9, no. 4, p. 87, 2025

    2025

  41. [41]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, Jul. 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 36, Dec. 2023, pp. 46 595–46 623

    2023

  43. [43]

    A survey of direct preference optimization,

    S. Liu, W. Fang, Z. Hu, J. Zhang, Y . Zhou, K. Zhang, R. Tu, T.-E. Lin, F. Huang, M. Songet al., “A survey of direct preference optimization,” arXiv preprint arXiv:2503.11701, Mar. 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Scaling down to scale up: A guide to parameter-efficient fine-tuning,

    V . Lialin, V . Deshpande, X. Yao, and A. Rumshisky, “Scaling down to scale up: A guide to parameter-efficient fine-tuning,”arXiv preprint arXiv:2303.15647, Mar. 2023. 30

    Pith/arXiv arXiv 2023

  45. [45]

    Safety alignment should be made more than just a few tokens deep,

    X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson, “Safety alignment should be made more than just a few tokens deep,” inProc. Int. Conf. Learn. Represent. (ICLR), Apr. 2025

    2025

  46. [46]

    Shadow alignment: The ease of subverting safely-aligned language models,

    X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y . Wang, X. Zhao, and D. Lin, “Shadow alignment: The ease of subverting safely-aligned language models,” arXiv preprint arXiv:2310.02949, Oct. 2023, accessed: 2025-10-08. [Online]. Available: http://arxiv.org/abs/2310.0 2949

    Pith/arXiv arXiv 2023

  47. [47]

    Keeping LLMs aligned after fine-tuning: The crucial role of prompt templates,

    K. Lyu, H. Zhao, X. Gu, D. Yu, A. Goyal, and S. Arora, “Keeping LLMs aligned after fine-tuning: The crucial role of prompt templates,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 37, Dec. 2024, pp. 118 603–118 631

    2024

  48. [48]

    ChatBug: A common vulnerability of aligned LLMs induced by chat templates,

    F. Jiang, Z. Xu, L. Niu, B. Y . Lin, and R. Poovendran, “ChatBug: A common vulnerability of aligned LLMs induced by chat templates,” in Proc. AAAI Conf. Artif. Intell. (AAAI), 2025, pp. 27 347–27 355

    2025

  49. [49]

    Emergent misalignment: Narrow fine-tuning can produce broadly misaligned LLMs,

    J. Betley, D. C. H. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans, “Emergent misalignment: Narrow fine-tuning can produce broadly misaligned LLMs,” inProc. Int. Conf. Mach. Learn. (ICML), 2025

    2025

  50. [50]

    GitHub - huggingface/huggingface hub: The official Python client for the Huggingface Hub

    huggingface.co, “GitHub - huggingface/huggingface hub: The official Python client for the Huggingface Hub.” https://github.com/huggingfa ce/huggingface hub, [Accessed: 24 February 2025]

    2025

  51. [51]

    Hub API Endpoints,

    ——, “Hub API Endpoints,” https://huggingface.co/docs/hub/api, [Accessed: 24 February 2025]

    2025

  52. [52]

    Qwen (Qwen),

    “Qwen (Qwen),” Sep. 2025. [Online]. Available: https://huggingface. co/Qwen

    2025

  53. [53]

    Tweet- Eval: Unified benchmark and comparative evaluation for tweet classi- fication,

    F. Barbieri, J. Camacho-Collados, L. E. Anke, and L. Neves, “Tweet- Eval: Unified benchmark and comparative evaluation for tweet classi- fication,” inFindings Assoc. Comput. Linguist. (EMNLP), Nov. 2020, pp. 1644–1650

    2020

  54. [54]

    cardiffnlp/tweet eval · Datasets at Hugging Face,

    “cardiffnlp/tweet eval · Datasets at Hugging Face,” Aug. 2025. [Online]. Available: https://huggingface.co/datasets/cardiffnlp/tweet e val

    2025

  55. [55]

    meta-llama (Meta Llama),

    “meta-llama (Meta Llama),” May 2025. [Online]. Available: https: //huggingface.co/meta-llama

    2025

  56. [56]

    HuggingFaceH4/ultrafeedback binarized · Datasets at Hugging Face — huggingface.co,

    HuggingFaceH4, “HuggingFaceH4/ultrafeedback binarized · Datasets at Hugging Face — huggingface.co,” https://huggingface.co/dataset s/HuggingFaceH4/ultrafeedback binarized, [Accessed 24 February 2025]

    2025

  57. [57]

    google (Google),

    “google (Google),” Feb. 2025. [Online]. Available: https://huggingfac e.co/google/models

    2025

  58. [58]

    Prefix-Tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-Tuning: Optimizing continuous prompts for generation,” inProc. 59th Annu. Meet. Assoc. Comput. Linguist. and 11th Int. Joint Conf. Nat. Lang. Process. (ACL-IJCNLP), Aug. 2021, pp. 4582–4597

    2021

  59. [59]

    Character-level convolutional networks for text classification,

    X. Zhang, J. Zhao, and Y . LeCun, “Character-level convolutional networks for text classification,” inAdv. Neural Inf. Process. Syst. (NeurIPS), vol. 28, Dec. 2015, pp. 649–657

    2015

  60. [60]

    fancyzhx/ag news · Datasets at Hugging Face,

    “fancyzhx/ag news · Datasets at Hugging Face,” Aug. 2024. [Online]. Available: https://huggingface.co/datasets/fancyzhx/ag news

    2024

  61. [61]

    mistralai (Mistral AI ),

    “mistralai (Mistral AI ),” Aug. 2025. [Online]. Available: https: //huggingface.co/mistralai/models

    2025

  62. [62]

    HuggingFaceH4/ultrachat 200k · Datasets at Hug- ging Face — huggingface.co,

    HuggingFaceH4, “HuggingFaceH4/ultrachat 200k · Datasets at Hug- ging Face — huggingface.co,” https://huggingface.co/datasets/Huggin gFaceH4/ultrachat 200k, [Accessed 24 February 2025]

    2025

  63. [63]

    openai-community/gpt2 · Hugging Face

    “openai-community/gpt2 · Hugging Face.” [Online]. Available: https://huggingface.co/openai-community/gpt2

  64. [64]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” inProc. EMNLP Workshop BlackboxNLP, Nov. 2018, pp. 353–355

    2018

  65. [65]

    nyu-mll/glue · Datasets at Hugging Face,

    “nyu-mll/glue · Datasets at Hugging Face,” Dec. 2023. [Online]. Available: https://huggingface.co/datasets/nyu-mll/glue

    2023

  66. [66]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wuet al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, Feb. 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Mistral 7B|Mistral AI

    “Mistral 7B|Mistral AI.” [Online]. Available: https://mistral.ai/news/ announcing-mistral-7b

  68. [68]

    mistralai/Mistral-7B-Instruct-v0.3 · Hugging Face

    “mistralai/Mistral-7B-Instruct-v0.3 · Hugging Face.” [Online]. Available: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

  69. [69]

    an issue of prefix tuning on llama3-8b · Issue #454 · meta-llama/llama- cookbook — github.com,

    “an issue of prefix tuning on llama3-8b · Issue #454 · meta-llama/llama- cookbook — github.com,” https://github.com/meta-llama/llama-coo kbook/issues/454#issuecomment-2089584315, [Accessed 24 February 2025]

    2025

  70. [70]

    Zephyr: Direct distillation of lm alignment,

    L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y . Belkada, S. Huang, L. V on Werra, C. Fourrier, N. Habibet al., “Zephyr: Direct distillation of lm alignment,”arXiv preprint arXiv:2310.16944, 2023

    Pith/arXiv arXiv 2023

  71. [71]

    ULTRAFEEDBACK: Boosting language models with scaled AI feedback,

    G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y . Ni, G. Xie, R. Xie, Y . Linet al., “ULTRAFEEDBACK: Boosting language models with scaled AI feedback,” inProc. Int. Conf. Mach. Learn. (ICML). PMLR, Jul. 2024, pp. 9722–9744

    2024

  72. [72]

    openbmb/UltraFeedback · Datasets at Hugging Face,

    OpenBMB, “openbmb/UltraFeedback · Datasets at Hugging Face,” ht tps://huggingface.co/datasets/openbmb/UltraFeedback, May 2025

    2025

  73. [73]

    Enhancing chat language models by scaling high-quality instructional conversations,

    N. Ding, Y . Chen, B. Xu, Y . Qin, S. Hu, Z. Liu, M. Sun, and B. Zhou, “Enhancing chat language models by scaling high-quality instructional conversations,” inProc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP), Dec. 2023, pp. 3029–3051

    2023

  74. [74]

    Introducing chatgpt,

    OpenAI, “Introducing chatgpt,” https://openai.com/blog/chatgpt, 2022, [Accessed 24 February 2025]

    2022

  75. [75]

    stingning/ultrachat · Datasets at Hugging Face,

    N. Ding, “stingning/ultrachat · Datasets at Hugging Face,” https://hu ggingface.co/datasets/stingning/ultrachat, Oct. 2023

    2023

  76. [76]

    TRL: Transformer Reinforcement Learning,

    L. von Werra, Y . Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lam- bert, S. Huang, K. Rasul, and Q. Gallou ´edec, “TRL: Transformer Reinforcement Learning,” https://github.com/huggingface/trl, 2020

    2020

  77. [77]

    PEFT: State-of-the-art Parameter-Efficient Fine-Tuning Methods,

    S. Mangrulkar, S. Gugger, L. Debut, Y . Belkada, S. Paul, and B. Bossan, “PEFT: State-of-the-art Parameter-Efficient Fine-Tuning Methods,” https://github.com/huggingface/peft, 2022

    2022

  78. [79]

    A survey on fairness in large language models,

    Y . Li, M. Du, R. Song, X. Wang, and Y . Wang, “A survey on fairness in large language models,”arXiv preprint arXiv:2308.10149, Aug. 2023

    Pith/arXiv arXiv 2023

  79. [80]

    Fairness in Large Language Models: A Taxonomic Survey,

    Z. Chu, Z. Wang, and W. Zhang, “Fairness in Large Language Models: A Taxonomic Survey,”ACM SIGKDD Explorations Newsletter, vol. 26, no. 1, pp. 34–48, Jul. 2024

    2024

  80. [81]

    Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models,

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonsoet al., “Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models,”Transactions on Machine Learning Research, vol. 2023, no. 5, pp. 1–95, 2023

    2023

Showing first 80 references.