Pith. sign in

REVIEW 31 cited by

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.12148 v1 pith:WU7WW4YZ submitted 2023-12-19 cs.CL

Lingling Xu , Haoran Xie , Si-Zhao Joe Qin , Xiaohui Tao , Fu Lee Wang This is my paper
classification cs.CL
keywords peftfine-tuningmethodslanguagemodelsplmsparametersapplications
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the continuous growth in the number of parameters of transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, we conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 31 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

    cs.LG 2026-02 conditional novelty 7.0 of 10

    In a Gaussian single-index model with one-pass SGD, the LoRA escape time scales as τ(μ) log d / 2, where τ(μ) increases with pre-training strength μ and diverges for odd Hermite activations at a critical μ.

  2. Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

    cs.CR 2026-04 unverdicted novelty 6.5 of 10

    Succinct zero-knowledge proofs can certify that a fine-tuned model differs from a base model only by norm-bounded, low-rank, or sparse parameter drift, with cost set by that structure rather than model size.

  3. Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation

    cs.CL 2025-11 conditional novelty 6.0 of 10

    Fine-tuning on speaker-attributed, action-tagged transcripts from public meetings lets LLM agents mimic government meeting participants well enough that human judges often cannot tell them from real people.

  4. Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Spatial understanding in multimodal LLMs plateaus quickly as training data grows, and position encoding in the visual encoder is the more influential factor.

  5. GRID: Scaling Task-Agnostic Inference in Continual Prompt Tuning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    GRID combines output-space constrained decoding with gradient-guided prompt compression for task-agnostic, bounded-memory continual prompt tuning.

  6. Implementing Adaptations for Vision AutoRegressive Model

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Fine-tuned Vision AutoRegressive models mostly beat a strong diffusion baseline on downstream image generation, but DP fine-tuning yields poor FID scores.

  7. Defending Against Prompt Injection With a Few DefensiveTokens

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Five optimized token embeddings prepended to the prompt reduce prompt-injection attack success to near zero on standard benchmarks while preserving most model utility.

  8. LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Low-rank extrapolation of an aligned model's weight update (LoX) reduces how much later fine-tuning erodes safety refusal behavior.

  9. Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

    cs.CL 2026-07 conditional novelty 5.0 of 10

    In medication-leaflet generation, ORPO and GRPO beat SFT and a zero-shot GPT-5 baseline, but GRPO's cross-domain advantage is partly built into its reward functions.

  10. Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

    cs.CL 2026-01 conditional novelty 5.0 of 10

    Fine-tuned LLMs can reconstruct item characteristic curves from multiple-choice item text, giving useful estimates of IRT difficulty and discrimination without live student response data.

  11. Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks

    cs.AI 2025-10 conditional novelty 5.0 of 10

    Fine-tuning an LLM on synthetic toxic dialogues makes it harass in 95–97% of multi-turn conversations in Llama and ~99% in Gemini; memory and planning attacks also raise closed-source vulnerability.

  12. PB-IAD: Utilizing multimodal foundation models for semantic industrial anomaly detection in dynamic manufacturing environments

    cs.CV 2025-08 conditional novelty 5.0 of 10

    With carefully layered prompts and one or three reference samples, GPT-4.1 detects anomalies in cable images and crimp-force features at F1 levels that PatchCore and Isolation Forest reach only after training on dozen...

  13. Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    SKD-CAG erases adversarial text triggers from diffusion models by distilling the model's own clean outputs through cross-attention guidance, claiming 100% and 93% removal for pixel and style backdoors.

  14. CIgrate: Automating CI Service Migration with Large Language Models

    cs.SE 2025-07 reject novelty 5.0 of 10

    The abstract reports that a fine-tuned Gemma 3 12B outperforms the rule-based CIMig tool on Travis-to-GitHub Actions CI migration, but the manuscript body contains only a planned study with no experimental results.

  15. The Impact of Fine-tuning Large Language Models on Automated Program Repair

    cs.SE 2025-07 conditional novelty 5.0 of 10

    On three Java APR benchmarks, LoRA and IA3 adapters match or beat full-model fine-tuning for most tested code LLMs while training less than one percent of parameters.

  16. Enhancing RLHF with Human Gaze Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Using predicted human gaze to shape rewards lets RLHF converge 1.3 to 2 times faster with similar final policy quality.

  17. Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CorDA++ uses data-driven SVD to initialize LoRA adapters, adding per-layer covariance selection and rank allocation that reduce forgetting and speed convergence compared to LoRA, PiSSA, QLoRA, and other baselines.

  18. SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks

    cs.CR 2025-06 conditional novelty 5.0 of 10

    SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.

  19. Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Kvasir-VQA-x1 expands Kvasir-VQA with 159,549 LLM-generated question-answer pairs stratified into three complexity levels, plus a robustness track using weakly augmented images.

  20. Meta-Learning and Synthetic Data for Automated Pretraining and Finetuning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A cumulative dissertation demonstrating meta-learned surrogates and synthetic data generators for automated model selection, finetuning, augmentation, and reinforcement-learning environments.

  21. Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification

    cs.LG 2025-06 conditional novelty 5.0 of 10

    TensorGuard classifies fine-tuned LLMs into their base-model families with 94% accuracy by clustering statistical features of weight gradients under random input perturbations.

  22. Schema as Parameterized Tools for Universal Information Extraction

    cs.CL 2025-06 conditional novelty 5.0 of 10

    SPT turns predefined IE schemas into trainable token embeddings, enabling one small model to handle closed, open, and on-demand extraction through schema retrieval, generation, and infilling.

  23. Continual Learning in Vision-Language Models via Aligned Model Merging

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PAM merges a task-specific LoRA into a global LoRA and re-initializes sign-conflicting weights during training, reducing catastrophic forgetting in continual VLM learning.

  24. Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models

    cs.AI 2025-05 conditional novelty 5.0 of 10

    ProLoRA transfers pre-trained LoRA, DoRA, and FouRA adapters between diffusion models in a single closed-form projection step, without retraining on the target model.

  25. Fine-tuning on simulated data outperforms prompting for agent tone of voice

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Fine-tuning a 1B-parameter LLM on as few as 100 synthetically generated, readability-filtered samples achieved conversational tone more reliably than a verbose system prompt.

  26. QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation

    cs.CV 2025-07 conditional novelty 4.0 of 10

    QR-LoRA freezes the QR-decomposed basis of pretrained weights, trains only a residual matrix, and reports halved trainable parameters with improved content-style disentanglement in diffusion models.

  27. Reconstructing Biological Pathways by Applying Selective Incremental Learning to (Very) Small Language Models

    q-bio.MN 2025-07 conditional novelty 4.0 of 10

    A small BERT model trained on only 117 of 517 curated regulatory relationships selected as confident errors reaches 93% balanced accuracy, outperforming a policy that also includes uncertain correct examples.

  28. The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A vector-search segmenter plus fine-tuned LLaMA models outperforms regex for legal NER, with LoRA fine-tuning sharply reducing hallucinations.

  29. Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Cuff-KT generates personalized output-layer parameters for knowledge tracing models without fine-tuning, reporting AUC improvements of about 10% and 4% under intra- and inter-learner shifts.

  30. MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors

    cs.CL 2025-05 conditional novelty 4.0 of 10

    MSA-MathEval, a LoRA-tuned Mathstral-7B ensemble with disagreement-aware inference, ranked 1st in Providing Guidance and top-5 in all tracks at BEA 2025.

  31. DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

    cs.CR 2025-09 reject novelty 3.0 of 10

    DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.

Pith tools