Pith. sign in

REVIEW 17 cited by

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.15647 v2 pith:6K5T6HIQ submitted 2023-03-28 cs.CL

Vladislav Lialin , Vijeta Deshpande , Xiaowei Yao , Anna Rumshisky This is my paper
classification cs.CL
keywords methodsfine-tuningmodelscomparisonefficiencylanguageonlyparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper presents a systematic overview of parameter-efficient fine-tuning methods, covering over 50 papers published between early 2019 and mid-2024. These methods aim to address the challenges of fine-tuning large language models by training only a small subset of parameters. We provide a taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency in fine-tuning multibillion-scale language models. We also conduct an extensive head-to-head experimental comparison of 15 diverse PEFT methods, evaluating their performance and efficiency on models up to 11B parameters. Our findings reveal that methods previously shown to surpass a strong LoRA baseline face difficulties in resource-constrained settings, where hyperparameter optimization is limited and the network is fine-tuned only for a few epochs. Finally, we provide a set of practical recommendations for using PEFT methods and outline potential future research directions.

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prompt engineering using order-of-addition experiments: An application to generating two-level fractional factorial designs

    stat.AP 2026-07 accept novelty 7.0 of 10

    Order-of-addition designs and logistic pairwise-ordering models measure and optimize prompt-element order, lifting LLM success on 16-run fractional factorial design tasks from low teens or mid-thirties to near 100%.

  2. Can Gradient Descent Simulate Prompting?

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A MAML-style meta-training objective makes a single gradient step on new text recover part of the performance that prompting achieves, on reversal-curse and passage-QA tasks.

  3. One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization

    cs.SD 2026-01 unverdicted novelty 6.0 of 10

    LLMs using in-context learning and fine-tuning on listener experiment data generate equalization settings that align better with population preferences than random sampling or static presets.

  4. Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs

    cs.AI 2025-11 conditional novelty 6.0 of 10

    Benign PEFT fine-tuning changes LLM safety and fairness: adapter-based methods (LoRA, IA3) preserve alignment better than prompt-based methods, and the base model strongly moderates outcomes.

  5. CALM: A Framework for Continuous, Adaptive, and LLM-Mediated Anomaly Detection in Time-Series Streams

    cs.LG 2025-08 reject novelty 6.0 of 10

    CALM uses an LLM-as-a-Judge to curate anomalies for continuous fine-tuning of a time-series foundation model, improving anomaly detection on held-out stream segments.

  6. AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A PPO-plus-diffusion policy adaptively allocates LoRA ranks per layer based on channel SNR and data complexity, improving accuracy by up to 0.69% and cutting transmitted parameters by 12.5% over AdaLoRA.

  7. Time Series Foundation Models for Multivariate Financial Time Series Forecasting

    q-fin.GN 2025-07 reject novelty 6.0 of 10

    Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...

  8. Parameter-Efficient Fine-Tuning of Foundation Models for CLP Speech Classification

    eess.AS 2025-07 conditional novelty 5.0 of 10

    Parameter-efficient fine-tuning (LoRA/DoRA) of Whisper improves cleft lip and palate severity classification over frozen embeddings and handcrafted features on two child speech corpora.

  9. Enhancing RLHF with Human Gaze Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Using predicted human gaze to shape rewards lets RLHF converge 1.3 to 2 times faster with similar final policy quality.

  10. Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches

    cs.CL 2025-12 unverdicted novelty 4.0 of 10

    Embedding-based QLoRA fine-tuning of causal LLMs matches BERT on single-label patent classification with 10–30x fewer trainable parameters, while instruction-tuning wins on multi-label classification only with ≥100M t...

  11. LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning

    cs.CV 2025-06 reject novelty 4.0 of 10

    LARGO adds per-matrix learned projection radii and SVD-based initialization to LoRA and reports improved out-of-distribution accuracy on DomainNet, ImageNet, and VQA benchmarks.

  12. Large Language Models for Detection of Life-Threatening Texts

    cs.CL 2025-06 conditional novelty 4.0 of 10

    LoRA-fine-tuned Mistral and Llama-2 (7B) outperform traditional text classifiers for life-threatening text detection across six datasets; upsampling helps traditional methods but not these LLMs.

  13. Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims PEFT is a strict, less robust, lower-capacity subset of full fine-tuning, but the mathematical proofs contain load-bearing errors and the experiments, while suggestive, cannot repair them.

  14. 15,500 Seconds: Lean UAV Classification Using EfficientNet and Lightweight Fine-Tuning

    cs.LG 2025-05 reject novelty 4.0 of 10

    On a private 3,100-clip, 31-class drone audio dataset, full fine-tuning of EfficientNet-B0 with three augmentations reached 95.95% validation accuracy, the best of all compared models and PEFT methods.

  15. CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generation

    cs.AI 2025-07 reject novelty 3.0 of 10

    A 7B model trained with GRPO and a sparse execution-correctness reward reaches 59.97% execution accuracy on BIRD dev, though the evaluation protocol and baseline numbers contain inconsistencies.

  16. Optimising Language Models for Downstream Tasks: A Post-Training Perspective

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.

  17. Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

    cs.CL 2025-06 reject novelty 2.0 of 10

    A progressive scheduling trick that updates only the last remaining blocks in later epochs reduces parameter-update counts by about 25% with roughly unchanged GLUE and SQuAD scores.

Pith tools