Pith. sign in

REVIEW 7 cited by

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.06904 v2 pith:QYGOUEPQ submitted 2022-03-14 cs.CL cs.AIcs.LG

Ning Ding , Yujia Qin , Guang Yang , Fuchao Wei , Zonghan Yang , Yusheng Su , Shengding Hu , Yulin Chen
show 12 more authors
Chi-Min Chan Weize Chen Jing Yi Weilin Zhao Xiaozhi Wang Zhiyuan Liu Hai-Tao Zheng Jianfei Chen Yang Liu Jie Tang Juanzi Li Maosong Sun
This is my paper
classification cs.CLcs.AIcs.LG
keywords deltatuningmethodsfine-tuningplmsadaptationapproachescomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite the success, the process of fine-tuning large-scale PLMs brings prohibitive adaptation costs. In fact, fine-tuning all the parameters of a colossal model and retaining separate instances for different tasks are practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this paper. In contrast with the standard fine-tuning, delta tuning only fine-tunes a small portion of the model parameters while keeping the rest untouched, largely reducing both the computation and storage costs. Recent studies have demonstrated that a series of delta tuning methods with distinct tuned parameter selection could achieve performance on a par with full-parameter fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In this paper, we first formally describe the problem of delta tuning and then comprehensively review recent delta tuning approaches. We also propose a unified categorization criterion that divide existing delta tuning methods into three groups: addition-based, specification-based, and reparameterization-based methods. Though initially proposed as an efficient method to steer large models, we believe that some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. To this end, we discuss the theoretical principles underlying the effectiveness of delta tuning and propose frameworks to interpret delta tuning from the perspective of optimization and optimal control, respectively. Furthermore, we provide a holistic empirical study of representative methods, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. The experimental results also cover the analysis of combinatorial, scaling and transferable properties of delta tuning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Z-PEFT: Zero-shot Backdoor Detection in Parameter-Efficient Fine-Tuning via Canonical Spectral Signatures

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Head-wise spectral statistics of LoRA query/value updates plus logistic regression detect backdoored PEFT adapters, improving zero-shot transfer across attacks at a fraction of PEFTGuard's cost.

  2. K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    K-Merge merges each incoming LoRA with its most similar stored adapter using an order-invariant running average, governed by a storage budget and (in K-Merge++) a similarity threshold, preserving task performance with...

  3. PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter

    cs.CV 2025-05 conditional novelty 6.0 of 10

    PMA adapts frozen point cloud models by ordering and fusing all intermediate layer features with Mamba, achieving parameter-efficient gains on ScanObjectNN, ModelNet40, and ShapeNetPart.

  4. Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 5.5 of 10

    A monolithic multimodal LLM that cuts pre-training data by 58% and first-token latency by up to 69% while matching or beating its predecessor on 15 benchmarks.

  5. Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters

    stat.ML 2025-06 reject novelty 5.0 of 10

    The paper proves an upper bound of about sqrt(r/N) on the LoRA generalization gap and claims a matching lower bound, but both proofs contain structural gaps.

  6. Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Fed-HeLLo allocates different LoRA layers to clients of different resource levels using importance scores and geometric patterns, improving federated fine-tuning accuracy over random allocation baselines.

  7. LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs

    cs.LG 2025-07 conditional novelty 4.0 of 10

    LoRA adapters for new tasks can be generated on a CPU by softmin-weighted mixing of a large bank of pre-trained adapters, using dataset distribution distances as weights.

Pith tools