REVIEW 7 cited by
Parameter-Efficient Fine-Tuning for Foundation Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at \url{https://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models}.
Forward citations
Cited by 7 Pith papers
-
GoodVibe: Security-by-Vibe for LLM-Based Code Generation
Retraining only the neurons most responsive to security judgments raises a code LLM's default safe-code rate to 87.5% on C++ (average) with about 1.9 million trainable parameters.
-
Vision Transformer Finetuning Benefits from Non-Smooth Components
For vision transformers, components with higher input-output sensitivity (attention and feedforward layers) yield better and more stable fine-tuning accuracy than smoother LayerNorm components.
-
CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
A Fisher-information-guided dynamic selection over a toolbox of LoRA, adapter, and frequency-adapter modules improves cross-domain remote sensing segmentation over static PEFT methods.
-
Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
On consumer GPUs, LoRA+ gives the best energy-focused fine-tuning score in 19 of 24 small-model task configurations, while QLoRA wins the memory-focused score when peak VRAM is the binding constraint.
-
Taming LLMs by Scaling Learning Rates with Gradient Grouping
An optimizer wrapper that clusters per-layer momentum and scales learning rates by cluster-wise median deviations improves perplexity and accuracy across LLM and MLLM training, and lets LoRA pretraining approach full-...
-
A Numerical PDEs Approach to Evolution Equations in Shape Analysis Based on Regularized Morphoelasticity
Regularized morphoelasticity yields a high-order elliptic system for continuous shape evolution that is solved by mixed finite elements in FEniCSx within an LDDMM-style optimal-control growth model.
-
C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning
Adding zeroth- and first-order flatness penalties to continual learning losses yields small consistent accuracy gains across seven methods, with the gated C-Flat++ variant at roughly 30% of the update cost.
Discussion (0). Sign in to comment.