Pith. sign in

REVIEW 24 cited by

Towards a Unified View of Parameter-Efficient Transfer Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.04366 v3 pith:EYCV5NY6 submitted 2021-10-08 cs.CL cs.LG

Junxian He , Chunting Zhou , Xuezhe Ma , Taylor Berg-Kirkpatrick , Graham Neubig This is my paper
classification cs.CLcs.LG
keywords methodsdesignlearningparameter-efficientparameterstransferunifiedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

  2. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  3. RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation

    cs.CV 2026-07 reject novelty 6.0 of 10

    Training-free LoRA rank allocation via virtual low-rank noise probing that scores modules by Fréchet distance and effective rank of activation shifts outperforms prior heuristic and training-based allocators on standa...

  4. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  5. CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    CeRA overcomes LoRA's linear ceiling by injecting non-linear SiLU gating and dropout, outperforming high-rank LoRA on complex math reasoning with 1/8 the parameters.

  6. PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.

  7. MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts

    cs.AI 2025-10 conditional novelty 6.0 of 10

    MHA-RAG encodes retrieved exemplars into order-invariant soft prompts via multi-head attention, claiming ~20-point effective-accuracy gains over RAG at ~10x lower inference FLOPs.

  8. PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.

  9. Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ALTA adapts a frozen masked-pretrained X-ray encoder to language with 8% trainable parameters and temporal-multiview inputs, improving medical retrieval and zero-shot classification.

  10. GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    GaRA-SAM improves SAM's robustness to image corruption by using input-dependent gating to adjust the effective rank of low-rank adapters, beating prior methods on robust segmentation benchmarks.

  11. iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection

    cs.CV 2025-05 conditional novelty 6.0 of 10

    iDPA improves incremental medical object detection by generating instance-level prompts from bounding-box regions and decoupling prompt attention in a frozen GLIP model.

  12. Improved Representation Steering for Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RePS, a reference-free bidirectional preference optimization objective, improves representation steering and suppression for Gemma models, outperforming language-modeling objectives and approaching prompting performance.

  13. AI-driven Remote Facial Skin Hydration and TEWL Assessment from Selfie Images: A Systematic Solution

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Selfie images can be used to predict skin hydration and water loss with R2 up to about 0.35, using a new dataset of 336 panelists and an adapter-based vision transformer.

  14. PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An iterative hybrid pruning method selects which PEFT modules to keep at each transformer layer, matching or improving fixed PEFT baselines on GLUE at 1% trainable parameters.

  15. A Systematic Review of Poisoning Attacks Against Large Language Models

    cs.CR 2025-06 conditional novelty 5.0 of 10

    A systematic review that organizes 65 LLM poisoning papers into a threat model with four attack specifications and generalized metrics.

  16. SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods

    cs.CL 2025-05 conditional novelty 5.0 of 10

    The authors release sense-annotated WSD/WiC datasets for ten low-resource languages and report that English-based zero-shot transfer often beats small in-language fine-tuning, while mixed training usually helps.

  17. HOFT: Householder Orthogonal Fine-tuning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    HOFT and SHOFT fine-tune foundation models with two Householder-built orthogonal matrices, matching or beating LoRA, DoRA, OFT, BOFT and HRA on reasoning, translation, image generation and math.

  18. Capacity and Redundancy Trade-offs in Multi-Task Learning

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A shared representation's total per-task information is bounded by capacity plus label redundancy; clustered sharing wins exactly when interference reduction exceeds the redundancy it loses.

  19. Weight Spectra Induced Efficient Model Adaptation

    cs.LG 2025-05 reject novelty 4.0 of 10

    Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.

  20. MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.

  21. Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Cuff-KT generates personalized output-layer parameters for knowledge tracing models without fine-tuning, reporting AUC improvements of about 10% and 4% under intra- and inter-learner shifts.

  22. Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MoLA adapts a pre-trained short-horizon forecaster to multiple forecast steps via segment-specific mixtures of shared low-rank adapters, reporting modest mean-squared-error gains over the base models on most of eight ...

  23. Learning Text Styles: A Study on Transfer, Attribution, and Verification

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A thesis compiles published work claiming that lightweight adapters, contrastive disentanglement, and instruction tuning improve text style transfer, authorship attribution, and authorship verification.

  24. Slimming Down LLMs Without Losing Their Minds

    cs.CL 2025-06 reject novelty 2.0 of 10

    QLoRA fine-tuning on Alpaca data drops a 1B model's math accuracy from 33.5% to 3.7% and reduces MMLU-CS accuracy by 13 points, while HellaSwag stays flat.

Pith tools