Pith. sign in

REVIEW 16 cited by

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.01933 v3 pith:3IK7DXWL submitted 2023-04-04 cs.CL

Zhiqiang Hu , Lei Wang , Yihuai Lan , Wanyu Xu , Ee-Peng Lim , Lidong Bing , Xing Xu , Soujanya Poria
show 1 more author
Roy Ka-Wei Lee
This is my paper
classification cs.CL
keywords llmsmethodsadapter-basedadaptersfine-tuningpeftreasoningadapter
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLM-Adapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks. The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  2. Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

    cs.LG 2026-01 reject novelty 6.0 of 10

    SALR combines static pruning of frozen weights with a trainable truncated-SVD low-rank residual adapter to match LoRA accuracy at 50% sparsity, cutting model size ~2x and giving ~1.7x inference speedup.

  3. Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product

    cs.LG 2025-08 conditional novelty 6.0 of 10

    KRAdapter, a Khatri-Rao product adapter, produces full-rank high-effective-rank weight updates for parameter-efficient fine-tuning and reports improved out-of-distribution performance over LoRA and other full-rank PEF...

  4. Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    IBDR couples Bayesian LoRA fine-tuning with a diversity-promoting divergence loss and Wasserstein distributional robustness, improving average ensemble accuracy on VTAB-1K and commonsense reasoning benchmarks.

  5. Advantageous Parameter Expansion Training Makes Better Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    APEX expands high-activation parameters into low-activation ones during LLM training and reports gains, but the continued pre-training evidence is confounded by different learning rate schedules.

  6. Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution

    cs.AI 2025-05 unverdicted novelty 6.0 of 10

    SEFT fine-tunes sparse LLMs by dynamically growing and pruning a sparse set of weight updates, preserving target sparsity while improving downstream task performance over sparsity-preserving baselines.

  7. RTTC: Reward-Guided Collaborative Test-Time Compute

    cs.CL 2025-08 reject novelty 5.0 of 10

    RTTC uses a reward model to route each query to direct inference, retrieval-augmented generation, or test-time training, plus caching to cut redundant work.

  8. HuggingGraph: Understanding the Supply Chain of LLM Ecosystem

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A directed heterogeneous graph of 402,654 Hugging Face models and datasets is constructed and analyzed to reveal supply-chain dependencies and structural patterns such as a connected core and heavy-tailed reuse.

  9. Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    CRFT selects critical internal representations via attention and saliency scores and fine-tunes only them, improving GSM8K accuracy over ReFT from 29.0% to 32.8% on LLaMA-2-7B.

  10. Taming LLMs by Scaling Learning Rates with Gradient Grouping

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An optimizer wrapper that clusters per-layer momentum and scales learning rates by cluster-wise median deviations improves perplexity and accuracy across LLM and MLLM training, and lets LoRA pretraining approach full-...

  11. On the Generalization vs Fidelity Paradox in Knowledge Distillation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Knowledge distillation improves small language models by up to 10% on reasoning tasks, but the claimed reasoning-fidelity paradox is not well supported by the BLEU-based metric used.

  12. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  13. Weight Spectra Induced Efficient Model Adaptation

    cs.LG 2025-05 reject novelty 4.0 of 10

    Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.

  14. MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.

  15. Learning Text Styles: A Study on Transfer, Attribution, and Verification

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A thesis compiles published work claiming that lightweight adapters, contrastive disentanglement, and instruction tuning improve text style transfer, authorship attribution, and authorship verification.

  16. Open-Vocabulary Object Detection in UAV Imagery: A Review and Future Perspectives

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A structured review that divides aerial open-vocabulary detection methods into pseudo-labeling and CLIP-driven integration families and catalogs the missing benchmarks in the field.

Pith tools