Pith. sign in

REVIEW 4 cited by

DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.17357 v3 pith:VZBV5DNC submitted 2024-05-27 cs.CL

Yulong Mao , Kaiyu Huang , Changhao Guan , Ganglin Bao , Fengran Mo , Jinan Xu This is my paper · ORCID
classification cs.CL
keywords dorafine-tuningbudgetparameterdynamicloraadaptationlow-rank
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at https://github.com/MIkumikumi0116/DoRA

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Block-wise and cell-wise LoRA-MoE routers break static LoRA gradient conflicts on multi-context matrix tasks, with cell-level gates matching a global router on uniform shifts and beating it on heterogeneous ones.

  2. Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.

  3. DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection

    cs.CV 2025-02 reject novelty 4.0 of 10

    DiffoRA selects a subset of modules for LoRA fine-tuning using a learned binary mask, reporting improved accuracy over standard LoRA on GLUE and SQuAD.

  4. Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

    cs.CL 2025-06 reject novelty 2.0 of 10

    A progressive scheduling trick that updates only the last remaining blocks in later epochs reduces parameter-update counts by about 25% with roughly unchanged GLUE and SQuAD scores.

Pith tools