Pith. sign in

REVIEW 4 major objections 6 minor 64 references

DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DiffoRA claims that learning which modules to adapt—via a Differential Adaptation Matrix—consistently beats LoRA, AdaLoRA, and adapter baselines on GLUE and SQuAD, with a theoretical argument that selection improves convergence and…

desk verdict Module selection for LoRA is plausible empirically, but the theoretical proof assumes the PSD condition it needs to prove, and the CoLA headline is within noise. read the letter →

arxiv 2502.08905 v2 pith:EKIBJMN4 submitted 2025-02-13 cs.CV

Tangyu Jiang , Haodi Wang , Chun Yuan This is my paper · ORCID
classification cs.CV MSC 68T0768T50
keywords parameter-efficientfine-tuninglow-rankadaptationmoduleselectiondifferentialmatrixGramconvergencerateGLUESQuAD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiffoRA argues that during parameter-efficient fine-tuning, most of the modules in a pre-trained model should be left untouched, and only the essential ones adapted. The paper proposes learning a Differential Adaptation Matrix (DAM) that, for each layer and module, learns a weight deciding whether that module's LoRA adapter should be active, then binarizes the weights and fine-tunes only the selected modules. It claims this consistently beats LoRA, AdaLoRA, and adapter baselines on GLUE and SQuAD—for instance a 1.51% gain on CoLA over the previous best—and that selection also explains a theoretical convergence and generalization improvement through the minimum eigenvalue of the network's Gram matrix. If right, it turns module selection itself into a learnable, theoretically motivated component of PEFT.

What carries the argument

The Differential Adaptation Matrix (DAM), denoted Γ, is a per-module binary gate applied element-wise to the LoRA update; the paper analyzes an element-wise formulation where γr ◦ wr enters the ReLU network. The argument runs through the Gram matrix H∞Γ,w0 of a single-hidden-layer ReLU network in the over-parameterized regime: Theorem 1 claims the mask raises the minimum eigenvalue (λΓ0 ≥ λ0) via the Weyl inequality, which yields faster convergence and better generalization, and the practical construction uses continuous relaxation to [0,1], top-K discretization per layer, and weight-sharing for unselected modules to handle near-uniform relaxed weights.

What would settle it

Compute, on a real DeBERTa or RoBERTa model after DiffoRA's second stage, the minimum eigenvalue of the Gram matrix with the learned binary mask and without it (using the paper's Equation 6 and 7); if λΓ0 ≥ λ0 fails for any dataset, Theorem 1 collapses. Alternatively, ablate the learned selection against random module selection with the same K and weight-sharing: if random selection matches DiffoRA's numbers, the learned DAM is not the source of the gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that not every module needs a low-rank adapter, and that learning which modules to adapt improves both accuracy and parameter efficiency over methods that adapt all modules or only vary their ranks. Its proposal, DiffoRA, learns a Differential Adaptation Matrix (DAM) via continuous relaxation, selects the top-K entries per layer, and then fine-tunes only those modules, with unselected modules sharing weights across layers to avoid discarding useful signal. The authors also claim two theoretical results: that the minimum eigenvalue of the Gram matrix with the selection mask is at least as large as without it, giving a faster convergence rate, and that this lowers the generalization bound; both rest on the assumption that the mask makes the expected activation-indicator matrix positive semidefinite relative to the unmasked one. Empirically, DiffoRA reports consistent gains over LoRA, AdaLoRA, and adapter baselines across all eight GLUE tasks, both SQuAD versions, and LLaMA backbones, matching or beating full fine-tuning while updating as little as 0.08% of parameters.

Load-bearing premise

The theory assumes that masking off some modules can only improve the Gram matrix, meaning the condition IΓw − Iw ⪰ 0 always holds for the masks the method actually produces; this is exactly the benefit the paper is trying to prove, and it is never verified for the discrete top-K masks, while the single-hidden-layer NTK analysis is assumed to carry over to deep transformers without argument.

Editorial extensions

If this is right

  • DiffoRA's selections are task- and backbone-specific, yet consistently improve over all-module LoRA, implying module relevance is not uniform and can be learned.
  • Because unselected modules share weights, the method adds no parameter overhead while still letting ambiguous modules contribute to fine-tuning.
  • The gains hold at small parameter budgets: on SQuADv1.1 DiffoRA surpasses full fine-tuning while updating only 0.08% of parameters.
  • Applying DiffoRA on top of LoRA+ also improves over LoRA+, suggesting selection is complementary to rank-optimization and learning-rate schemes.
  • The theory predicts the selected mask should improve the Gram matrix conditioning, which is testable directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not measure Gram-matrix eigenvalues on actual models; a direct computation on a trained DiffoRA mask would test whether λΓ0 ≥ λ0 holds in practice and whether the continuous relaxation lands in the regime assumed by the proof.
  • If module selection is the active ingredient, the top-K mask may transfer across tasks with similar structure (e.g., from CoLA to SST-2), which the paper does not study but could be verified with the released code.
  • The selection view could be combined with rank-allocation methods (picking modules and then varying their interior ranks), which the paper keeps separate.
  • The weight-sharing fallback suggests that when the relaxed weights are nearly uniform, selection alone is not enough—a testable hypothesis about which datasets need the fallback.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DiffoRA, a parameter-efficient fine-tuning scheme built on LoRA that learns a Differentiable Adaptation Matrix (DAM) to select which modules (e.g., QKV projections, dense layers) should receive low-rank updates. The mask is obtained by continuous relaxation followed by top-K binarization, with a weight-sharing strategy for unselected modules. The authors claim that the selective matrix increases the minimum eigenvalue of the associated Gram matrix, thereby improving convergence rate and generalization (Theorems 1 and 2), and they report state-of-the-art results on GLUE, SQuAD, and LLaMA benchmarks. The manuscript includes an anonymous code link, detailed training settings, and a proof appendix.

Significance. If validated, the idea of selecting which modules to adapt, rather than only adjusting ranks inside every LoRA module, would be a useful and broadly applicable contribution to PEFT. The experimental setup is reasonable, covering GLUE, SQuAD, and LLaMA-3.2-1B/7B, and the paper ships code and hyperparameter details. However, the central theoretical result is not established: Theorem 1 relies entirely on an unverified and in fact false-for-feasible-masks PSD premise, and the headline empirical comparison on CoLA is within one standard deviation of the reported DiffoRA spread. As written, the paper's claims are stronger than the evidence supports.

major comments (4)
  1. [Section 3.2, Theorem 1; Appendix A.1, Eqs. (17)-(19)] The proof that λΓ_0 ≥ λ0 assumes I^Γ_w − I_w ⪰ 0, but this premise is neither proved for the optimized Γ nor verified for the binarized Γ used in the experiments. It is not a harmless regularity condition: for the paper's own single-hidden-layer ReLU model, take d=1, x=(1,2), w0=-1, γ=0.5; then each entry of I^Γ_w − I_w is negative (e.g., the diagonal entries equal P(w≥2)−P(w≥1) ≈ −0.136), so I^Γ_w − I_w is not PSD. Since the only link from Γ to convergence is this PSD difference via Weyl's inequality, the conclusion in Eq. (8) is unsupported for the masks DiffoRA actually constructs. The hidden-node requirement m = Ω(n^6d^2/(λΓ_0)^4δ^3) compounds the problem by depending on the eigenvalue the theorem is supposed to bound.
  2. [Section 3.2, Theorem 2 and Eq. (9)] Theorem 2 inherits the unverified premise λΓ_0 ≥ λ0, and the generalization bound depends inversely on λΓ_0; without a lower bound on λΓ_0, the existence of m* and the stated O(1/λΓ_0) rates are not established. In addition, the paper provides no argument that the single-hidden-layer ReLU NTK framework of Du et al. transfers to the multi-layer Transformer modules (WQ, WK, WV, WI, WO, WD) to which the theory is invoked, so the theoretical explanation of the empirical gains remains unsupported even apart from the PSD issue.
  3. [Table 1] The headline CoLA gain of 1.51% over AdaLoRA is not statistically supported: DiffoRA is reported as 71.55 ± 1.33, so the difference is roughly one standard deviation of the reported measure, and no error bars are given for AdaLoRA or any other baseline. The '↑0.80' average gain in the last column is likewise reported without variance or a significance test. Because the central empirical claim is that DiffoRA 'consistently outperforms' existing methods, the paper should report per-method means and standard deviations over multiple seeds, or justify why the observed differences are meaningful despite the reported spread.
  4. [Section 4.3, Algorithm 1] The weight-sharing strategy is in tension with the theoretical model. In Eq. (5), γr=0 disables the row completely, but after discretization, unselected modules are not frozen; they receive a shared low-rank update ΔW_s (Algorithm 1, lines 11-15). Thus the model actually trained does not match the masked model analyzed in Theorems 1-2. The paper should either analyze the weight-shared objective or present results with truly frozen unselected modules as a separate ablation.
minor comments (6)
  1. [Section 4.1, Eq. (10) and Algorithm 1] The constraint ∑_j ¯γ_{i,j} = 1 appears in the text, but the pseudocode does not describe how this constraint is enforced after gradient updates; please clarify the parameterization (e.g., softmax) and the projection step.
  2. [Section 5.4] There are typos in the paragraph: 'effectivenss' should be 'effectiveness', and 'QQP..' should be 'QQP'.
  3. [Table 2] In the SQuADv2.0 AdaLoRA row, '84.6 /87.5' has a stray space and is inconsistent with the formatting of other entries.
  4. [Section 3.1, Eq. (3)] The statement 'where d, k ≪ r' appears to be a typo; in LoRA the rank r is typically much smaller than d and k. Please correct the inequality.
  5. [Section 4.2, Eq. (12)] The definition of δ_i as the K-th largest entry of ¯γ_i should specify how ties are broken to make the top-K selection deterministic.
  6. [Section 7] The limitations paragraph discusses only the sample-rate selection and does not acknowledge that Theorem 1's key premise I^Γ_w − I_w ⪰ 0 is unverified; the latter should be explicitly listed as a limitation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is an explicit conditional statement whose proof is a valid matrix-inequality consequence of its stated PSD premise, and the empirical comparisons are self-contained against external benchmarks.

full rationale

The central theoretical claim is conditional: Theorem 1 assumes I^Γ_w − I^w ⪰ 0 and then derives λΓ_0 ≥ λ0 via the Schur product theorem and Weyl's inequality (Appendix A.1, Eqs. 17–19). This derivation is mathematically valid; the assumption is not a hidden restatement of the conclusion, although it is unproven and may fail for the binarized DAM actually produced by continuous relaxation and top-K discretization. That is a soundness and verification gap, not a circular reduction: no fitted parameter is renamed as a prediction, no uniqueness claim is imported from the authors' earlier work, and no conclusion is made identical to an input by construction. The empirical claims rest on held-out GLUE and SQuAD comparisons against LoRA, AdaLoRA, full fine-tuning, and several other baselines, so the experimental evaluation is self-contained. The only self-citation, [33] (MECO), appears in a general remark about over-parameterized networks and is not load-bearing for any theorem or result. The acknowledged limitation in Section 7 (sample rate chosen empirically) is an honest hyperparameter caveat rather than circular reasoning. Therefore no circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method depends on per-dataset tuning of the selection ratio and LoRA rank, an unverified PSD assumption in the main theorem, and the implicit transfer of a single-hidden-layer analysis to deep transformers. The DAM is the main invented component, but it is an algorithm-internal object rather than an empirically falsifiable entity.

free parameters (4)
  • Selection ratio ρ (sample rate) = 0.2 to 0.9, dataset-specific (e.g., 0.5 for GLUE, 0.4/0.5/0.7 optimal in Table 5)
    Chosen by validation performance per dataset; acknowledged in Section 7 as empirical.
  • LoRA rank r = 1 to 15 depending on dataset and parameter budget (Tables 6-7)
    Set per dataset to match baseline parameter budgets; not derived from theory.
  • LoRA dropout (LoRA drop) = 0 to 0.35 per dataset (Table 6)
    Hyperparameter tuned per task; not described in the main text.
  • K (number of selected modules per layer) = K = floor(rho * N), e.g., 3 for N=6 at rho=0.5, 2 for STS-B
    Determined by rho, which is tuned per dataset.
assumptions (4)
  • ad hoc to paper I^Γ_w - I_w ⪰ 0 for the learned Γ (the selective matrix makes the expected ReLU activation indicator difference PSD)
    Assumed in Theorem 1 and used in Appendix A.1 Eqs. 17-19 to prove λΓ_0 ≥ λ0; never verified for the Γ produced by the method.
  • domain assumption Single-hidden-layer ReLU NTK analysis (Du et al. 2019) applies to LoRA fine-tuning of multi-layer transformers
    Theorems 1 and 2 are proved for a one-hidden-layer network with ReLU, then used to justify module selection in deep LLMs without a transfer argument.
  • domain assumption The training and validation split used for bi-level optimization provides a reliable signal for module selection
    Stage 1 optimizes Γ on a validation subset; the paper does not analyze how this generalizes across tasks.
  • standard math Weyl inequalities and Courant minimax principle
    Used in the proofs of Theorems 1 and 2, stated as standard results in matrix analysis.
invented entities (1)
  • DAM (Differentiable Adaptation Matrix) Γ
    purpose: Learned binary mask that selects which modules receive LoRA updates
    The central algorithmic construct; its effectiveness is only demonstrated on the benchmarks reported in this paper, with no external falsifiable prediction or independent validation beyond the anonymous code URL.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection." pith.science (2026). https://pith.science/paper/EKIBJMN4

@misc{pith2026250208905,
  author       = {Pith},
  title        = {Pith review of: DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKIBJMN4}},
  note         = {Machine review of arXiv:2502.08905}
}
read the original abstract

The Parameter-Efficient Fine-Tuning (PEFT) methods have been extensively researched for large language models in downstream tasks. Among all the existing approaches, the Low-Rank Adaptation (LoRA) has gained popularity for its streamlined design by incorporating low-rank matrices into existing pre-trained models. Though effective, LoRA, as well as its adaptive optimizations, either allocate the same matrix to all the modules or adjust the interior rank of the components based on importance scoring indicators. In this paper, we argue that not all the modules in LLMs are suitable and necessary to be fine-tuned. Enlightened by this insight, we propose a new PEFT scheme called DiffoRA, which enables adaptive adoption of the low-rank decomposition matrices. At the core of DiffoRA lies a Differential Adaptation Matrix (DAM) to determine which module is the most suitable and essential for fine-tuning. We theoretically explain how the designed matrix impacts the convergence rate and generalization capability of a pre-trained model. We then construct the DAM via continuous relaxation and discretization with weight-sharing optimizations. We fully implement DiffoRA and design comprehensive experiments to evaluate its performance. The experimental results demonstrate that DiffoRA delivers state-of-the-art results across multiple benchmarks.

Figures

Figures reproduced from arXiv: 2502.08905 by the authors.

Figure 1
Figure 1. The loss (left) and final accuracy (right) obtained on the CoLA dataset and DeBERTaV3- [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of DiffoRA contains two stages. In stage one (left part), the initialized [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The module weights of the DeBERTaV3-base model in the CoLA and SST-2 datasets. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 53 canonical work pages

  1. [1]

    Dtllm-vlt: Diverse text generation for visual language tracking based on llm,

    X. Li, X. Feng, S. Hu, M. Wu, D. Zhang, J. Zhang, and K. Huang, “Dtllm-vlt: Diverse text generation for visual language tracking based on llm,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7283–7292

    work page 2024

  2. [2]

    Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation,

    L. Qu, S. Wu, H. Fei, L. Nie, and T.-S. Chua, “Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 643–654

    work page 2023

  3. [3]

    Contrastive prefer- ence optimization: Pushing the boundaries of llm performance in machine translation,

    H. Xu, A. Sharaf, Y . Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y . J. Kim, “Contrastive prefer- ence optimization: Pushing the boundaries of llm performance in machine translation,” in International Conference on Machine Learning. PMLR, 2024, pp. 55 204–55 224

    work page 2024

  4. [4]

    Designing heterogeneous llm agents for financial sentiment analysis,

    F. Xing, “Designing heterogeneous llm agents for financial sentiment analysis,” ACM Transactions on Management Information Systems, 2024

    work page 2024

  5. [5]

    Toolqa: A dataset for llm question answering with external tools,

    Y . Zhuang, Y . Yu, K. Wang, H. Sun, and C. Zhang, “Toolqa: A dataset for llm question answering with external tools,” Advances in Neural Information Processing Systems, vol. 36, pp. 50 117–50 143, 2023

    work page 2023

  6. [6]

    P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks,

    X. Liu, K. Ji, Y . Fu, W. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 61–68

    work page 2022

  7. [7]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 3045–3059

    2021

  8. [8]

    Parameter-efficient transfer learning with diff pruning,

    D. Guo, A. M. Rush, and Y . Kim, “Parameter-efficient transfer learning with diff pruning,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 4884–4896

    work page 2021

Show all 64 references
  1. [9]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022

    2022

  2. [10]

    Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning,

    R. Zhang, R. Qiang, S. A. Somayajula, and P. Xie, “Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

    2024

  3. [11]

    Adaptive budget allocation for parameter-efficient fine-tuning,

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” in The Eleventh International Conference on Learning Representations, 2023

    2023

  4. [12]

    Alora: Allocating low-rank adaptation for fine-tuning large language models,

    Z. Liu, J. Lyn, W. Zhu, and X. Tian, “Alora: Allocating low-rank adaptation for fine-tuning large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long...

    2024

  5. [13]

    Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution,

    Y . Mao, K. Huang, C. Guan, G. Bao, F. Mo, and J. Xu, “Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution,” arXiv preprint arXiv:2405.17357, 2024

    2024 arXiv

  6. [14]

    Lora-drop: Efficient lora parameter pruning based on output evaluation,

    H. Zhou, X. Lu, W. Xu, C. Zhu, T. Zhao, and M. Yang, “Lora-drop: Efficient lora parameter pruning based on output evaluation,” arXiv preprint arXiv:2402.07721, 2024

    2024 arXiv

  7. [15]

    Unveiling lora intrinsic ranks via salience analysis,

    W. Ke, J. Wang, P. Wang, J. Liu, D. Nie, G. Li, and Y . Li, “Unveiling lora intrinsic ranks via salience analysis,” in Advances in Neural Information Processing Systems, 2024

    2024

  8. [16]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in International Conference on Learning Representations, 2019. 10

    2019

  9. [17]

    Efficient parametrization of multi-domain deep neural networks,

    S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Efficient parametrization of multi-domain deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8119–8127

    2018

  10. [18]

    Velora: Memory efficient training using rank-1 sub-token projections,

    R. Miles, P. Reddy, I. Elezi, and J. Deng, “Velora: Memory efficient training using rank-1 sub-token projections,” in Advances in Neural Information Processing Systems. Curran Associates, Inc., 2024

    2024

  11. [19]

    Hifi: High-information attention heads hold for parameter-efficient model adaptation,

    A. Gui and H. Xiao, “Hifi: High-information attention heads hold for parameter-efficient model adaptation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 8521–8537

    2023

  12. [20]

    Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked language-models,

    E. B. Zaken, Y . Goldberg, and S. Ravfogel, “Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked language-models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 1–9

    2022

  13. [21]

    What would elsa do? freezing layers during transformer fine-tuning,

    J. Lee, R. Tang, and J. Lin, “What would elsa do? freezing layers during transformer fine-tuning,”arXiv preprint arXiv:1911.03090, 2019

    1911 arXiv

  14. [22]

    VeRA: Vector-based random matrix adaptation,

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano, “VeRA: Vector-based random matrix adaptation,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    LoRA+: Efficient low rank adaptation of large models,

    S. Hayou, N. Ghosh, and B. Yu, “LoRA+: Efficient low rank adaptation of large models,” inForty-first International Conference on Machine Learning, 2024

    2024

  16. [24]

    DoRA: Weight-decomposed low-rank adaptation,

    S. yang Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “DoRA: Weight-decomposed low-rank adaptation,” inForty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  18. [26]

    Tied-lora: Enhancing parameter efficiency of lora with weight tying,

    A. Renduchintala, T. Konuk, and O. Kuchaiev, “Tied-lora: Enhancing parameter efficiency of lora with weight tying,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long...

    2024

  19. [27]

    Lora+: Efficient low rank adaptation of large models,

    S. Hayou, N. Ghosh, and B. Yu, “Lora+: Efficient low rank adaptation of large models,” inInternational Conference on Machine Learning. PMLR, 2024, pp. 17 783–17 806

    2024

  20. [28]

    LoRA-pro: Are low-rank adapters properly optimized?

    Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan, “LoRA-pro: Are low-rank adapters properly optimized?” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  21. [29]

    Hydralora: An asymmetric lora architecture for efficient fine-tuning,

    C. Tian, Z. Shi, Z. Guo, L. Li, and C.-Z. Xu, “Hydralora: An asymmetric lora architecture for efficient fine-tuning,” Advances in Neural Information Processing Systems, vol. 37, pp. 9565–9584, 2024

    2024

  22. [30]

    Sparse low-rank adaptation of pre-trained language models,

    N. Ding, X. Lv, Q. Wang, Y . Chen, B. Zhou, Z. Liu, and M. Sun, “Sparse low-rank adaptation of pre-trained language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 4133–4145

    2023

  23. [31]

    Gradient descent provably optimizes over-parameterized neural networks,

    S. S. Du, X. Zhai, B. Poczos, and A. Singh, “Gradient descent provably optimizes over-parameterized neural networks,” in International Conference on Learning Representations, 2019

    2019

  24. [32]

    Visualizing the loss landscape of neural nets,

    H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” Advances in neural information processing systems, vol. 31, 2018

    2018

  25. [33]

    Meco: zero-shot nas with one data and single forward pass via minimum eigenvalue of correlation,

    T. Jiang, H. Wang, and R. Bie, “Meco: zero-shot nas with one data and single forward pass via minimum eigenvalue of correlation,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  26. [34]

    R. A. Horn and C. R. Johnson, Matrix analysis. Cambridge university press, 2012

    2012

  27. [35]

    Generalization bounds of stochastic gradient descent for wide and deep neural networks,

    Y . Cao and Q. Gu, “Generalization bounds of stochastic gradient descent for wide and deep neural networks,” Advances in neural information processing systems, vol. 32, 2019

    2019

  28. [36]

    Generalization properties of nas under activation and skip connection search,

    Z. Zhu, F. Liu, G. Chrysos, and V . Cevher, “Generalization properties of nas under activation and skip connection search,” Advances in Neural Information Processing Systems, vol. 35, pp. 23 551–23 565, 2022

    2022

  29. [37]

    DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,

    P. He, J. Gao, and W. Chen, “DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,” in The Eleventh International Conference on Learning Representations, 2023. 11

    2023

  30. [38]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” inProceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, T. Linzen, G. Ch...

    2018

  31. [39]

    SQuAD: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds., Nov. 2016

    2016

  32. [40]

    Know what you don’t know: Unanswerable questions for SQuAD,

    P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), I. Gurevych and Y . Miyao, Eds. Melbourne, Australia: Associat...

    2018

  33. [41]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  34. [42]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  35. [43]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, “Roberta: A robustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, vol. 364, 2019

    1907 arXiv

  36. [44]

    Magnus and H

    J. Magnus and H. Neudecker,Matrix Differential Calculus with Applications in Statistics and Econometrics (Revised Edition). John Wiley & Sons Ltd, 1999

    1999

  37. [45]

    G. H. Golub and C. F. Van Loan, Matrix computations. JHU press, 2013

    2013

  38. [46]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational conference on machine learning. PMLR, 2019, pp. 2790–2799

    2019

  39. [47]

    Adapterfusion: Non-destructive task composition for transfer learning,

    J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych, “Adapterfusion: Non-destructive task composition for transfer learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 487–503. ...

    2021

  40. [48]

    (vii) LoRA+ [27] employs different learning rates to update the low-rank matrices

    is a sparse-fine-tuning method for pre-trained models that updates only a small subset of the bias terms; (iii) Houlsby adapter [46] adds a few trainable modules inserted between layers of a pre-trained model, allowing for task-specific tuning without altering the entire model...

  41. [49]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Our contributions and scope can be accurately reflected in the abstract and introduction. Guidelines: • The answer NA ...

  42. [50]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations can be found in Section 7. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the pape...

  43. [51]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 18 Justification: We provide the full set of assumptions and a complete proof in the Appendix A. Guidelines...

  44. [52]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  45. [53]

    All the datasets we used are public

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 19 Answer: [Yes] Justification: We provided an anonymous...

  46. [54]

    For additional experimental settings on various benchmarks, we provided the details in Appendix D

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We have described our experim...

  47. [55]

    We have run our experiments multiple times with different seeds and provided the mean and standard deviation values in Section 5

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We have clearly stated the factors of variability ...

  48. [56]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We have provided the ha...

  49. [57]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We confirm that we preserve the anonymity and conform to the NeurIPS Codes of E...

  50. [58]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: This work is fundamental research in computer science and has no societal impact. Guidelines: • The answer...

  51. [59]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper pose...

  52. [60]

    We have properly cited these resources in our paper

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All the codes an...

  53. [61]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have provided an anonymous URL link for our codes. Guidelines: • The answer NA means that the paper does not rele...

  54. [62]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  55. [63]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  56. [64]

    Answer: [NA] Justification: We utilize LLM only for grammar and formatting purposes

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.