Pith. sign in

REVIEW 4 major objections 4 minor 59 references

Improving LoRA with Variational Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing AdamW with the IVON variational optimizer turns LoRA finetuning into Bayesian learning at nearly the same cost, and a 10% uncertainty-guided prune improves both accuracy and calibration on billion-scale LLMs.

desk verdict A plausible and cheap empirical win for LoRA finetuning, but the claim that uncertainty-guided pruning is the reason is not yet backed by the right controls. read the letter →

arxiv 2506.14280 v1 pith:LA4DEKKI submitted 2025-06-17 cs.LG cs.AIcs.CLstat.ML

Bai Cong , Nico Daheim , Yuesong Shen , Rio Yokota , Mohammad Emtiyaz Khan , Thomas Möllenhoff This is my paper
classification cs.LGcs.AIcs.CLstat.ML
keywords LoRAvariationalinferenceIVONparameter-efficientfinetuningBayesiandeeplearninguncertainty-guidedpruningcalibrationtest-timecomputescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LoRA finetuning is cheap but tends to give overconfident, slightly less accurate models. This paper tries to fix both problems by swapping the standard AdamW optimizer for IVON, a variational inference algorithm that is nearly identical in implementation and speed, and that returns a diagonal Gaussian posterior over the LoRA parameters as a byproduct. On top of that posterior, the paper adds a post-training step that prunes the 10% of parameters with the highest posterior variance in each LoRA matrix. The claim is that this combination, IVON-LoRA, improves average accuracy by 1.3% and reduces expected calibration error by 5.4% over AdamW on Llama-3.2-3B commonsense reasoning, matches or beats established Bayesian LoRA baselines, and enables posterior sampling for test-time compute scaling. The intended takeaway is that Bayesian LoRA need not cost extra data passes, Hessian post-processing, or accuracy.

What carries the argument

The engine is IVON (Improved Variational Online Newton), a natural-gradient variational optimizer that works as a drop-in replacement for AdamW. It maintains a diagonal Gaussian posterior $q(\theta) = \mathcal{N}(m, \mathrm{diag}(v))$ over the LoRA parameters and obtains the variances essentially for free as $v = 1/(\lambda(h+\delta))$, where $h$ is an online estimate of the diagonal Hessian, $\delta$ the weight decay, and $\lambda$ an effective sample size that can be adjusted at test time to act like an inverse temperature. Two procedures hang off this posterior: uncertainty-guided pruning, which zeros the 10% of parameters with the largest variance in each LoRA matrix, and posterior sampling, which feeds multiple sampled models into minimum-Bayes-risk decoding for sequence generation. The Hessian-based variance is the load-bearing quantity: it is what identifies prunable parameters and what makes the posterior samples nontrivial.

What would settle it

In any of the six commonsense reasoning setups, replace the 10% highest-variance masks with random masks at equal pruning strength; if random pruning matches or beats variance-guided pruning in accuracy and expected calibration error on the larger datasets, the claim that IVON's posterior variance identifies prunable parameters is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that variational learning with IVON is a better way to train LoRA adapters than point-estimate optimization, and that the posterior it yields is useful beyond calibration. IVON minimizes the expected loss under a diagonal Gaussian plus a KL term, producing a mean and a variance for every LoRA entry at nearly AdamW's cost, with the variance obtained from an online diagonal Hessian estimate. The paper then introduces uncertainty-guided pruning (UGP): after training, it sets to zero the 10% of entries with the largest posterior variance in each matrix, arguing that high-uncertainty parameters are the ones the model can do without. In the main results, IVON-LoRA improves accuracy over AdamW by 1.3% on six commonsense reasoning tasks while cutting expected calibration error by 5.4%; unlike BLoB or Laplace-LoRA, posterior sampling does not degrade accuracy, and at 512 test-time samples on GSM8k with Qwen-2.5-3B it gains 3.7% accuracy over AdamW.

Load-bearing premise

The load-bearing bet is that the diagonal posterior variance learned by IVON over the raw LoRA entries reliably marks parameters that are safe to zero out, so that pruning the 10% highest-variance entries in each matrix helps rather than hurts, even though the induced posterior over the product $BA$ is non-Gaussian.

Editorial extensions

If this is right

  • LoRA finetuning can be made Bayesian with almost no extra compute: no additional data pass, no Kronecker-factored Hessian, and no model linearization, just an optimizer swap.
  • A fixed 10% variance-based prune after training improves calibration and, in several settings, accuracy, while random pruning at the same rate does not reproduce the gains.
  • Posterior sampling with minimum-Bayes-risk decoding turns extra compute at test time into accuracy, and the gap over AdamW widens as the n-best list grows to 512 samples.
  • Scaling the effective sample size $\lambda$ at test time acts like an inverse temperature and improves out-of-distribution accuracy by about 1% while leaving calibration roughly unchanged.
  • The gains appear across model families and scales: Llama-3.2-3B on commonsense reasoning, Qwen-2.5-3B on GSM8k and Conala, and DeBERTa-v3-base on GLUE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because $v$ comes from an online diagonal Hessian estimate, UGP is effectively a cheap second-order pruning rule, though the paper does not frame it that way; this suggests it could be linked to classic Hessian-based pruning work.
  • The paper's own limitation section notes that the posterior over the product $BA$ is non-Gaussian, so pruning the raw entries rather than the induced product posterior is a heuristic; a natural extension would be to prune on the induced variance of $BA$.
  • The result that posterior samples help most at large $n$ hints that IVON-LoRA could serve as an inference-time compute-scaling axis, an application the paper only gestures at.
  • A testable extension is to choose the pruning fraction per layer from the variance spectrum rather than fixing 10% everywhere, since the paper's own Figure 3 shows the optimum varies by dataset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes IVON-LoRA, a method that replaces the AdamW optimizer with the IVON variational optimizer during LoRA finetuning, yielding a diagonal Gaussian posterior over LoRA parameters. It further proposes Uncertainty-Guided Pruning (UGP), which after training removes 10% of the highest-variance entries in each LoRA matrix. The paper reports improved accuracy and calibration over AdamW and over the Bayesian baselines Laplace-LoRA and BLoB on six commonsense reasoning datasets with Llama-3.2-3B, improved GSM8k and Conala performance with Qwen-2.5-3B, improved GLUE scores with DeBERTa-v3-base, and test-time compute scaling via posterior sampling with MBR decoding. The central claims are that IVON is a drop-in replacement with negligible overhead and that UGP is essential to the gains.

Significance. If the results hold, IVON-LoRA would be a practically valuable, nearly free upgrade to LoRA: it provides a posterior, better calibration, and some accuracy gains without the post-hoc passes required by Laplace-LoRA. The paper's strengths include multiple model families and tasks, standard errors for the main Table 1 and Table 2 comparisons, a head-to-head with strong recent Bayesian baselines, and a clear profiling of training overhead. However, the load-bearing UGP mechanism is not adequately isolated, and some headline numbers are reported without uncertainty quantification; the significance of the paper depends on closing those gaps.

major comments (4)
  1. [Sec. 4 and Fig. 3] The claim that IVON's posterior variance is a good indicator of parameter importance is supported only by a comparison against random pruning. Because the default 10% UGP step is used in every main table, the operative signal is not identified: a control that prunes, say, the 10% smallest |m_i|, the 10% smallest SNR |m_i|/sqrt(v_i), or the 10% lowest second-moment entries from an AdamW run would be needed to show that variance, rather than structured sparsity in any form, drives the gains. Without such controls, the headline improvements cannot be attributed to the variational posterior.
  2. [Table 3 and Fig. 2] The GSM8k/Conala results and the test-time scaling curves are reported without standard errors or confidence bands, even though the main tables elsewhere report standard errors over five runs. The reported gains (for example, 68.69 vs. 66.87 on GSM8k at 32 outputs, and the 3.7% improvement at n=512) may be within run-to-run variability; the paper should either run multiple seeds or provide a variance estimate. Similarly, Figure 2b's claim that pruning is essential rests on single-run curves.
  3. [Secs. 5.2, 5.3 and App. A.2] The pruning ratio (10%) and the test-time temperature multiplier tau are selected after inspecting the validation curves in Fig. 3 and Table 6, while the AdamW baseline's learning rate is grid-searched and IVON's own hyperparameters are fixed without an equivalent search. This asymmetry makes it difficult to interpret the reported gains as method-level improvements rather than selection artifacts. A held-out selection procedure or sensitivity analysis over lambda, tau, and the pruning ratio is needed before the headline numbers can be trusted.
  4. [Sec. 6 (Limitations)] The paper acknowledges that the posterior over the product BA is non-Gaussian, but UGP ranks the diagonal variances of the raw entries of A and B as if these variances measured output-relevant importance. Since (B, cA) and (B/c, A) define the same function for any nonzero c, this ranking is not invariant to a reparameterization of LoRA; the paper should either prove invariance under the training dynamics or introduce a canonicalization, otherwise the UGP importance measure is not well-defined even if the empirical curves are favorable.
minor comments (4)
  1. [Eq. (1)] The notation min_q(theta) is slightly abusive because the optimization variable is the parameterization of q; consider writing min_{m,v} for clarity.
  2. [Abstract and Table 1] The phrase 'reduce ECE by 5.4%' is ambiguous; the table shows a change from 18.7 to 13.3, so it should read 'by 5.4 percentage points' or similar.
  3. [Table 3] The header 'CodeBLEU↑Syntax↑Data Flow↑Code-BertScore↑' and the corresponding numbers are visually merged; please format them as separate columns.
  4. [Fig. 1 caption] The caption ends with a stray '2.9' that appears to be a leftover artifact.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reported accuracy/ECE gains are external-benchmark measurements; the IVON self-citation is present but re-validated on new models in this paper, and the UGP pruning choice is validation-set tuning, not a fitted parameter renamed as a prediction.

full rationale

The paper's central claims -- a 1.3% accuracy gain and 5.4% ECE reduction over AdamW on Llama-3.2-3B commonsense tasks, the GSM8k test-time compute-scaling gains, and the UGP-vs-random-pruning comparison of Fig. 3 -- are empirical measurements on external benchmarks (accuracy, ECE, NLL, CodeBLEU), not quantities derived from the variational objective. No equation reduces an output to an input by construction: Eq. (1) is IVON's standard variational objective as defined in prior work, and the variance estimator v = 1/(lambda(h+delta)) is also taken from Shen et al. [41]; the headline numbers come from forward passes on validation splits. The 10% pruning ratio ('we set the pruning ratio to 10% and apply it to the LoRA adapters after training by default') and the lambda values ('we empirically set it to 5x10^5') are validation-set hyperparameter choices, which is ordinary tuning rather than a fitted parameter renamed as a prediction. Self-citation of IVON is substantial -- the entire method is inherited from a prior paper with seven overlapping authors -- but it is not load-bearing in a circular sense, because the paper re-establishes the method's utility on new model families (Llama-3.2-3B, Qwen-2.5-3B, DeBERTa-v3-base) with standard errors over 5 runs. The Limitations section (Sec. 6) admits that the posterior over the product BA is non-Gaussian while UGP uses diagonal variances of the raw A and B entries; this weakens the justification of the importance criterion but is a correctness/robustness concern, not circularity, since the pruning benefit is measured against random pruning on held-out data. The absence of controls such as magnitude-based pruning is a coverage gap for the mechanism claim, not a circular reduction. Verdict: no significant circularity; score reflects minor method-level self-citation only.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The empirical gains of IVON-LoRA rest on several hand-set hyperparameters, notably lambda and the 10% UGP ratio, as well as on inherited IVON approximations. No new entities are introduced; the method is a combination of existing components (LoRA, IVON, Hessian-style pruning, MBR decoding).

free parameters (4)
  • lambda (effective dataset size) = 5e5 (WG-S, ARC-E, ARC-C), 1e6 (WG-M, OBQA, BoolQ), 1e7 (GSM8k), 5e6 (GLUE)
    Chosen per dataset in Secs. A.2.1, A.3.3, A.4.1. It scales the posterior variance, which drives UGP and posterior sampling, so headline gains depend on a favorable hand-picked value.
  • UGP pruning ratio = 10% of each LoRA matrix
    Set as default in Sec. 4 after the ablation in Fig. 3; the optimal level varies by dataset and pruning strength is a free knob, not a predicted quantity.
  • IVON hyperparameters (lr, beta2, gradient clip, h init) = lr 5e-4/3e-2/1e-2; beta2 0.9998/0.99999/0.9998; clip 0.1/0.01/0.02; h 5e-3
    Hand-set per task in Secs. A.2.1, A.3.3, A.4.1; these shape the learned mean and variance and are not derived from first principles.
  • test-time temperature scale tau = 2, 5, 10
    Table 6 shows OOD accuracy improves at tau=5/10; this scaling is selected post hoc on the same evaluation sets, so the reported gain includes selection.
assumptions (4)
  • domain assumption The variational objective in Eq. (1), with a diagonal Gaussian posterior q and zero-mean isotropic prior, is an adequate approximation for LoRA finetuning.
    Sec. 3 inherits IVON's approximations, including v=1/(lambda(h+delta)) and one-sample Monte Carlo; no error analysis is provided for LoRA posteriors.
  • domain assumption Largest posterior variance identifies the least important LoRA parameters.
    Sec. 4 states this as the basis for UGP; Fig. 3 gives empirical support against random pruning, but the principle is assumed and not proven.
  • domain assumption LoRA's low-rank update W=W0+BA with rank 8 is expressive enough for the finetuning tasks considered.
    Standard LoRA assumption from Hu et al. [20], used throughout Sec. 2; the paper does not test rank sensitivity.
  • domain assumption Int8 base-model quantization in commonsense experiments does not change relative method rankings.
    Sec. A.2 applies int8 quantization to Llama-3.2-3B for all methods; if quantization interacts differently with IVON and AdamW, comparisons could shift.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LoRA with Variational Learning." pith.science (2026). https://pith.science/paper/LA4DEKKI

@misc{pith2026250614280,
  author       = {Pith},
  title        = {Pith review of: Improving LoRA with Variational Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LA4DEKKI}},
  note         = {Machine review of arXiv:2506.14280}
}
read the original abstract

Bayesian methods have recently been used to improve LoRA finetuning and, although they improve calibration, their effect on other metrics (such as accuracy) is marginal and can sometimes even be detrimental. Moreover, Bayesian methods also increase computational overheads and require additional tricks for them to work well. Here, we fix these issues by using a recently proposed variational algorithm called IVON. We show that IVON is easy to implement and has similar costs to AdamW, and yet it can also drastically improve many metrics by using a simple posterior pruning technique. We present extensive results on billion-scale LLMs (Llama and Qwen series) going way beyond the scale of existing applications of IVON. For example, we finetune a Llama-3.2-3B model on a set of commonsense reasoning tasks and improve accuracy over AdamW by 1.3% and reduce ECE by 5.4%, outperforming AdamW and other recent Bayesian methods like Laplace-LoRA and BLoB. Overall, our results show that variational learning with IVON can effectively improve LoRA finetuning.

Figures

Figures reproduced from arXiv: 2506.14280 by the authors.

Figure 1
Figure 1. Overview and benefits of IVON-LoRA. (a) IVON-LoRA estimates a diagonal Gaussian pos [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Improvements obtained with IVON-LoRA on GSM8k increase with n-best-size. For smaller [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Uncertainty-Guided Pruning is essential for improving the performance of IVON-LoRA. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The training speeds of IVON and AdamW are similar. We plot validation accuracies [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 46 canonical work pages

  1. [1]

    Stop measuring calibration when humans disagree

    Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernández. Stop measuring calibration when humans disagree. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    work page 2022

  2. [2]

    BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. InAssociation for Computational Linguistics (ACL), 2022

    work page 2022

  3. [3]

    LoRA learns less and forgets less.arXiv:2405.09673, 2024

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less.arXiv:2405.09673, 2024

    arXiv 2024

  4. [4]

    Weight uncertainty in neural network

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. InInternational Conference on Machine Learning (ICML), 2015

    work page 2015

  5. [5]

    A Bayesian Interpretation of Adaptive Low-Rank Adaptation

    Haolin Chen and Philip N Garner. A Bayesian interpretation of adaptive low-rank adaptation. arXiv:2409.10673, 2024

    work page Pith review arXiv 2024

  6. [6]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InAnnual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019

    work page 2019

  7. [7]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv:1803.05457, 2018

    arXiv 2018

  8. [8]

    Training verifiers to solve math word problems.arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv:2110.14168, 2021

    arXiv

Show all 59 references
  1. [9]

    Uncertainty-aware decoding with minimum Bayes risk

    Nico Daheim, Clara Meister, Thomas Möllenhoff, and Iryna Gurevych. Uncertainty-aware decoding with minimum Bayes risk. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  2. [10]

    QLoRA: Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  3. [11]

    Shaving weights with Occam’s razor: Bayesian sparsification for neural networks using the marginal likelihood

    Rayen Dhahri, Alexander Immer, Bertrand Charpentier, Stephan Günnemann, and Vincent Fortuin. Shaving weights with Occam’s razor: Bayesian sparsification for neural networks using the marginal likelihood. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  4. [12]

    Sparse low-rank adaptation of pre-trained language models

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  5. [13]

    The Llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Practical variational inference for neural networks

    Alex Graves. Practical variational inference for neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), 2011

    2011

  7. [15]

    The safe Bayesian - learning the learning rate via the mixability gap

    Peter Grünwald. The safe Bayesian - learning the learning rate via the mixability gap. InAlgorithmic Learning Theory (ALT), 2012

    2012

  8. [16]

    DeBERTa: Decoding-enhanced BERT with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. InInternational Conference on Learning Representations (ICLR), 2021. 10

    2021

  9. [17]

    SparseAdapter: An easy approach for improving the parameter-efficiency of adapters

    Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. SparseAdapter: An easy approach for improving the parameter-efficiency of adapters. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    2022

  10. [18]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  11. [19]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019

    2019

  12. [20]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations (ICLR), 2022

    2022

  13. [21]

    Improving predictions of Bayesian neural nets via local linearization

    Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local linearization. InInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2021

    2021

  14. [22]

    VeRA: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  15. [23]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. InAdvances in Neural Information Processing Systems (NeurIPS), 1989

    1989

  16. [24]

    Flat-LoRA: Low-rank adaption over a flat loss landscape.arXiv:2409.14396, 2024

    Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang. Flat-LoRA: Low-rank adaption over a flat loss landscape.arXiv:2409.14396, 2024

    2024 arXiv

  17. [25]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pap...

    2021

  18. [26]

    LoftQ: LoRA-fine-tuning-aware quantization for large language models

    Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoftQ: LoRA-fine-tuning-aware quantization for large language models. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  19. [27]

    ReLoRA: High-rank training through low-rank updates

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReLoRA: High-rank training through low-rank updates. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  20. [28]

    PAC-tuning: Fine- tuning pre-trained language models with PAC-driven perturbed gradient descent

    Guangliang Liu, Zhiyu Xue, Xitong Zhang, Kristen Johnson, and Rongrong Wang. PAC-tuning: Fine- tuning pre-trained language models with PAC-driven perturbed gradient descent. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  21. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  22. [30]

    A practical Bayesian framework for backpropagation networks.Neural Computation, 4(3):448–472, 1992

    David JC MacKay. A practical Bayesian framework for backpropagation networks.Neural Computation, 4(3):448–472, 1992

    1992

  23. [31]

    PEFT: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/ peft, 2022

    2022

  24. [32]

    Optimizing neural networks with Kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. InInternational Conference on Machine Learning (ICML), 2015

    2015

  25. [33]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2018

    2018

  26. [34]

    Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models

    Emre Onal, Klemens Flöge, Emma Caldwell, Arsen Sheverdin, and Vincent Fortuin. Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models. InSymposium on Advances in Approximate Bayesian Inference (AABI), 2024

    2024

  27. [35]

    MAD-X: An adapter-based framework for multi-task cross-lingual transfer

    Jonas Pfeiffer, Ivan Vuli´c, Iryna Gurevych, and Sebastian Ruder. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2020. 11

    2020

  28. [36]

    Adapters: A unified library for parameter- efficient and modular transfer learning

    Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vuli´c, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: A unified library for parameter- efficient and modular transfer learning. InConference on Empirical Methods...

    2023

  29. [37]

    CodeBLEU: a method for automatic evaluation of code synthesis

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. CodeBLEU: a method for automatic evaluation of code synthesis. arXiv:2009.10297, 2020

    2009 arXiv

  30. [38]

    A scalable Laplace approximation for neural networks

    Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. InInternational Conference on Learning Representations (ICLR), 2018

    2018

  31. [39]

    AdapterDrop: On the efficiency of adapters in transformers

    Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2021

    2021

  32. [40]

    WinoGrande: An adversarial Winograd schema challenge at scale.Communications of the ACM, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale.Communications of the ACM, 2021

    2021

  33. [41]

    Variational learning is effective for large deep networks

    Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational learning is effective for large deep networks. InInternational Conference on Machine Learn...

    2024

  34. [42]

    Qwen2.5 technical report.arXiv:2412.15115, 2024

    Qwen Team. Qwen2.5 technical report.arXiv:2412.15115, 2024

    2024 arXiv

  35. [43]

    DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Lingu...

    2023

  36. [44]

    Low-rank variational Bayes correction to the Laplace method.J

    Janet van Niekerk and Haavard Rue. Low-rank variational Bayes correction to the Laplace method.J. Mach. Learn. Res. (JMLR), 2024

    2024

  37. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  38. [46]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  39. [47]

    LoRA ensembles for large language model fine-tuning, 2023

    Xi Wang, Laurence Aitchison, and Maja Rudolph. LoRA ensembles for large language model fine-tuning, 2023

    2023

  40. [48]

    BLoB: Bayesian low-rank adaptation by backpropagation for large language models

    Yibin Wang, Haizhou Shi, Ligong Han, Dimitris N Metaxas, and Hao Wang. BLoB: Bayesian low-rank adaptation by backpropagation for large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  41. [49]

    Flipout: Efficient pseudo-independent weight perturbations on mini-batches

    Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo-independent weight perturbations on mini-batches. InInternational Conference on Learning Representations (ICLR), 2018

    2018

  42. [50]

    QA-LoRA: Quantization-aware low-rank adaptation of large language models

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, XIAOPENG ZHANG, and Qi Tian. QA-LoRA: Quantization-aware low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2024

    2024

  43. [51]

    Bayesian low-rank adaptation for large language models

    Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  44. [52]

    Learning to mine aligned code and natural language pairs from Stack Overflow

    Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from Stack Overflow. In2018 IEEE/ACM 15th international conference on mining software repositories (MSR), 2018

    2018

  45. [53]

    Optimal information processing and Bayes’s theorem.The American Statistician, 42(4): 278–280, 1988

    Arnold Zellner. Optimal information processing and Bayes’s theorem.The American Statistician, 42(4): 278–280, 1988

    1988

  46. [54]

    LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning.arXiv:2308.03303, 2023

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning.arXiv:2308.03303, 2023. 12

    2023 arXiv

  47. [55]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  48. [56]

    From ε-entropy to KL-entropy: Analysis of minimum information complexity density estimation.The Annals of Statistics, 2006

    Tong Zhang. From ε-entropy to KL-entropy: Analysis of minimum information complexity density estimation.The Annals of Statistics, 2006

    2006

  49. [57]

    GIFT-SW: Gaussian noise injected fine-tuning of salient weights for LLMs

    Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov, Mariya Krylova, Aleksandr Zuev, and Evgeny Burnaev. GIFT-SW: Gaussian noise injected fine-tuning of salient weights for LLMs. arXiv:2408.15300, 2024

    2024 arXiv

  50. [58]

    True"/"False

    Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. CodeBERTScore: Evaluating code generation with pretrained models of code. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2023. 13 A Details on Experimental Setup A.1 General Setup We utilize th...

    2023

  51. [59]

    For full-parameter and LoRA finetuning, we pick1×10 −4 and5×10 −4, respectively

    We conduct a grid search for the learning rate for both full-parameter and LoRA finetuning with AdamW. For full-parameter and LoRA finetuning, we pick1×10 −4 and5×10 −4, respectively. IVON-LoRAFor IVON-LoRA, we set the learning rate to 1×10 −2, β1 to 0.9, β2 to 0.9998, and imp...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.