Pith. sign in

REVIEW 3 major objections 7 minor 55 references

The paper claims that all gradient-based LoRA initialization methods are special cases of a single two-parameter family — a spectral whitening exponent and an Adam-like diagonal exponent — and that the best operating point is usually neithe

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:22 UTC pith:4JLQ4HYS

load-bearing objection A genuinely useful reframing of LoRA initialization as a tunable continuum, honestly reported, but the headline gains are oracle-bound and the interior-optimum story has an unresolved noise confound. the 3 major comments →

arxiv 2607.26247 v1 pith:4JLQ4HYS submitted 2026-07-28 cs.LG

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Dianze Liu , Farshid Ghezelbash This is my paper
classification cs.LG
keywords LoRA initializationK-FAC preconditioningnatural gradientparameter-efficient fine-tuningcurvature whiteningAdam scalinglow-rank adaptationspectral statistics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that existing ways of initializing LoRA adapters from the loss gradient are not separate techniques but special cases of one two-parameter family, ULoRA, governed by a spectral whitening exponent α and an Adam-like diagonal exponent β. The claim is supported by a full learning-rate sweep over the family on RoBERTa, T5, and LLaMA-2-7B: the best operating point is task- and model-dependent, frequently lies strictly inside the family away from the published endpoints (raw gradient vs. full inverse-square-root whitening), and a tuned ULoRA configuration matches or exceeds full fine-tuning on all five RoBERTa GLUE tasks. The paper also introduces ULoRA-Auto, a search-free rule that picks per-layer exponents from spectral statistics measured during the initialization pass, recovering most of the grid-search upper bound at no extra cost. A sympathetic reader would care because it turns a fixed design decision into a tunable dimension, and suggests that intermediate preconditioning — not the extremes — is often the right choice.

Core claim

The central claim is that a single two-parameter family, P_{α,β} = T^{-α/2} [d_T^{-β/2} ⊙ ∇W L ⊙ d_S^{-β/2}] S^{-α/2}, contains all existing gradient-based LoRA initializations as special cases: (0,0) is LoRA-GA/LoRA-One, (1,0) is CG-LoRA, (0.5,0) matches Shampoo, (2,0) is K-FAC natural gradient. Sweeping the family under a full learning-rate search, the paper finds the best operating point is task-dependent and frequently interior (e.g., (1.5,0.5) on RoBERTa QNLI and SST-2), and that a tuned point matches or beats full fine-tuning on all five RoBERTa GLUE tasks. It also claims the right exponents can be read off spectral statistics: ULoRA-Auto selects per-layer α and β from effective rank,

What carries the argument

The carrying object is Eq. (1), the ULoRA preconditioned gradient, built from K-FAC Kronecker factors S and T (input and output-gradient covariances over one initialization batch) and their diagonals d_S, d_T. The spectral whitening exponent α rescales K-FAC eigenvalues; the diagonal exponent β applies Adam-like per-feature normalization. To avoid forming the full matrix, the gradient is projected into the top-s Rayleigh–Ritz eigenspace of S and T, where the exponents are applied, and the rank-r SVD is back-projected and balanced into A0 and B0 via a QR plus tiny-SVD realization. ULoRA-Auto sets exponents from measurable spectral statistics, making the continuum navigable at zero search cost

Load-bearing premise

The load-bearing premise is that K-FAC curvature factors computed from one small initialization batch are reliable enough that the whitening exponent α and diagonal exponent β capture meaningful geometry; the paper concedes it offers no theory of the optimal exponent under curvature-estimation noise, and that ULoRA-Auto's diagnostics measure estimation quality rather than task-level utility. If that link is weak, the zero-cost selection is unjustified and the continuum is an

What would settle it

A concrete falsifier: pick a task where ULoRA-Auto picks moderate whitening (e.g., QNLI with RoBERTa), then measure test accuracy while systematically increasing the noise in the initialization batch (e.g., smaller batches or noisier labels). If Auto's α does not decline as estimation noise rises, or if grid-searched optimal α stays constant, the spectral-statistics-to-optimal-exponent link is broken. More directly: if there exists a task where the grid-searched optimum consistently lies outside [0,2]×[0,1] at (0,0) or (2,1) with a large margin, the claim that the optimum is 'frequently interi

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the continuum view is right, comparisons among LoRA-GA, LoRA-One, and CG-LoRA are not comparisons of distinct mechanisms but of different points on the same curve; re-tuning the exponent may obviate the need to switch methods.
  • The empirical finding that the optimum is often interior implies that published endpoint methods leave accuracy on the table; deployable systems should treat α and β as tunable hyperparameters rather than fixed architectural choices.
  • ULoRA-Auto's near-parity with the oracle grid search suggests that spectral statistics computed in one initialization pass carry enough signal to set the preconditioning strength, which would remove the cost of grid search.
  • The task dependence of the optimum — raw gradient for GSM8K, strong whitening for HumanEval — means a single global recommendation for preconditioning strength is unlikely to hold across tasks; per-task or per-layer selection is needed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is whether the same two-parameter continuum generalizes beyond LoRA initialization to the ongoing optimizer state during fine-tuning, effectively unifying Adam, Shampoo, and K-FAC steps into one schedule; the paper does not pursue this, but Eq. (1) is written as a preconditioner for arbitrary gradient steps.
  • If ULoRA-Auto's heuristic is reliable, it implies that the condition number and effective rank of the K-FAC factors are cheap proxies for curvature-estimation noise; one could probe this by corrupting the initialization batch (e.g., adding label noise) and checking whether Auto's α decreases as predicted.
  • The paper's claim that the optimum tracks curvature-estimate reliability suggests a direct comparison: on tasks where the same model is trained with tiny versus large initialization batches, the optimal α should shift monotonically; this is not tested but follows from the paper's own explanation.
  • The negative-α failure region found on MRPC hints that amplifying high-curvature directions is actively harmful; an adversarial check would be to verify whether any real task ever benefits from α<0, or whether the published negative region is a generic pathology.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, with a spectral whitening exponent α and an Adam-like diagonal exponent β. It shows that existing methods such as LoRA-GA/LoRA-One (raw gradient), CG-LoRA (full inverse-square-root whitening), Shampoo-style quarter-power preconditioning, and K-FAC natural gradient are special cases of a single formula (Eq. 1). The initialization pipeline uses Rayleigh–Ritz projected K-FAC curvature factors S and T, computes a projected preconditioned gradient, and extracts a balanced low-rank adapter via QR/SVD (Eqs. 4–10). Empirically, the authors perform a grid sweep over α∈{0,0.5,1,1.5,2} and β∈{0,0.5,1} on RoBERTa-base, T5-base, and LLaMA-2-7B, reporting that the best grid point (an oracle upper bound selected per task on test metrics) matches or exceeds full fine-tuning on five GLUE tasks and is competitive on GSM8K, HumanEval, and MMLU. They also introduce ULoRA-Auto, a heuristic that sets per-layer exponents from spectral statistics (effective rank, condition number, off-diagonal energy) at zero search cost, and report that it ranks at or near the top among deployable methods.

Significance. If the empirical claims held, the paper would make a useful conceptual contribution: it provides a clean unification of gradient-based LoRA initializations, a concrete interpolation between known endpoints, and a deployable heuristic that avoids grid search. The unification derivation (Eqs. 1–10) is coherent and the paper is honest about several limitations, including the oracle nature of the grid-searched results, the lack of theory for the optimal exponent under curvature-estimation noise, and the heuristic status of ULoRA-Auto. The full learning-rate sweep and multi-seed reporting are good practice. However, the headline empirical claims are substantially weaker than the abstract suggests once the oracle protocol, small margins relative to seed noise, and the absence of any control for curvature-estimation noise are taken into account. The core mathematical framing is sound and publishable, but the paper's central empirical claims require additional validation.

major comments (3)
  1. [Results / Tables 2–4; Limitations] The headline 'matches or exceeds full fine-tuning' is based on the oracle row 'ULoRA (UB)', where the best (α,β) was selected per task on test metrics, as the Limitations explicitly concede. For example, in Table 2 the margins over full fine-tuning are 0.1 points on CoLA (81.1 vs 81.0), a tie on MNLI, 0.2 on MRPC, and several comparable rows in Table 3 are within one standard deviation. Selecting the best of 15 grid points on the test set and then reporting that point as the family's performance is not an honest model-selection protocol; it will overstate the achievable gains. The claim that 'no single fixed preconditioning strength dominates' and that the optimum is task-dependent also rests on test-set argmax, which is sensitive to noise. Please re-derive the main conclusions using validation-set selection or nested cross-validation, and report significance or effect sizes over seeds.
  2. [Eq. (2), Curvature Estimation; Limitations; ULoRA-Auto] The central empirical claim—that the optimal α lies strictly inside the continuum—is confounded by estimation noise in the K-FAC curvature factors. S and T are computed from a single small initialization batch (Eq. 2). Fractional inverse-power whitening with a noisy covariance shrinks small-eigenvalue directions, so an interior α may simply be the optimal shrinkage for estimation noise rather than a property of the true loss geometry. The paper provides no control experiment that varies the initialization batch size or the number of curvature samples to see whether the optimal α moves toward the endpoints (e.g., α=1 or α=2) as the estimate improves. The Limitations explicitly state that there is 'no theoretical characterization of the optimal exponent as a function of curvature-estimation noise,' and the ULoRA-Auto section concedes that the diagnostics 'measure estimation quality, not ta
  3. [ULoRA-Auto, Eq. (12); Figure 3] The paper concludes that ULoRA-Auto 'provides direct evidence that the right exponent is predictable from spectral statistics,' but the rule is an empirically motivated heuristic with ad hoc constants (2 in the denominator, exponent 2, and β=(1−ω)^2). The text itself states that the diagnostics 'measure estimation quality, not task-level utility,' which undercuts the predictability claim. Furthermore, Auto's per-layer exponents enlarge the search space relative to the global grid, so the fact that Auto beats the global grid on MMLU is not evidence of predictive validity—it may simply be a wider search. To support the predictability claim, the authors should validate the Auto-selected exponents against exponents selected on held-out validation data (not test data), or compare Auto's choices with a random per-layer assignment to control for the increased flexibility. As written, the claim
minor comments (7)
  1. [Eq. (1)] The vectors d_S and d_T are used in Eq. (1) but defined only in Eq. (3). Consider defining them before Eq. (1) or adding a parenthetical.
  2. [Curvature Estimation and Gradient Projection] The oversample size in the Rayleigh–Ritz step (s = r + oversample) is never specified. Since this is a free parameter that affects the curvature approximation, it should be reported (e.g., in Configurations or in Table 1).
  3. [Cross-Entropy Correction] The description of Φ is terse: it is not clear whether Φ is accumulated in the original space or the projected space, and whether the same α exponent is used in Φ and in F^{α,β}. Please make the exact computation explicit.
  4. [Figure 2] The fine grid step for the extended α,β∈[−3,+3] sweep is not stated. Please add the step size to the caption or text.
  5. [ULoRA-Auto] The text says 'βS near 1, βT low' but does not specify which of the two Kronecker sides S or T corresponds to input versus output. Define the convention clearly.
  6. [Global Normalization] The normalization factor γ=16 is a fixed hyperparameter; the paper does not discuss its sensitivity. A sentence on whether results change with γ would help.
  7. [Section heading] Section heading reads 'LoRA-Auto' while the rest of the paper uses 'ULoRA-Auto'. Please make the terminology consistent.

Circularity Check

0 steps flagged

No significant circularity; the unification is a definitional parameterization, the oracle grid is explicitly labeled, and the auto heuristic is a disclosed empirical rule.

full rationale

The paper's central theoretical object is Eq. (1), a two-parameter family P_{α,β} defined so that known optimizers and LoRA initializations correspond to particular exponent values. This is a definitional unification rather than a derivation chain: the family is constructed to contain those points, so the "recovery" of LoRA-GA, LoRA-One, CG-LoRA, Adam-like scaling, and Shampoo is by design. That is a legitimate organizational contribution, not a circular prediction, because the authors do not claim to derive the existing methods from independent principles. The main potential concern is the grid-searched ULoRA (UB) result reported as "matches or exceeds full fine-tuning." However, the paper labels this explicitly as an oracle upper bound and states in Limitations: "The grid-searched ULoRA numbers select (α, β) per task on test metrics, an oracle protocol; ULoRA-Auto is the deployable variant and should be read as our single-configuration result." Because the selection protocol is disclosed and the claim is presented as an upper bound, not as a deployable prediction, this does not rise to the level of a fitted-input-called-prediction circularity. ULoRA-Auto is a search-free heuristic. The paper itself says it "is a heuristic rather than a derived optimum" and concedes in Limitations that "the ULoRA-Auto rule is empirically motivated, and we provide no theoretical characterization of the optimal exponent as a function of curvature-estimation noise." A heuristic that is not derived is a weakness of theoretical support, not a circular derivation. Similarly, the statement that "the diagnostics measure estimation quality, not task-level utility" weakens the claim that spectral statistics predict task-level optimal exponents, but it is an honest limitation rather than a self-referential reduction. There are no load-bearing self-citations, no imported uniqueness theorems, and no evidence that a fitted parameter is being renamed as an independent prediction. The empirical grid-search results are benchmark observations, and the auto rule is an admittedly empirical rule. Overall, the paper's claims are either definitional, explicitly labeled oracle results, or disclosed heuristics; no step in the derivation chain reduces to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central derivation is a reparameterization of known K-FAC/Shampoo/Adam preconditioners, so the family itself needs few axioms; the fragile assumptions are empirical: reliability of single-batch curvature estimates, and the ULoRA-Auto rule as a hand-fitted heuristic.

free parameters (4)
  • α, β grid-search exponents (ULoRA UB) = see Tables 2-4 (e.g., RoBERTa MRPC (1.0,0.0), LLaMA GSM8K (0.0,0.0))
    Chosen per task on test metrics (oracle protocol, stated in Limitations), so the headline performance is a fitted upper bound.
  • ULoRA-Auto rule constants (2, 1, exponent 2) = α*=2/(1+ln κ/(ln s·reff)); β*=(1-ω)^2
    Hand-selected heuristic; paper admits it is 'empirically motivated' with 'no theoretical characterization' (Limitations, Auto section).
  • Normalization factor γ=16 = 16
    Chosen by hand in Eq. 10 section ('with γ=16'); no ablation or derivation.
  • Rayleigh-Ritz oversample size = unspecified
    s=r+oversample (Method, Eq. 4) but oversample value never given; affects the projected subspace.
axioms (5)
  • domain assumption K-FAC factorization approximates the layer Hessian as S⊗T (Eq. 2)
    Standard but approximate for non-linear networks; noise at 7B scale acknowledged in results.
  • domain assumption A single small initialization batch yields representative gradient and curvature statistics
    The whole method rests on this; paper notes 7B curvature estimates are 'the noisiest of our settings'.
  • domain assumption Rayleigh-Ritz projection with s=r+oversample captures the directions that matter for the subspace
    The low-rank extraction happens only in this projected basis (Eqs. 4-5).
  • standard math Cross-entropy output curvature Λ=diag(p)-pp^T from Martens and Grosse (2015) applies and the Fisher correction in Step 7 is valid
    Taken from cited K-FAC literature; details delegated to CG-LoRA preprint.
  • ad hoc to paper ULoRA-Auto spectral diagnostics (reff, κ, ω) predict task-level optimal exponents
    Eq. (12) is asserted, not derived; paper says diagnostics 'measure estimation quality, not task-level utility'.

pith-pipeline@v1.3.0-alltime-deepseek · 11872 in / 17266 out tokens · 150192 ms · 2026-08-01T00:22:13.215893+00:00 · methodology

0 comments
read the original abstract

Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.

Figures

Figures reproduced from arXiv: 2607.26247 by Dianze Liu, Farshid Ghezelbash.

Figure 1
Figure 1. Figure 1: Overview of ULoRA. Top left: the continuum interpolation space spanned by spectral whitening ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Heatmap of the fine grid search for α, β ∈ [−3, +3] on GLUE MRPC, averaged across 3 seed runs. The maximum accuracy is marked by a star. Limitations. The grid-searched ULoRA numbers select (α, β) per task on test metrics, an oracle protocol; ULoRA￾Auto is the deployable variant and should be read as our single-configuration result. The grid-searched family further assumes a single global (α, β) shared acro… view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer exponents selected by ULoRA-Auto [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 9 linked inside Pith

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  9. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  10. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  11. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  12. [12]

    International Conference on Learning Representations , year=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations , year=

  13. [13]

    Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages=

    Understanding the difficulty of training deep feedforward neural networks , author=. Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages=. 2010 , organization=

    2010

  14. [14]

    Proceedings of the IEEE international conference on computer vision , pages=

    Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification , author=. Proceedings of the IEEE international conference on computer vision , pages=

  15. [15]

    Advances in Neural Information Processing Systems , volume=

    PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models , author=. Advances in Neural Information Processing Systems , volume=. 2024 , url=

    2024

  16. [16]

    International Conference on Learning Representations (ICLR) , year=

    AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning , author=. International Conference on Learning Representations (ICLR) , year=

  17. [17]

    Valipour, Mojtaba and Rezagholizadeh, Mehdi and Kobyzev, Ivan and Ghodsi, Ali , booktitle=. Dy

  18. [18]

    Wang, James and others , booktitle=

  19. [19]

    Hong, Han and others , booktitle=

  20. [20]

    Neural Computation , volume=

    Natural Gradient Works Efficiently in Learning , author=. Neural Computation , volume=. 1998 , publisher=

    1998

  21. [21]

    Journal of Machine Learning Research , volume=

    New Insights and Perspectives on the Natural Gradient Method , author=. Journal of Machine Learning Research , volume=

  22. [22]

    , booktitle =

    Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R. , booktitle =

  23. [23]

    Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin , journal =

  24. [24]

    Learning Multiple Layers of Features from Tiny Images , author =

  25. [25]

    Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops , pages =

    3D Object Representations for Fine-Grained Categorization , author =. Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops , pages =

  26. [26]

    IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , volume=

    EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification , author=. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , volume=. 2019 , publisher=

    2019

  27. [27]

    International Conference on Learning Representations (ICLR) , year =

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author =. International Conference on Learning Representations (ICLR) , year =

  28. [28]

    Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others , journal =

  29. [29]

    Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages=

    Optimizing neural networks with kronecker-factored approximate curvature , author=. Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages=. 2015 , organization=

    2015

  30. [30]

    arXiv preprint arXiv:2602.04998 , year=

    Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning , author=. arXiv preprint arXiv:2602.04998 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2502.01235 , year=

    Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently , author=. arXiv preprint arXiv:2502.01235 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2603.29824 , year=

    Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space , author=. arXiv preprint arXiv:2603.29824 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2312.03732 , year=

    A rank stabilization scaling factor for fine-tuning with lora , author=. arXiv preprint arXiv:2312.03732 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2402.12354 , year=

    Lora+: Efficient low rank adaptation of large models , author=. arXiv preprint arXiv:2402.12354 , year=

    Pith/arXiv arXiv

  36. [36]

    Forty-first International Conference on Machine Learning , year=

    Dora: Weight-decomposed low-rank adaptation , author=. Forty-first International Conference on Machine Learning , year=

  37. [37]

    International Conference on Learning Representations , volume=

    Loftq: Lora-fine-tuning-aware quantization for large language models , author=. International Conference on Learning Representations , volume=

  38. [38]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Parameter Efficient Fine-tuning via Explained Variance Adaptation , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  39. [39]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  40. [40]

    Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=

    GLUE: A multi-task benchmark and analysis platform for natural language understanding , author=. Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=

    2018

  41. [41]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2108.02818 , year=

    Evaluating clip: towards characterization of broader capabilities and downstream implications , author=. arXiv preprint arXiv:2108.02818 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv , year =

    Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and Bikel, Dan and Blecher, Lukas and Ferrer, Cristian Canton and Chen, Moya and Cucurull, Guillem and Esiobu, David and Fernandes, Jude and Fu, Jeremy and Fu, Wenyi...

  44. [44]

    and Li, Zhenguo and Weller, Adrian and Liu, Weiyang , title =

    Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T. and Li, Zhenguo and Weller, Adrian and Liu, Weiyang , title =. arXiv , year =

  45. [45]

    arXiv , year =

    Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John , title =. arXiv , year =

  46. [46]

    arXiv , year =

    Wei, Yuxiang and Wang, Zhe and Liu, Jiawei and Ding, Yifeng and Zhang, Lingming , title =. arXiv , year =

  47. [47]

    arXiv , year =

    Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and Ray, Alex and Puri, Raul and Krueger, Gretchen and Petrov, Michael and Khlaaf, Heidy and Sastry, Girish and Mishkin, Pamela and Chan, Brooke and Gray, Scott and...

  48. [48]

    The Twelfth International Conference on Learning Representations (ICLR 2024) , year =

    Xu, Can and Sun, Qingfeng and Zheng, Kai and Geng, Xiubo and Zhao, Pu and Feng, Jiazhan and Tao, Chongyang and Lin, Qingwei and Jiang, Daxin , title =. The Twelfth International Conference on Learning Representations (ICLR 2024) , year =

    2024

  49. [49]

    arXiv , year =

    Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob , title =. arXiv , year =

  50. [50]

    International Conference on Machine Learning , pages=

    Shampoo: Preconditioned stochastic tensor optimization , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  51. [51]

    International Conference on Machine Learning , pages=

    Attention is not all you need: Pure attention loses rank doubly exponentially , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  52. [52]

    arXiv preprint arXiv:2002.09018 , year=

    Scalable second order optimization for deep learning , author=. arXiv preprint arXiv:2002.09018 , year=

    Pith/arXiv arXiv 2002

  53. [53]

    2007 15th European Signal Processing Conference , pages=

    The effective rank: A measure of effective dimensionality , author=. 2007 15th European Signal Processing Conference , pages=. 2007 , organization=

    2007

  54. [54]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Adahessian: An adaptive second order optimizer for machine learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  55. [55]

    Hyperparameter Optimization in Machine Learning: Make Your Machine Learning and Deep Learning Models More Efficient , pages=

    Optuna and autoML , author=. Hyperparameter Optimization in Machine Learning: Make Your Machine Learning and Deep Learning Models More Efficient , pages=. 2020 , publisher=

    2020