Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OSoRA: adapting a language model by tuning only singular values and one output-dimension vector matches or beats LoRA, VeRA, and PiSSA with r+d trainable parameters.

desk verdict Eq. (5) cancels the adaptation term, so OSoRA as written is a no-op; the underlying idea is salvageable but the paper needs major revision. read the letter →

arxiv 2505.14350 v2 pith:YE26JPZL submitted 2025-05-20 cs.CL

Jialong Han , Si Zhang , Ke Zhang This is my paper
classification cs.CL
keywords parameter-efficientfine-tuninglow-rankadaptationsingularvaluedecompositionlearnablescalingvectorslargelanguagemodelsSVDinitializationLoRAextensionmodelmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OSoRA is a parameter-efficient fine-tuning method for large language models that claims a full low-rank adaptation can be driven by just $r+d$ learnable scalars: the top-$r$ singular values of a pretrained weight matrix plus one output-dimension scaling vector, with the corresponding singular vectors frozen. The paper argues that the top singular subspace of the original weights already carries the directions a downstream task needs, so during training only the strengths of those directions and their per-output scaling have to be adjusted. On commonsense reasoning and mathematics benchmarks across LLaMA2-13B, Qwen1.5-7B, Qwen2.5-32B, Mistral-7B, and LLaMA3-8B, OSoRA reports accuracy comparable to or better than LoRA, VeRA, DoRA, and PiSSA at a fraction of the trainable parameter count. Ablations support the joint training of both learnable vectors, and the paper shows the trained vectors can be merged back into a single weight matrix so inference adds no overhead. The appeal is that it makes fine-tuning larger models feasible on limited hardware without giving up accuracy.

What carries the argument

The central object is the SVD-based reparameterization of Equation (4): $y = W'_0 x + \Lambda_O U_r \Lambda_{S_r} V_r^\top x$. Here $U_r \in \mathbb{R}^{d\times r}$ and $V_r \in \mathbb{R}^{k\times r}$ are the frozen left and right singular vectors belonging to the $r$ largest singular values of the pretrained weight $W_0$, and $\Lambda_O$ and $\Lambda_{S_r}$ are diagonal matrices built from the two trainable vectors: $O \in \mathbb{R}^d$ scales each output dimension, while $S_r \in \mathbb{R}^r$ weights each rank component. Because the singular vectors are fixed, the parameter count is $r+d$ regardless of the input dimension $k$, growing linearly rather than exponentially in $r$. The same vectors that are frozen during training are also what makes merging possible: after training, $\Lambda_O U_r \Lambda_{S_r} V_r^\top$ is a rank-$r$ matrix that can be added into $W'_0$, so inference is just the original model with a modified weight matrix.

What would settle it

A direct falsifier is to take OSoRA and replace the SVD-derived $U_r, V_r$ in Equation (4) with a random orthogonal basis, as VeRA does, while keeping the same initialization of $S_r$ from $W_0$'s singular values and the same all-ones $O$. If this random-basis variant matches OSoRA's accuracy on the reported benchmarks, then the top-singular-subspace assumption is not doing the work the paper assigns to it.

Watch

Extended reading notes

Core claim

The central claim is that adapting a large language model does not require learning a full low-rank update matrix. After computing the SVD of a pretrained weight matrix $W_0 = U\Sigma V^\top$, OSoRA freezes the top-$r$ singular vectors $U_r$ and $V_r$ and learns only the corresponding singular values $S_r \in \mathbb{R}^r$ and an output-dimension scaling vector $O \in \mathbb{R}^d$. The adapted forward pass is $y = W'_0 x + \Lambda_O U_r \Lambda_{S_r} V_r^\top x$, with $W'_0$ the original weight minus the top-$r$ component, so the entire task-specific update lives in the fixed subspace spanned by the top singular vectors. The paper reports that this $r+d$-parameter scheme matches or exceeds LoRA, VeRA, PiSSA, and DoRA on commonsense reasoning and mathematical reasoning benchmarks, and that ablations show both $S_r$ and $O$ must be trained jointly for best results.

Load-bearing premise

The method assumes that the dominant directions of the pretrained weights are already the directions a new task needs; if the task requires changing the model along directions those dominant directions do not cover, OSoRA has no way to express the change.

Editorial extensions

If this is right

  • OSoRA makes the parameter cost of fine-tuning independent of the input dimension $k$, so the same budget covers very wide matrices; for example, a rank-512 adaptation on a 7B model uses roughly $r+d$ instead of $r(d+k)$ trainable parameters.
  • After training, the learned vectors merge into a single weight matrix, so deployed checkpoints are identical in size to the original model and inference adds no extra memory or computation.
  • Because only $S_r$ and $O$ are optimized, optimizer memory is minimal, which permits larger ranks (512, 1024) before memory constraints bite.
  • The ablations imply that a PEFT method can spend its entire trainable budget on spectral scaling plus per-output rescaling, and that initializing $O$ with ones is a necessary ingredient.
  • Composing OSoRA with DoRA improves MATH and GSM8K further with roughly 360 thousand trainable parameters versus DoRA's roughly 7 million, indicating the method composes with other PEFT decompositions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: OSoRA is effectively a spectral gating operation—it removes the top-$r$ component of $W_0$ and re-inserts it with rescaled singular values—so tasks whose desired updates align with the principal subspace should benefit most, while tasks needing orthogonal directions should hit a performance ceiling; replacing $U_r$, $V_r$ with a random orthogonal basis while keeping the same $S_r$ initi
  • Inference: Since $O$ has one scalar per output dimension, OSoRA can express per-output rescaling that LoRA cannot with the same rank; on attention projections this may amount to reweighting attention heads, a hypothesis the paper does not test.
  • Inference: By freezing $U_r$ and $V_r$, the effective hypothesis space is a rank-$r$ manifold with fixed tangent directions; the method should be most attractive when the task's required update is concentrated in the top-$r$ singular subspace of the pretrained weights, and a cheap predictor of degradation is the energy of the task update orthogonal to that subspace, $\mathrm{tr}(\Delta W^\top (I -
  • Inference: The paper's ablation that fixing $O$ (OSoRA**) hurts more than fixing $S_r$ suggests the output-dimension vector carries much of the adaptation signal; a testable extension is to initialize $O$ from a trained LoRA's row norms rather than all-ones and check whether that closes the remaining gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OSoRA, a parameter-efficient fine-tuning method that performs SVD on pretrained weight matrices, freezes the top-r singular vector matrices, and trains only a singular-value vector S_r and an output-dimension vector O. The authors claim that OSoRA matches or exceeds LoRA, VeRA, DoRA, and PiSSA on commonsense-reasoning and mathematics benchmarks while using only r+d trainable parameters. The paper includes parameter-count analysis, gradient formulas, rank-robustness experiments, and ablation studies.

Significance. If the method were correctly specified, OSoRA would be a useful contribution to PEFT: it combines the parameter efficiency of VeRA with an SVD-based initialization inspired by PiSSA, and the experimental evaluation covers multiple model families and benchmarks without circularity. The paper also ships honest limitations, including the unproven fixed-subspace premise. However, as written, the central formulation is internally inconsistent: Equation (5) defines the frozen residual in terms of the trainable matrices, which makes the forward pass exactly W0x and the method vacuous. This is a load-bearing error that must be corrected before the reported results can be attributed to the stated method.

major comments (3)
  1. [§3.2, Eqs. (4)–(5)] The definition of W'_0 in Eq. (5) uses the trainable matrices Λ_O and Λ_Sr. Substituting Eq. (5) into Eq. (4) gives y = W0x identically, for any values of O and S_r, so no adaptation occurs as written. The accompanying text says W'_0 is the frozen component after excluding the top-r singular contribution, which indicates the intended definition should use the initial singular values, e.g., W'_0 = W0 − U_r Σ_r V_r^T. This is not a typo in an isolated formula: the same definition propagates to the merged-weight Eq. (7) and the DoRA combination Eq. (12), and it determines what the benchmark tables in Section 4 claim to measure. Please correct Eq. (5) (and any downstream uses) and state explicitly that W'_0 is frozen with respect to the initial SVD values.
  2. [§3.6, Eq. (10)] Equation (10) as printed has a dimension mismatch: ∂L/∂∆W is d×k, U_r is d×r, Λ_Sr is r×r, and V_r^T is r×k, so the product is d×k and its diag is undefined. The correct chain-rule expression for ∂L/∂O should be diag( ∂L/∂∆W V_r Λ_Sr U_r^T ), up to ordering and transposes. This formula is part of the theoretical justification for the optimization dynamics, so it needs to be fixed or removed if it is only illustrative.
  3. [§4, Tables 1–6 and §6] The experimental results cannot be interpreted as evidence for OSoRA until the Eq. (5) issue is resolved, because the stated forward pass is identical to the frozen pretrained model. The authors should either confirm that the implementation used W'_0 = W0 − U_r Σ_r V_r^T (or an equivalent frozen residual) and add a numerical check that the merged weight at initialization equals W0 exactly, or provide code. The Limitations section already concedes that the fixed-subspace premise is unproven, but that is secondary to the need for a non-vacuous formulation.
minor comments (5)
  1. [§4.4, Figure 3] Figure 3 includes a VeRA bar in the MATH/GSM8K comparison, but the text describes only three OSoRA variants; please clarify the figure legend and the comparison intent.
  2. [Appendix A, Table 7] In Table 7, multiple methods (LoRA, DoRA, PiSSA) share merged cells for r and learning rate, which makes it difficult to see which hyperparameters apply to each method; please separate the rows or use a clearer layout.
  3. [§1] The citation of Abacha et al. (2025), a medical-error-detection paper, to motivate the computational cost of fine-tuning large models, seems off-topic; a general LLM-scaling citation would be more appropriate.
  4. [§4.4, Table 4 and surrounding text] The paper uses both 'OSoRAG' and 'OSoRA G' for the Gaussian-initialized variant; please standardize the notation.
  5. [§3.5, Eq. (8)] The parameter-ratio equation is correct, but the intermediate step would be easier to follow if the factorization (r+d)/r(d+k) were shown as 1/(d+k) + d/[r(d+k)] explicitly before the limit statement.

Circularity Check

1 steps flagged · score 10.0 of 10

Eq. (5) defines W'_0 as W0 minus the exact learnable term that Eq. (4) adds back, so the OSoRA forward pass is identically W0x and the central adaptation claim is vacuous as written.

  1. self definitional [Section 3.2, Eq. (4)-(5); also Section 3.3, Eq. (7)]
    "The adaptation can be formulated as: y = W ′ 0x + ∆W x= W ′ 0x + ΛOUrΛSr V ⊤ r x (4) ... W ′ 0 represents the frozen component of the weight matrix after excluding the contribution of the top r singular values and the corresponding singular vectors, which can be written as: W ′ 0 = W0 − ΛOUrΛSr V ⊤ r (5)"

    Substituting Eq. (5) into Eq. (4) yields y = W0x identically, because Eq. (5) subtracts precisely the term Eq. (4) adds. ΛO and ΛSr are the learnable quantities (O initialized as all-ones, Sr learnable singular values), so W'_0 is not a frozen initial residual; it cancels the adapted contribution at every training step. The forward pass of OSoRA is therefore by construction identical to the pretrained model, and the claimed adaptation, ablations, and benchmark gains cannot be produced by the stated equations. The same cancellation is repeated in the merge Eq. (7), W = W'_0 + ΛOUrΛSrVr^T = W0. The Limitations section's fixed-subspace caveat is secondary; as written the update is definitionally vacuous.

full rationale

The central construction is self-definitional: Eq. (4) defines the adapted output as W'_0 x plus the low-rank term, and Eq. (5) defines W'_0 as W0 minus that same low-rank term, so the two cancel exactly and y = W0x for all values of the trainable vectors. Thus OSoRA, as written, is equivalent to doing nothing, and the paper's central claim that updating only singular values and one output-dimension vector suffices for adaptation reduces by construction to its input (the unadapted pretrained weight). I found no same-author self-citations in the reference list and no fitted-parameter-then-predicted benchmark circularity; the experiments are evaluated on external benchmarks against published baselines, which would be independent evidence if the equations defined a non-vacuous update. The Section 6 Limitations statement that the fixed top-singular-vector subspace may fail on tasks needing large departures is an honest empirical caveat and is noted, but it does not repair the Eq. (4)-(5) cancellation. Other issues, such as describing LoRA's parameter count as growing exponentially when it is r(d+k), are correctness concerns outside circularity. Because the adapted forward pass is exactly the base forward pass by definition, the circularity score is 10.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. O and Sr are standard trainable parameter vectors. The load-bearing assumptions are about the suitability of the SVD subspace and the sufficiency of the two scaling vectors, neither of which is proven.

assumptions (3)
  • domain assumption The top r singular vectors of the pretrained weight matrix provide a suitable subspace for adaptation.
    Assumed in Eq. (4) with Ur and Vr frozen; no theory proves this subspace is sufficient, only empirical results.
  • domain assumption A single output-dimension scaling vector O plus singular value scaling Sr suffices to express the needed update.
    This is the paper's central hypothesis; tested only through ablations and benchmark accuracy, not derived.
  • domain assumption The frozen residual W'_0 can be precomputed and stored without affecting training.
    Implied by Eq. (5) if corrected to use initial values; the paper does not discuss numerical stability or storage of W'_0.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/YE26JPZL

@misc{pith2026250514350,
  author       = {Pith},
  title        = {Pith review of: OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YE26JPZL}},
  note         = {Machine review of arXiv:2505.14350}
}
read the original abstract

Fine-tuning Large Language Models (LLMs) has become increasingly challenging due to their massive scale and associated computational costs. Parameter-Efficient Fine-Tuning (PEFT) methodologies have been proposed as computational alternatives; however, their implementations still require significant resources. In this paper, we present OSoRA (Output-Dimension and Singular-Value Initialized Low-Rank Adaptation), a novel PEFT method for LLMs. OSoRA extends Low-Rank Adaptation (LoRA) by integrating Singular Value Decomposition (SVD) with learnable scaling vectors in a unified framework. It first performs an SVD of pre-trained weight matrices, then optimizes an output-dimension vector during training, while keeping the corresponding singular vector matrices frozen. OSoRA substantially reduces computational resource requirements by minimizing the number of trainable parameters during fine-tuning. Comprehensive evaluations across mathematical reasoning, common sense reasoning, and other benchmarks demonstrate that OSoRA achieves comparable or superior performance to state-of-the-art methods like LoRA and VeRA, while maintaining a linear parameter scaling even as the rank increases to higher dimensions. Our ablation studies further confirm that jointly training both the singular values and the output-dimension vector is critical for optimal performance.

Figures

Figures reproduced from arXiv: 2505.14350 by the authors.

Figure 1
Figure 1. Parameter count comparison among adap￾tation methods at varying ranks on Qwen2-7B model. The results demonstrate that LoRA exhibits exponen￾tial growth in trainable parameters with increasing rank, whereas both VeRA and OSoRA maintain efficient lin￾ear scaling in their parameter count. Parameter-Efficient Fine-Tuning (PEFT) method￾ologies, which enable fine-tuning LLMs by selec￾tively updating only a minimal subset … view at source ↗
Figure 2
Figure 2. Schematic comparison of LoRA (left), VeRA (middle) and OSoRA (right). LoRA adapts pretrained [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ablation study on the impact of training dif [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Training only per-member singular values of pretrained weight matrices, while freezing singular vectors, produces an implicit ensemble with calibration near deep ensembles at <1% parameter overhead.

Reference graph

Works this paper leans on

16 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [4]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, V olume 1 (Long and Short Papers) , pages 2924–2936, Min- neapolis, Minnesota. Association for Computational Linguistics. Peter Clark,...

    work page 2019

  2. [9]

    arXiv preprint arXiv:2310.06825

    Mistral 7B. arXiv preprint arXiv:2310.06825. Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano

    arXiv

  3. [10]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045–3059, Online and Punta Cana, Domini- can Republic

    The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045–3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, a...

    work page 2021

  4. [11]

    arXiv preprint arXiv:2110.07602

    P- tuning v2: Prompt tuning can be comparable to fine- tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602. Fanxu Meng, Zhaohui Wang, and Muhan Zhang

    arXiv

  5. [12]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381–2391, Brussels, Belgium

    Can a Suit of Armor Conduct Elec- tricity? A New Dataset for Open Book Question An- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Da...

    work page 2018

  6. [13]

    arXiv preprint arXiv:2412.15115

    Qwen2.5 Tech- nical Report. arXiv preprint arXiv:2412.15115. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman

    arXiv

  7. [14]

    Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 4463– 4473, Hong Kong, China. Association for Computa- tional Linguistics. Alon Talmor, Jonathan Herzig, N...

    work page 2019

  8. [15]

    CommonsenseQA: A Ques- tion Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Qwen Team

    work page 2019

Show all 16 references
  1. [16]

    arXiv preprint arXiv:2307.09288

    Llama 2: Open Founda- tion and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    arXiv

  2. [2018]

    arXiv preprint arXiv:1803.05457

    Think you have Solved Question An- swering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al

    arXiv

  3. [2019]

    arXiv preprint arXiv:1911.11641

    PIQA: Reasoning about Physical Commonsense in Natural Language. arXiv preprint arXiv:1911.11641. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, and Amanda Askell

    1911 arXiv

  4. [2020]

    In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901

    Language Models are Few-Shot Learn- ers. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Asso- ciates, Inc. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova

    1901

  5. [2021]

    arXiv preprint arXiv:2110.14168

    Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. OpenCompass Contributors

    arXiv

  6. [2023]

    arXiv preprint arXiv:2305.14314

    QLoRA: Efficient Finetuning of Quantized LLMs. arXiv preprint arXiv:2305.14314. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan

    arXiv

  7. [2024]

    arXiv preprint arXiv:2407.21783

    The Llama 3 Herd of Mod- els. arXiv preprint arXiv:2407.21783. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt

    arXiv

  8. [2025]

    arXiv preprint arXiv:2412.19260

    MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes. arXiv preprint arXiv:2412.19260. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.