Pith. sign in

REVIEW 4 major objections 5 minor 13 references

UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a VeRA-style frozen-matrix adapter, whose only trainable parameters are two scaling vectors, can match or beat LoRA's fine-tuning performance across NLU, NLG, instruction-tuning, and image classification, provided…

desk verdict A genuinely new PEFT mechanism that looks competitive, but the paper's storage accounting undercounts what UORA actually needs to ship. read the letter →

arxiv 2505.20154 v1 pith:CSAN5QTG submitted 2025-05-26 cs.CL

Xueyan Zhang , Jinman Zhao , Zhifei Yang , Yibo Zhong , Shuhao Guan , Linbo Cao , Yining Wang This is my paper
classification cs.CL
keywords parameter-efficientfine-tuninglow-rankadaptationfrozenrandomprojectionscalingvectorsinterpolation-basedreinitializationLoRAVelargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Uniform Orthogonal Reinitialization Adaptation (UORA), a parameter-efficient fine-tuning method that keeps the frozen random projection matrices of VeRA and trains only two scaling vectors, but periodically reinitializes rows and columns of those frozen matrices when the learned scaling entries are small. This lets UORA use a rank close to LoRA's instead of VeRA's much larger ranks, reporting competitive or better performance with roughly 15x fewer trainable parameters than LoRA on GLUE and 8x fewer on E2E. The authors also show competitive results on arithmetic instruction tuning with LLaMA models and on ViT image classification. If the method holds up, it means the expensive low-rank matrices in LoRA-style adapters can be replaced by frozen matrices plus a cheap, selective refresh rule.

What carries the argument

The central mechanism is interpolation-based reinitialization of frozen low-rank matrices guided by scaling-vector magnitude. UORA adopts VeRA's parameterization where A and B are frozen random matrices and only diagonal scaling vectors Λ_d and Λ_b are trained, but adds a rule: when a scaling entry falls below threshold τ for k consecutive steps, the matching frozen column and row are refreshed by linear interpolation v_new = α v_old + (1−α) v_rand. This refresh is what lets UORA operate at ranks comparable to LoRA (e.g., 16–32) instead of VeRA's typical 256–1024, keeping the trainable parameter count at roughly d_model + r per tuned layer.

What would settle it

Run UORA on a GLUE task such as MRPC with four configurations: reinitialization disabled (k=0), magnitude-guided reinitialization as proposed, random indices reinitialized at the same rate, and reinitialization of the largest-magnitude indices. If randomly reinitialized frozen directions match or beat the magnitude-guided version, or if reinitializing the largest-magnitude directions performs equally well, then the claim that small scaling-vector entries mark unimportant frozen dimensions is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper claims that selective reinitialization of frozen projection matrices restores the expressiveness that VeRA loses when its rank is reduced, so that scaling-vector adaptation can match LoRA with far fewer trainable parameters. Concretely, UORA freezes orthogonally initialized random matrices A and B and trains scaling vectors d and b such that the weight update is ΔW = Λ_b B Λ_d A, and whenever an entry of d stays below a threshold τ for k consecutive steps, the corresponding column of A and row of B are replaced via v_new = α v_old + (1−α) v_rand. On GLUE, UORA reports an average of 86.5 on RoBERTa-base with 0.019M trainable parameters versus LoRA's 85.2 with 0.3M, and 88.5 on RoBERTa-large with 0.049M; on E2E it reports BLEU 66.67 on GPT-2 Medium with 0.051M parameters versus LoRA's 67.14 with 0.4M.

Load-bearing premise

The method's gains rest on the assumption that a small entry in the trained scaling vector reliably marks the corresponding frozen row and column as unimportant, so replacing those directions helps training rather than discarding useful structure.

Editorial extensions

If this is right

  • UORA cuts the trainable parameter count of LoRA-style adaptation by roughly an order of magnitude on GLUE (15x) and E2E (8x) with no reported loss in average performance.
  • Because the projection matrices stay frozen and only vectors are trained, the learned update can still be merged into the pretrained weights for zero inference latency.
  • UORA can use ranks close to LoRA's where VeRA needed much larger ranks, reducing both the compute during training and the bytes of stored adapter weights.
  • The reinitialization rule transfers across domains, including GLUE, E2E generation, arithmetic instruction tuning on LLaMA models, and ViT image classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The magnitude heuristic is a proxy: a small learned scaling value does not necessarily prove the corresponding frozen direction is useless, so reinitialization may be acting as a structured exploration mechanism or as an implicit way to increase effective rank; a test that distinguishes these is comparing magnitude-guided reinitialization against random reinitialization at the same rate.
  • UORA's success suggests the frozen random matrices in VeRA are the bottleneck at low rank, and periodic refresh is a cheaper substitute for making them trainable, pointing to a broader design space where frozen components are refreshed rather than updated by gradients.
  • The threshold and count hyperparameters (τ, k) are tuned per task, and an adaptive criterion based on gradient signal or validation loss could be a natural testable extension that removes the manual tuning burden.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes UORA (Uniform Orthogonal Reinitialization Adaptation), a VeRA-style PEFT method in which two low-rank matrices A and B are randomly initialized and frozen, while trainable scaling vectors d and b parameterize the update. The novel mechanism is that whenever an entry of d remains below a threshold tau for k consecutive steps, the corresponding column of A and row of B are reinitialized using linear interpolation with factor alpha. The authors claim state-of-the-art parameter efficiency, reporting roughly 15x and 8x fewer trainable parameters than LoRA on GLUE and E2E, and present experiments on GLUE, E2E, arithmetic reasoning, and image classification, together with ablations of initialization, tau, k, and alpha.

Significance. If the claims held, the paper would make a useful empirical contribution to PEFT by showing that selectively reinitializing low-magnitude frozen directions can reduce the number of trainable parameters relative to LoRA while retaining competitive performance. The code is released, and the ablations over tau, k, alpha, and initialization are a strength. However, the central efficiency claim is weakened by a storage-accounting inconsistency and by incomplete benchmark reporting, so the significance as stated is not yet established.

major comments (4)
  1. [Section 3.1, Eqs. (3)-(4); Table 1; Section 6] The advertised parameter efficiency is not a like-for-like storage comparison. UORA overwrites entries of the frozen matrices A and B during training (Eqs. 3-4), and Section 6 explicitly acknowledges that "UORA updates the frozen matrices." These modifications depend on the training trajectory, so the final adapter cannot be reconstructed from the initial random seed plus the trained vectors d and b unless all reinitialization events are logged. Without such logging, deploying the adapter requires storing the trained A and B, which have the same dimensions as LoRA's matrices (2*d_model*r per layer). Table 1 nonetheless charges UORA only for d+b, understating its storage footprint by roughly a factor of 2*d_model*r/(d_model+r). Consequently, the 15x and 8x parameter reductions claimed in the abstract and Section 5, and the storage-efficiency comparisons in Table 1, are not valid as stated.
  2. [Section 4.1, Table 2] The GLUE evaluation omits MNLI and QQP, the two largest tasks in the benchmark, and the reported "Avg." is computed over the remaining six tasks. This average is not directly comparable to standard GLUE averages reported in prior LoRA/VeRA papers, and the claim of a 15x parameter reduction is based on this nonstandard average. The paper should either complete the GLUE benchmark or clearly label the average as a partial-GLUE average and avoid direct comparisons with full-GLUE results from prior work.
  3. [Sections 1 and 4; Tables 2-5] The claim of achieving "state-of-the-art" parameter efficiency is not supported by the baselines chosen. The comparison set lacks recent PEFT methods that also report very low trainable-parameter counts, such as PiSSA, DoRA, or other LoRA variants, and the instruction-tuning and vision experiments include only LoRA, VeRA, and (in vision) head/full fine-tuning. Without a broader baseline set, the "state-of-the-art" assertion is unsupported. Additionally, the E2E results in Table 3 and instruction-tuning results in Table 4 are reported without standard deviations or significance tests, despite small performance differences among the methods.
  4. [Section 3.2 and Appendix B] The core mechanism rests on the heuristic that a small magnitude in the scaling vector d reliably marks the corresponding frozen row/column as unimportant. This is asserted with a citation to a pruning paper (Sun et al., 2024), but no direct evidence is provided for VeRA-style adapters, and the behavior is controlled by tuned thresholds tau and counts k that vary across tasks (Tables 6-9). The ablations in Appendix B show sensitivity to tau and k on single tasks, but they do not test the heuristic itself, e.g., by comparing the reinitialized dimensions against an oracle or a random-dimension baseline. Without such a test, the risk remains that the performance gains could come from the extra stochasticity of reinitialization rather than from the magnitude heuristic.
minor comments (5)
  1. [Section 6] The Limitations section contains a typo: "Altough" should be "Although."
  2. [Section 3.2] The phrase "UoRA" is used in the Figure 1 caption and elsewhere with inconsistent capitalization; please use "UORA" uniformly.
  3. [Section 4.3] The dataset name "SV AMP" is written with an extra space and should be "SVAMP" throughout.
  4. [Appendix A.1] The tuning strategy reports that rank 32 is needed for large models and that tau=1e-4 is best on AddSub, yet the GLUE experiments use tau values of 1e-5 and 8e-6. A brief explanation of how the final hyperparameters were selected across tasks would improve reproducibility.
  5. [Related Work] Several references (e.g., Cai et al., 2025; Bi et al., 2025a; Sun et al., 2025) are not directly used by the method or the experiments; tightening the related-work section to the PEFT and reinitialization literature would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: UORA's performance results are empirically independent of its fitted heuristics, and the only self-citations are non-load-bearing related-work mentions.

full rationale

I walked the paper's derivation chain. The forward pass (Eq. 2) is inherited from VeRA, and the reinitialization mechanism (Eqs. 3-4) is an algorithmic intervention justified by an external magnitude heuristic cited to Sun et al. (2024), not by a self-citation. The central performance claims are empirical benchmark results, not quantities derived from fitted parameters. Hyperparameters such as tau, k, and alpha are tuned on the same tasks where performance is reported, but the paper presents them as tuned hyperparameters, not as predictions, so the tuned-input-called-prediction pattern does not apply. The parameter-count equation (Eq. 5) is a definition, not a derivation, and the claimed 15x/8x parameter reductions are arithmetic comparisons under that definition. There is a genuine accounting concern: Section 3.1 says A and B are reinitialized based on the training trajectory, and Section 6 admits 'UORA updates the frozen matrices for improved performance,' yet Table 1 counts only d and b for UORA storage; this is an internal efficiency-accounting inconsistency, but it is not circularity because no performance result is equivalent to the fitted values. The only self-citations in the paper (e.g., Hu et al. 2024 and Fang et al. 2023, which include UORA co-authors) appear in related-work surveys and are not load-bearing for the method's claims. Overall, the derivation chain is empirically self-contained and does not reduce to its inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. The contribution is a training-time update rule applied to existing VeRA-style frozen matrices. The central claim relies on the empirical magnitude heuristic and on hyperparameters tau, k, alpha, and rank, all tuned on the evaluated tasks.

free parameters (5)
  • tau (reinitialization threshold) = 1e-5 (SST-2, MRPC, CoLA), 8e-6 (QNLI, RTE, STS-B), 5e-5 (instruction tuning), 1 (E2E)
    Controls when a low-magnitude scaling dimension is considered unused and triggers reinitialization. Tuned per dataset in Appendix A.
  • k (consecutive count) = 1 (most tasks)
    Number of consecutive iterations a dimension must stay below tau before reinitialization. Tuned to balance reinitialization frequency.
  • alpha (interpolation factor) = 0.7
    Blending factor between old and random values in Equation 4. Chosen empirically; Table 13 shows performance varies with alpha.
  • rank r = 16 (RoBERTa GLUE), 32 (E2E, instruction tuning, ViT)
    Rank of the frozen random matrices. Much lower than VeRA's rank 1024 in most experiments, which is key to the claimed efficiency gain.
  • initial value of scaling vector d = 0.1
    Initialization taken from VeRA. Affects the early reinitialization dynamics and is a hand-chosen constant.
assumptions (4)
  • domain assumption Pretrained weight updates can be captured by frozen random low-rank matrices A, B with trainable diagonal scaling vectors.
    Adopted from VeRA (Kopiczko et al., 2023). UORA inherits this parameterization, and its central claim depends on it.
  • domain assumption Scaling-vector magnitude indicates dimension importance.
    Stated in Section 3.2, citing Sun et al. (2024). This heuristic justifies choosing which frozen rows and columns to reinitialize.
  • ad hoc to paper Reinitializing low-magnitude frozen dimensions improves expressiveness and training without disrupting convergence.
    The core mechanism of UORA. No proof is given; the support is empirical and depends on tuned tau, k, and alpha. If this fails, the method's advantage over VeRA disappears.
  • domain assumption Orthogonal uniform initialization improves gradient flow and training stability.
    Invoked in Section 3.2 with citations to Hu et al. (2020) and Huang et al. (2021). The ablation in Table 10 shows only small differences among initialization methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models." pith.science (2026). https://pith.science/paper/CSAN5QTG

@misc{pith2026250520154,
  author       = {Pith},
  title        = {Pith review of: UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CSAN5QTG}},
  note         = {Machine review of arXiv:2505.20154}
}
read the original abstract

This paper introduces Uniform Orthogonal Reinitialization Adaptation (UORA), a novel parameter-efficient fine-tuning (PEFT) approach for Large Language Models (LLMs). UORA achieves state-of-the-art performance and parameter efficiency by leveraging a low-rank approximation method to reduce the number of trainable parameters. Unlike existing methods such as LoRA and VeRA, UORA employs an interpolation-based reparametrization mechanism that selectively reinitializes rows and columns in frozen projection matrices, guided by the vector magnitude heuristic. This results in substantially fewer trainable parameters compared to LoRA and outperforms VeRA in computation and storage efficiency. Comprehensive experiments across various benchmarks demonstrate UORA's superiority in achieving competitive fine-tuning performance with negligible computational overhead. We demonstrate its performance on GLUE and E2E benchmarks and its effectiveness in instruction-tuning large language models and image classification models. Our contributions establish a new paradigm for scalable and resource-efficient fine-tuning of LLMs.

Figures

Figures reproduced from arXiv: 2505.20154 by the authors.

Figure 1
Figure 1. Overview of LoRA (left) and UORA (right). LoRA trains a pair of projection matrices, namely A and B, with low rank r. The update to the pretrained weights is thus represented as ∆W = A × B. UORA adopts the similar strategy as VeRA; both projection matrices are frozen and randomized. A pair of scaling vectors, ⃗d and ⃗b, is trained to adapt the frozen matrices. The key difference is that UORA applies interpolation re… view at source ↗
Figure 2
Figure 2. Performance vs. number parameters of LoRA [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages

  1. [3]

    InThe Twelfth International Conference on Learning Representations

    LQ-loRA: Low-rank plus quantized matrix de- composition for efficient language model finetuning. InThe Twelfth International Conference on Learning Representations. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. Warp: Word-level adversarial reprogramming.arXiv preprint arXiv:2101.00121. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- K...

    arXiv 2021

  2. [4]

    LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

    Longrecipe: Recipe for efficient long context generalization in large language models.Preprint, arXiv:2409.00509. Wei Huang, Weitao Du, and Richard Yi Da Xu. 2021. On the neural tangent kernel of deep networks with orthogonal initialization. InProceedings of the Thir- tieth International Joint Conference on Artificial In- telligence, IJCAI-21, pages 2577–...

    work page Pith review arXiv 2021

  3. [9]

    InThe Twelfth International Conference on Learning Representations

    A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations. Zexu Sun, Yiju Guo, Yankai Lin, Xu Chen, Qi Qi, Xing Tang, xiuqiang He, and Ji-Rong Wen. 2025. Uncer- tainty and influence aware reward model refinement for reinforcement learning from human feedback. In The Thirteenth Int...

    arXiv 2025

  4. [11]

    Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Ji- aming Ji, Yingying Zhang, Zhijiang Guo, Yaodong Yang, Muhan Zhang, and Debing Zhang

    Advancing parameter efficiency in fine- tuning via representation editing.arXiv preprint arXiv:2402.15179. Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Ji- aming Ji, Yingying Zhang, Zhijiang Guo, Yaodong Yang, Muhan Zhang, and Debing Zhang. 2025. Red- star: Does scaling long-cot data unlock better slow- reason...

    arXiv 2025

  5. [12]

    I Can’t Believe It’s Not Better! - Understanding Deep Learning Through Empirical Falsification

    Functional faithfulness in the wild: Circuit dis- covery with differentiable computation graph prun- ing.Preprint, arXiv:2407.03779. Sheheryar Zaidi, Tudor Berariu, Hyunjik Kim, Jorg Bornschein, Claudia Clopath, Yee Whye Teh, and Razvan Pascanu. 2023. When does re-initialization work? InProceedings on "I Can’t Believe It’s Not Better! - Understanding Deep...

    arXiv 2023

  6. [2014]

    InComputer vision–ECCV 2014: 13th European conference, zurich, Switzer- land, September 6-12, 2014, proceedings, part VI 13, pages 446–461

    Food-101–mining discriminative components with random forests. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzer- land, September 6-12, 2014, proceedings, part VI 13, pages 446–461. Springer. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amand...

    arXiv 2014

  7. [2015]

    Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano

    Parsing algebraic word problems into equa- tions.Transactions of the Association for Computa- tional Linguistics, 3:585–597. Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2023. Vera: Vector-based random matrix adaptation.arXiv preprint arXiv:2310.11454. Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learn- ing multiple layers of features from ti...

    arXiv 2023

  8. [2017]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal

    The e2e dataset: New challenges for end-to- end generation.arXiv preprint arXiv:1706.09254. Arkil Patel, Satwik Bhattamishra, and Navin Goyal

    arXiv

Show all 13 references
  1. [2020]

    Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen, and Benyou Wang

    Exploring versatile generative language model via parameter-efficient transfer learning.arXiv preprint arXiv:2004.03829. Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen, and Benyou Wang. 2024a. Rag-instruct: Boosting llms with diverse retrieval-augmented in- structions.P...

    2004 arXiv

  2. [2021]

    Association for Computational Linguistics

    Are NLP models really able to solve simple math word problems? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics. ...

    2021 arXiv

  3. [2023]

    main character, book, author, year of publication

    Modular transformers: Compressing trans- formers into modularized layers for flexible efficient inference. InFindings of the Association for Compu- tational Linguistics: ACL 2023, pages 10452–10465, Toronto, Canada. Association for Computational Lin- guistics. A Hyperparameter...

    2023

  4. [2024]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1932–1945, Bangkok, Thailand

    LoRAMoE: Alleviating world knowledge for- getting in large language models via MoE-style plu- gin. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1932–1945, Bangkok, Thailand. Association for Computati...

    1932 arXiv

  5. [2025]

    Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xin- song Zhang

    Text-to-cad generation through infusing vi- sual feedback in large language models.Preprint, arXiv:2501.19054. Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xin- song Zhang. 2023. EfficientVLM: Fast and accurate vision-language models via knowledge distillation and modal-adapt...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.