Pith. sign in

REVIEW 5 cited by

LoTR: Low Tensor Rank Weight Adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.01376 v2 pith:OOAUB6RG submitted 2024-02-02 cs.CL cs.AIcs.LG

Daniel Bershatsky , Daria Cherniuk , Talgat Daulbaev , Aleksandr Mikhalev , Ivan Oseledets This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords tensorfine-tuningllmslotrlow-rankadaptationallowsgradient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factorization of gradient update. We introduce LoTR, a novel approach for parameter-efficient fine-tuning of LLMs which represents a gradient update to parameters in a form of tensor decomposition. Low-rank adapter for each layer is constructed as a product of three matrices, and tensor structure arises from sharing left and right multipliers of this product among layers. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models. Moreover, the core tensor does not depend on original weight dimension and can be made arbitrary small, which allows for extremely cheap and fast downstream fine-tuning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chebyshev Manifold Adaptation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ChebyMA uses Chebyshev polynomial surfaces to parameterize weight updates, claiming a better parameter-accuracy trade-off than LoRA, TLoRA, and StelLA on CIFAR and text classification.

  2. KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

    cs.CL 2026-01 conditional novelty 6.0 of 10

    KromHC uses Kronecker products of small doubly stochastic matrices to make Hyper-Connection residual matrices exactly balanced with O(n^2C) parameters, and matches or beats prior variants on small LLM pretraining runs.

  3. Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization

    cs.LG 2025-07 conditional novelty 6.0 of 10

    SubLoRA projects the Hessian of the fine-tuning loss onto a submodular quadratic objective and uses greedy selection to automatically allocate LoRA ranks under a budget.

  4. Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networks

    eess.IV 2025-07 conditional novelty 4.0 of 10

    TenVOO represents 3D convolution weight updates as tensor networks, fine-tuning a brain MRI DDPM with only 0.3% of full trainable parameters while achieving competitive or better structural similarity on ADNI, PPMI, a...

  5. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

Pith tools