Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that initializing tensor adapters from the MPO decomposition of pretrained weights, rather than random noise, yields validation loss close to full fine-tuning and beats random-initialized tensor and low-rank methods with…

desk verdict A clean, plausible extension of PiSSA to tensor adaptation, with a controlled ablation supporting the initialization story, but the headline margins over the strongest baseline are conditional on tuning and protocol. read the letter →

arxiv 2412.20891 v1 pith:GTXGLLNP submitted 2024-12-30 cs.CL cs.LG

Xiaolin Hu , Xiang Cheng , Peiyu Liu , Wei Liu , Jian Luan , Bin Wang , Yong Liu This is my paper · ORCID
classification cs.CLcs.LG
keywords LargeLanguageModelsParameter-EfficientFine-TuningTensorDecompositionMatrixProductOperatorInitializationQuantizationLow-RankAdaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that how you initialize a tensor-based adapter matters as much as its shape. It claims that starting from the Matrix Product Operator (MPO) decomposition of a pretrained weight matrix, rather than from random noise, keeps the fine-tuned model close to the pre-training manifold and yields validation loss curves that track full fine-tuning. On eight commonsense reasoning tasks and two arithmetic datasets, the proposed DoTA is reported to outperform random-initialized tensor methods (LoRETTA, FLoRA) and low-rank baselines (LoRA, DoRA, PiSSA) while using a fraction of their parameters. A quantized variant, QDoTA, applies 4-bit NF4 quantization to the frozen residual and is claimed to match DoTA's commonsense performance while using less memory. If true, the result would make tensor decomposition a practical and parameter-efficient alternative to low-rank adaptation for LLM fine-tuning.

What carries the argument

The load-bearing tool is the Matrix Product Operator (MPO), a decomposition from quantum physics that represents a large matrix as a contracted chain of small fourth-order core tensors $\{T^{(k)}\}$. In DoTA, the pretrained weight $W_0$ is decomposed by iterative SVD (Algorithm 1), the core tensors become the trainable parameters, and the rank threshold $R$ controls their number. The frozen residual $W_{\mathrm{res}} = W_0 - \mathrm{MPO}(W_0)$ preserves the truncated detail and keeps the reconstruction exact before training. The number of trainable parameters is $\rho = \sum_{k=1}^{N} \bar{R}_{k-1} I_k J_k \bar{R}_k$, with $\bar{R}_0 = \bar{R}_N = 1$; for a $1024\times1024$ matrix with $N=5$, $R=8$, and $I_k = J_k = 4$, this is roughly 3.3K parameters. The same construction, with the residual quantized to NF4 and computations in BFloat16, gives QDoTA.

What would settle it

Re-run every baseline under identical training schedules, seed counts, and a shared hyperparameter search budget, and compare with random-initialized tensors of the same shape and parameter count; if DoTA's margin disappears or a random-initialized tensor matches its accuracy, the claimed initialization advantage is not sustained.

Watch

Extended reading notes

Core claim

The central claim is that the information lost by truncating an MPO decomposition can be separated from the information worth adapting. DoTA writes the adapted weight as a frozen residual $W_{\mathrm{res}} = W_0 - \mathrm{MPO}(W_0)$ plus a trainable sequence of core tensors that initialize as $\mathrm{MPO}(W_0)$. Because the tensors begin by reproducing the principal structure of the pretrained matrix, the model starts from the pretrained manifold instead of a random low-dimensional subspace, and the residual compensates for the truncation. The paper reports that this initialization, not the tensor format alone, is what drives performance: randomly initialized tensors of the same shape converge to a suboptimal validation loss, while DoTA's curve closely follows full fine-tuning. DoTA achieves 81.6% average accuracy on eight commonsense tasks with LLaMA2-7B at 0.15% trainable parameters, and 87.1% with LLaMA3-8B at 0.06%, surpassing all listed PEFT baselines.

Load-bearing premise

The reported margins assume that the baseline numbers imported from other papers (Full-FT, LoRA, DoRA in Table 1) were produced under settings equivalent to DoTA's runs, and that the hyperparameters $N=5$, $R=16$ selected by grid search on the same benchmark tasks do not inflate DoTA's advantage.

Editorial extensions

If this is right

  • DoTA with $N=5, R=16$ uses 0.15% (LLaMA2-7B) or 0.06% (LLaMA3-8B) of parameters while beating LoRA, DoRA, PiSSA, FLoRA, and LoRETTA on commonsense reasoning averages.
  • QDoTA, using 4-bit NF4 residual quantization, stays close to full fine-tuning on commonsense tasks while using 0.2% parameters, outperforming QLoRA and QPiSSA.
  • Randomly initialized tensors of the same shape converge to a worse validation loss than MPO-initialized ones, isolating initialization as the cause of DoTA's advantage.
  • Rank 16 is the best default; rank 8 degrades performance, especially on math, while rank 32 gives little or no gain over rank 16.
  • DoTA also beats PiSSA on GSM8K by a wide margin, showing the benefit extends beyond commonsense.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: any tensor or matrix adapter that freezes a residual and initializes from a spectral decomposition (SVD, MPO, Tucker) should inherit the same benefit; the active ingredient may be the initialization, not the tensor format.
  • Inference: the 4-bit quantization result suggests that the residual carries mostly low-precision detail, so even lower-bit formats or lossy compression of the residual could be tried while keeping the core tensors in higher precision.
  • Inference: a natural test the paper does not run: replace MPO with a plain truncated SVD of the same rank and shape; if it matches DoTA, the high-dimensional tensor structure is not the driver, only the principal-component initialization.
  • Inference: the validation-loss alignment with full fine-tuning implies DoTA may also serve as a better starting point for subsequent RLHF or continued pretraining than random-initialized adapters, since it stays closer to the pretrained distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DoTA, a parameter-efficient fine-tuning method that decomposes a pre-trained weight matrix into Matrix Product Operator (MPO) core tensors, initializes the trainable tensor adaptation from this decomposition, and keeps the truncation error in a frozen residual matrix. A quantized variant, QDoTA, is also introduced. The main empirical claim is that this initialization strategy lets DoTA outperform random-initialized tensor adaptation and existing PEFT baselines with fewer trainable parameters on commonsense and arithmetic reasoning tasks, supported by Table 1, a controlled initialization ablation in Figure 2, and rank-sensitivity experiments in Section 4.5.

Significance. If the central empirical claim holds, the contribution is a useful and timely one: it provides a concrete, structurally motivated initialization for tensor-based PEFT and shows a plausible mechanism by which preserving pretrained weight structure helps fine-tuning. The paper deserves credit for the controlled ablation in Section 4.4, which compares DoTA against a same-shape Gaussian-random tensor initialization and thereby directly tests the role of initialization rather than the tensor parameterization alone. The rank analysis in Section 4.5 also gives useful practical guidance. However, the headline comparison is currently weakened by hyperparameter selection on the evaluation benchmarks, by several baseline numbers imported from other papers, and by the absence of variance or significance measures for the small reported margins. The central claim is defensible but needs stronger experimental control before it can be accepted as stated.

major comments (4)
  1. [Section 4, first paragraph; Table 1] The headline comparison in Table 1 is not yet controlled for hyperparameter selection. Section 4 states that DoTA's N and R were selected by grid search over N in {3,5,7,9} and R in {16,32,64,128} on the same benchmark tasks used for evaluation, while baseline hyperparameters are fixed values from their original papers. Because the reported margin over the strongest baseline PiSSA is small (0.6 percentage points on LLaMA3-8B and 1.5 on LLaMA2-7B), the possibility that grid-search selection, rather than MPO initialization, produces the margin is not excluded. Please select hyperparameters on a disjoint validation split, evaluate DoTA at the default baseline ranks, or report the performance variation across the searched grid.
  2. [Table 1 footnote; Appendix B] Several baseline numbers are imported without being re-run under the protocol described in the paper. The Table 1 footnote says Full-FT, LoRA, DoRA, and DoRA† results are taken from prior papers, and Appendix B gives method-specific settings for the re-run methods but does not establish that the imported numbers were produced under the same training schedule, adapter layers, data split, one-epoch setting, and bf16 precision used for DoTA. Relative rankings can change with training length and learning-rate schedule, so the central claim of outperforming existing methods requires either re-running these baselines under the common protocol or documenting that the imported results were produced under equivalent conditions.
  3. [Table 1; Section 4.4, Figure 2] The paper does not report variance or significance for the task-accuracy comparison, and the controlled initialization ablation stops at evaluation loss. Table 1 reports only point averages over three random seeds, so a 0.6-1.5 percentage-point margin over PiSSA is not distinguished from seed noise. Figure 2 shows the mean and standard deviation of evaluation loss for DoTA, DoTA-Random, and Full-FT, but it does not report final downstream task accuracy for the random-initialization control. Please add per-seed accuracy, standard deviations, and ideally a significance test, and extend the Figure 2 comparison to final task accuracy so the initialization gap is directly tied to the claimed accuracy benefit.
  4. [Section 5 (Related Work); Section 4] QuanTA, the closest MPO-based tensor adaptation method to DoTA, is cited in Related Work but never appears in the experiments. Without a QuanTA baseline evaluated under the same protocol, the claim that DoTA improves over tensor adaptation methods is incomplete for the most directly comparable prior method. Please add QuanTA to the commonsense and mathematical reasoning comparisons or explicitly explain why it cannot be fairly included.
minor comments (4)
  1. [Abstract; Figure 2 caption; Section 4.4] The abstract and Section 4.4 refer to validation loss, while the Figure 2 caption and axis label say evaluation loss; please standardize the terminology.
  2. [Section 3.2, Equation (3)] The notation MPO(W0) is overloaded: it denotes the exact decomposition of W0 in the definition of Wres, but after rank truncation the contracted core tensors are trainable and no longer represent the exact MPO of W0. Using a distinct symbol for the truncated reconstruction, such as \tilde{W}, would avoid ambiguity.
  3. [Section 4.2, Figure 4] Figure 4 shows only PiSSA, LoRA, and DoTA, while the text says FLoRA and LoRETTA were also tested on mathematical reasoning but their results are not shown; providing the numerical values for all tested baselines would make the comparison complete and reproducible.
  4. [Appendix B] The experimental settings list one epoch, batch size 16, cosine schedule, warmup ratio 0.03, and no dropout; please state explicitly whether these settings also apply to the imported Full-FT, LoRA, and DoRA numbers or only to the results re-run by the authors.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DoTA's MPO initialization claim is tested against a random-initialization control and does not reduce to its own inputs.

full rationale

DoTA's central claim is that initializing tensor adapters with the MPO decomposition of pre-trained weights improves fine-tuning over random initialization. This claim is empirical: the paper compares against a random-initialized tensor of the same shape (Section 4.4, Figure 2) and against prior PEFT baselines (Table 1). The method's definition, W' = Wres + MPO(W0) with Wres = W0 - MPO(W0), is an exact initialization identity that ensures the adapted model starts at the pre-trained weights; this is a design choice, not a conclusion forced by the definition. The MPO decomposition itself is a standard mathematical procedure cited to external sources (Pirvu et al., Gao et al.), not to the authors' own prior work. There is no fitted parameter that is subsequently renamed as a prediction: hyperparameters N and R are selected by grid search on the evaluation tasks, which is a tuning and replicability concern, not circular reasoning. Imported baseline numbers from other papers are also a protocol-matching concern, but they do not make the derivation circular. No load-bearing step reduces, by the paper's own equations or by self-citation, to its own inputs. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The free parameters are standard PEFT hyperparameters (number of cores, rank threshold, tensor shapes, learning rates). The axioms include standard linear algebra about MPO decomposition and domain assumptions about the usefulness of decomposition-based initialization, which are empirically motivated rather than proven.

free parameters (4)
  • N (number of core tensors) = 5 (searched over {3,5,7,9})
    Controls the tensor decomposition depth and the parameter count; selected by performance on the benchmark tasks.
  • R (rank threshold) = 16 for commonsense, 32 for math and quantization
    Truncation rank for the MPO decomposition; main capacity knob for DoTA; tuned on the same benchmarks.
  • Tensor shape factors {Ik, Jk} = Table 2 (e.g., 4096 -> [4,4,8,8,4])
    Hand-chosen factorizations of each hidden dimension; different factorizations would change parameter count and performance.
  • Learning rates per method = 1e-4 to 3e-4 depending on method
    Method-specific learning rates are chosen independently; affects convergence and final accuracy.
assumptions (4)
  • standard math MPO decomposition with full ranks exactly reconstructs the original matrix.
    Algorithm 1 relies on the standard SVD-based MPO construction; this is a well-known property.
  • domain assumption The pretrained weight matrix contains latent structure whose decomposition provides a better initialization than random initialization.
    Central premise of the method, supported only by empirical ablation (Figure 2), not derived from theory.
  • domain assumption Freezing the residual Wres = W0 - MPO(W0) is safe because the trainable cores can still express the needed update.
    Borrowed from PiSSA; no theoretical guarantee is given in the paper.
  • ad hoc to paper Truncated MPO ranks preserve the components relevant to downstream task adaptation.
    The rank threshold R is a heuristic; the paper only provides empirical evidence on two model families.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models." pith.science (2026). https://pith.science/paper/GTXGLLNP

@misc{pith2026241220891,
  author       = {Pith},
  title        = {Pith review of: DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTXGLLNP}},
  note         = {Machine review of arXiv:2412.20891}
}
read the original abstract

Low-rank adaptation (LoRA) reduces the computational and memory demands of fine-tuning large language models (LLMs) by approximating updates with low-rank matrices. However, low-rank approximation in two-dimensional space fails to capture high-dimensional structures within the target matrix. Recently, tensor decomposition methods have been explored for fine-tuning LLMs, leveraging their ability to extract structured information. Yet, these approaches primarily rely on random initialization, and the impact of initialization on tensor adaptation remains underexplored. In this paper, we reveal that random initialization significantly diverges from the validation loss achieved by full fine-tuning. To address this, we propose Weight-Decomposed Tensor Adaptation (DoTA), which leverages the Matrix Product Operator (MPO) decomposition of pre-trained weights for effective initialization in fine-tuning LLMs. Additionally, we introduce QDoTA, a quantized version of DoTA designed for 4-bit quantization. Experiments on commonsense and arithmetic reasoning tasks show that DoTA outperforms random initialization methods with fewer parameters. QDoTA further reduces memory consumption and achieves comparable performance to DoTA on commonsense reasoning tasks. We will release our code to support future research.

Figures

Figures reproduced from arXiv: 2412.20891 by the authors.

Figure 1
Figure 1. Comparison of the number of train￾able parameters and performance across different methods on commonsense reason￾ing tasks using the LLaMA3-8B model. 2000 4000 6000 8000 10000 Train Steps 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 Evaluation Loss DoTA-Random DoTA Full FT [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The architecture of the proposed method. DoTA decomposes the pre-trained [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Performance of different methods on mathematical reasoning tasks using the LLaMA2-7B model We fine-tuned LLaMA2-7B on the MetaMathQA￾395K [29] dataset to enhance its mathematical capabilities and evaluated its performance on the MATH [29] and GSM8K [4] datasets. We also tested FLoRA and LoRETTA on these two datasets, but since their performance was inferior to PiSSA, their results are not shown in the figures. All r… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Comparison of the quantized versions of various methods on eight commonsense [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Accuracy across different ranks for various tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...

Reference graph

Works this paper leans on

30 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2402.01376 (2024)

    Bershatsky, D., Cherniuk, D., Daulbaev, T., Oseledets, I.: Lotr: Low tensor rank weight adaptation. arXiv preprint arXiv:2402.01376 (2024)

    arXiv 2024

  2. [2]

    Bommasani, R., Hudson, D.A., Adeli, E., Altman, e.a.: On the Opportunities and Risks of Foundation Models (Jul 2022), arXiv:2108.07258 [cs]

    arXiv 2022

  3. [3]

    arXiv preprint arXiv:2406.00132 (2024)

    Chen, Z., Dangovski, R., Loh, C., Dugan, O., Luo, D., Solja ˇci´c, M.: Quanta: Efficient high-rank fine-tuning of llms with quantum-informed tensor adaptation. arXiv preprint arXiv:2406.00132 (2024)

    arXiv 2024

  4. [4]

    arXiv preprint arXiv:2110.14168 (2021)

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

    arXiv 2021

  5. [5]

    SIAM Journal on Matrix Analysis and Applications 30(3), 1084–1127 (2008)

    De Silva, V ., Lim, L.H.: Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications 30(3), 1084–1127 (2008)

    work page 2008

  6. [6]

    Advances in Neural Information Processing Systems 36 (2024)

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  7. [7]

    In: Burstein, J., Doran, C., Solorio, T

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional trans- formers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN,...

    work page 2019

  8. [8]

    arXiv preprint arXiv:2407.21783 (2024) 12 X

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) 12 X. Hu et al

    arXiv 2024

Show all 30 references
  1. [9]

    Physical Review Research 2(2), 023300 (2020)

    Gao, Z.F., Cheng, S., He, R.Q., Xie, Z., Zhao, H.H., Lu, Z.Y ., Xiang, T.: Compressing deep neural networks by matrix product operators. Physical Review Research 2(2), 023300 (2020)

    2020

  2. [10]

    In: Arai, K

    Gao, Z.F., Liu, P., Zhao, W.X., Xie, Z.Y ., Wen, J.R., Lu, Z.Y .: Compression image dataset based on multiple matrix product states. In: Arai, K. (ed.) Advances in information and communication. pp. 621–638. Springer Nature Switzerland, Cham (2024)

    2024

  3. [11]

    Hayou, S., Ghosh, N., Yu, B.: LoRA+: Efficient Low Rank Adaptation of Large Models (Jul 2024), arXiv:2402.12354 [cs, stat]

    2024 arXiv

  4. [12]

    arXiv preprint arXiv:2110.04366 (2021)

    He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 (2021)

    2021 arXiv

  5. [13]

    arXiv preprint arXiv:2106.09685 (2021)

    Hu, E.J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  6. [14]

    arXiv preprint arXiv:2304.01933 (2023)

    Hu, Z., Wang, L., Lan, Y ., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S., Lee, R.K.W.: Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 (2023)

    2023 arXiv

  7. [15]

    Huang, M., Shen, A., Li, K., Peng, H., Li, B., Yu, H.: EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models (Jul 2024), arXiv:2407.21325 [cs]

    2024 arXiv

  8. [16]

    arXiv preprint arXiv:2312.03732 (2023)

    Kalajdzievski, D.: A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732 (2023)

    2023 arXiv

  9. [17]

    arXiv preprint arXiv:1804.08838 (2018)

    Li, C., Farkhoor, H., Liu, R., Yosinski, J.: Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838 (2018)

    2018 arXiv

  10. [18]

    arXiv preprint arXiv:2402.09353 (2024)

    Liu, S.Y ., Wang, C.Y ., Yin, H., Molchanov, P., Wang, Y .C.F., Cheng, K.T., Chen, M.H.: Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 (2024)

    2024 arXiv

  11. [19]

    arXiv preprint arXiv:2404.02948 (2024)

    Meng, F., Wang, Z., Zhang, M.: Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948 (2024)

    2024 arXiv

  12. [20]

    New Journal of Physics 12(2), 025012 (2010)

    Pirvu, B., Murg, V ., Cirac, J.I., Verstraete, F.: Matrix product operator representations. New Journal of Physics 12(2), 025012 (2010)

    2010

  13. [21]

    Qiu, S., Potapczynski, A., Finzi, M., Goldblum, M., Wilson, A.G.: Compute Better Spent: Replacing Dense Layers with Structured Matrices (Jun 2024), arXiv:2406.06248 [cs]

    2024 arXiv

  14. [22]

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners

  15. [23]

    Si, C., Wang, X., Yang, X., Xu, Z., Li, Q., Dai, J., Qiao, Y ., Yang, X., Shen, W.: FLoRA: Low-Rank Core Space for N-dimension (May 2024), arXiv:2405.14739 [cs]

    2024 arXiv

  16. [24]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  17. [25]

    Wang, H., Xiao, Z., Li, Y ., Wang, S., Chen, G., Chen, Y .: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (Jun 2024), arXiv:2406.09044 [cs]

    2024 arXiv

  18. [26]

    Wang, S., Yu, L., Li, J.: LoRA-GA: Low-Rank Adaptation with Gradient Approximation (Jul 2024), arXiv:2407.05000 [cs]

    2024 arXiv

  19. [27]

    Yang, Y ., Zhou, J., Wong, N., Zhang, Z.: LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models (Feb 2024), arXiv:2402.11417 [cs]

    2024 arXiv

  20. [28]

    Yin, C., Acun, B., Liu, X., Wu, C.J.: TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (Jan 2021), arXiv:2101.11714 [cs]

    2021 arXiv

  21. [29]

    arXiv preprint arXiv:2309.12284 (2023)

    Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y ., Kwok, J.T., Li, Z., Weller, A., Liu, W.: Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 (2023)

    2023 arXiv

  22. [30]

    Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y ., Chen, W., Zhao, T.: AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (Mar 2023)

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.