Pith. sign in

REVIEW 3 major objections 5 minor 36 references

ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured-matrix compression of LLMs works without fine-tuning once each layer's weights are first rotated by an orthogonal matrix chosen to make them more compressible; in most tests this beats fine-tuning-free baselines at 14-36%…

desk verdict Good method at 14-25% compression, but the abstract's 14-36% claim is contradicted by the paper's own tables. read the letter →

arxiv 2506.02818 v1 pith:GJPLQIFE submitted 2025-06-03 cs.CL cs.LG

Ekaterina Grishina , Mikhail Gorbunov , Maxim Rakhuba This is my paper
classification cs.CLcs.LG
keywords LLMcompressionstructuredmatricesKroneckerproductGSorthogonalProcrustesrotationalinvariancetraining-freezero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProcrustesGPT argues that the reason structured-matrix compression of pretrained large language models fails without fine-tuning is not the structured class itself but the coordinate system the weights are written in. Because transformer output is exactly invariant under a family of orthogonal transformations that rotate weights and the matching skip connections, the paper recovers degrees of freedom that make weights dramatically more compressible. For each layer it searches for the orthogonal matrix that best complements a chosen structured class, sum of Kronecker products or GS matrices, by solving an alternating Procrustes and projection optimization on calibration activations. The paper reports that this fine-tuning-free recipe improves WikiText2 perplexity and zero-shot accuracy over SliceGPT at comparable compression, and in most of the tested range (14-36% parameters removed) beats SVD-LLM, DISP-LLM, and SLEB as well, with degradation appearing at the 36% end.

What carries the argument

The load-bearing object is computational invariance: for an orthogonal matrix $Q$ with $Q^T Q = I$, the identity $$(X_{out}W_{out}+X_{skip})/\|X_{out}W_{out}+X_{skip}\|_F \cdot W_{in} = (X_{out}W_{out}Q+X_{skip}Q)/\|X_{out}W_{out}Q+X_{skip}Q\|_F \cdot (Q^T W_{in})$$ with $Q$ applied to the skip connection means the outputs are unchanged. The machinery is then the per-layer objective (3), minimizing $\|X_{out}(W_{out}Q-\widehat W_{out})\|_F^2 + \lambda_{in}\|X_{in}(W_{in}-Q\widehat W_{in})\|_F^2$ over orthogonal $Q$ and structured $\widehat W$, solved by alternating a Procrustes step (closed-form SVD in the Frobenius norm; Cayley-parametrized conjugate gradients in the weighted norm) with a structured projection step (SVD for sums of Kronecker products; block least squares for GS matrices, i.e. block-low-rank matrices up to row and column permutations).

What would settle it

Take a 7B model not in the paper's test set, compress it at 25% with ProcrustesGPT and with SliceGPT, and evaluate the same five zero-shot tasks; if SliceGPT wins on a majority of tasks, the headline claim fails. A second check is to measure, layer by layer, the activation drift between the original and compressed network on out-of-domain text, since growth of drift with depth would show the fixed-calibration-input assumption is violated.

Watch

Extended reading notes

Core claim

The paper's central claim is that a pretrained LLM can be compressed by 14-36% without any fine-tuning by first rotating each layer's weight matrices with an orthogonal transformation that makes them more compressible, then projecting onto a structured matrix class such as a sum of Kronecker products or a GS matrix. The rotation is not arbitrary: it is the solution of a per-layer optimization that minimizes the weighted output difference on calibration activations, and it exploits the exact invariance of transformer outputs under such rotations. The paper reports that this rotate-then-project procedure beats SliceGPT in generation perplexity and zero-shot accuracy at comparable compression rates on OPT and Llama2 models, and that the method also handles embedding and head matrices with a diagonal token-frequency weighting.

Load-bearing premise

The whole scheme assumes the calibration inputs seen by each layer stay the same after compression, because each layer's rotation and projection are solved independently and the orthogonal matrices in skip connections are deliberately left uncompressed; if approximation errors cascade from layer to layer, the per-layer objectives no longer describe the actual compressed network.

Editorial extensions

If this is right

  • Any structured matrix class with an efficient projection operator, not only Kronecker and GS families, can be plugged into the same rotation-then-project scheme.
  • Because per-layer problems decouple, the rotations can be computed in parallel across layers, which is how the method scales to 13B-parameter models on a single GPU.
  • Compressing the orthogonal matrices $Q_{\ell-1}^T Q_\ell$ that appear in skip connections is an explicitly identified avenue for further parameter reduction beyond the reported ratios.
  • The paper's own tables show accuracy relative to baselines deteriorating at the high end of the range, so the practical envelope of the claim is the lower to middle compression rates.
  • Storing each rotation through a skew-symmetric Cayley parametrization reduces its parameter cost from $d^2$ to $d(d-1)/2$, which partly offsets the extra weights introduced by rotating the network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is that the rotate-then-project recipe should transfer to other structured classes, such as tensor trains or block-low-rank-plus-sparse representations, provided a weighted projection oracle exists for that class.
  • Because the skip-connection rotations are deliberately left uncompressed, the reported compression percentages are upper bounds on actual weight savings; a stricter comparison would count the Cayley-parametrized rotations against the model budget.
  • The evaluation uses Wikipedia-style text and five commonsense and reasoning tasks, so out-of-domain robustness remains open; the fixed-calibration-input assumption makes distribution shift the most plausible failure mode.
  • Structured compressed weights may not align with the low-rank nature of LoRA-style adapters, so compressing in this way likely calls for new parameter-efficient fine-tuning methods built around the structured representation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ProcrustesGPT is a training-free LLM compression framework that first finds orthogonal transformations of layer weights to make them more compressible within a structured matrix class, then projects the rotated weights onto that class. The method is formulated as an alternating optimization between an orthogonal Procrustes step and a weighted structured projection step, applied to sums of Kronecker products and GS matrices. The authors evaluate on OPT and Llama2 models using WikiText2 perplexity and five zero-shot tasks, reporting that the method outperforms SliceGPT at roughly 15–20% compression and comparing favorably with SVD-LLM, DISP-LLM, and SLEB at 14–36% compression. The central claim in the abstract is that the approach yields more accurate results than alternative fine-tuning-free methods in most scenarios in the 14–36% range.

Significance. If the method's claims hold for the lower end of the reported compression range, this is a useful contribution: it provides a principled, parallelizable, training-free scheme that applies to multiple structured representations, and it explicitly connects the optimization formulation to SliceGPT via Proposition 4.3. The public code release and the use of machine-checkable linear-algebra reductions (e.g., SVD-based Kronecker projection, Cayley parametrization) are strengths. However, the abstract's 36% claim is contradicted by the paper's own tables, and the treatment of orthogonal-matrix storage in the compression ratio is not transparent. Both issues are load-bearing for the headline comparison, so the manuscript needs revision rather than acceptance in its current form.

major comments (3)
  1. [Section 5.4 and Tables 1–4] The abstract claims that 'in most scenarios, our approach yields more accurate results than alternative fine-tuning-free methods at comparable compression rates (in the range from 14% to 36%)'. This is not supported by the paper's own evidence. Section 6.3 concedes that 'ProcrustesGPT outperforms other baselines at lower compression rates, but its performance starts deteriorating at 36% compression of parameters.' Concretely, in Table 3 at ~36% compression, ProcrustesGPT (Kron) gives WikiText2 perplexity 19.55 on Llama2-7b versus 10.92 for DISP-LLM and 14.39 for SVD-LLM; on Llama2-13b it gives 16.88 versus 10.05 and 10.08. The GS variant also loses to DISP-LLM in both perplexity rows. In Table 4 at 36%, the Kron variant's zero-shot average (51.20) is below DISP-LLM (54.35) and SLEB (55.03). The data support the claim only for roughly 14–25% compression; please narrow the stated range or qualify the 36% case explicitly in the abstract and conclusion.
  2. [Section 3.3 and Equation (3)] The reported compression ratios appear to omit the parameter cost of the orthogonal matrices Q_l and Q_{l-1}^T Q_l that appear in skip connections. The text in Section 5.4 states that these 'should also be stored as the weights' and that the Cayley parametrization stores only the upper-triangular part, i.e., d(d-1)/2 parameters per matrix. For Llama2-7b, d=4096 and there are roughly two rotations per transformer layer, which amounts to hundreds of millions of additional parameters if Q_l is materialized at inference. If these parameters are not included in the '% of parameters compressed' columns, then the comparison to SVD-LLM, DISP-LLM, and SLEB at 'comparable compression rates' is not valid; if they are included, the paper should state this explicitly and show the calculation. Without this clarification, the central empirical comparison cannot be fully evaluated.
  3. [Section 3.3 and Equation (3)] The per-layer optimization in (3) uses calibration activations X_in^l and X_out^l from the original, uncompressed network, and layer independence is obtained by leaving Q_{l-1}^T Q_l uncompressed. However, once earlier layers are approximately compressed, the actual inputs to later layers in the compressed network deviate from these calibration activations, so the weighted objectives no longer describe the compressed model's behavior. The paper does not analyze this error propagation or test robustness to distribution shift away from the calibration set. This is a genuine limitation that is likely relevant at the high end of the claimed compression range (36%), where the method degrades sharply. Please add a discussion of this effect, or an experiment measuring sensitivity to calibration data (e.g., a different calibration set or task), and adjust the claims accordingly.
minor comments (5)
  1. [Abstract and Section 3.2] In the abstract and in Section 3.2, the notation 'cW' is used for compressed weights; it is never formally defined. Please define it at first use.
  2. [Section 2] There is a typo: 'prunning' should be 'pruning'.
  3. [Section 6.1] The caption of Table 2 and Table 4 uses the abbreviation 'HellaS.'; please use the full dataset name 'HellaSwag' or define the abbreviation in the caption.
  4. [Algorithm 3] In Algorithm 3, the subscripts in the projection steps appear swapped: cWout is written as the argmin over W in Sin, and cWin as the argmin over W in Sout. Based on the definitions in Section 3.2, these should be Sout and Sin, respectively. Please correct the algorithm or clarify the notation.
  5. [Appendix C] In the proof of Proposition 4.3, the statement 'Up to a normalization layer, this corresponds to ...' is vague. If the equivalence to SliceGPT is asserted, please specify exactly how the normalization is handled and whether the same claim holds for the full SliceGPT scheme without additional assumptions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the compression pipeline is an independent empirical optimization benchmark; the abstract's 14–36% range is internally overstated at 36%, a correctness issue rather than a circular one.

full rationale

Walking the derivation chain, I find no step that reduces to its own input. The central identity (Eq. 2) is a mathematical invariance of Frobenius-normalized layers, originally observed by Ashkboos et al. (2024), and the layer-wise objective (3) is a weighted least-squares Procrustes/projection problem whose optimal Q and compressed weights are computed numerically, not assumed. The structured-class projections (Kronecker SVD and GS-matrices) are externally specified routines; the only self-citation (Gorbunov et al., 2024, with overlapping authorship) supplies the GS-matrix class and its projection algorithm, but it is not the target result and the Kronecker results stand independently, so the self-citation is not load-bearing. The reported numbers are empirical benchmarks against external methods (SliceGPT, SVD-LLM, DISP-LLM, SLEB) on held-out WikiText2 test and five zero-shot tasks, so no fitted quantity is renamed as a prediction. I therefore find no circularity. Two non-circular issues should be noted: Section 6.3 explicitly concedes that 'ProcrustesGPT outperforms other baselines at lower compression rates, but its performance starts deteriorating at 36% compression of parameters,' which is internally inconsistent with the abstract's '14% to 36%' range; Table 3 and Table 4 confirm that at roughly 36% the method trails DISP-LLM and SVD-LLM on WikiText2 perplexity and zero-shot averages. This is a correctness and overclaiming risk, not a circularity, and the strongest honest claim is supported only in the lower part of the range, approximately 14–25% compression.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No invented physical or architectural entities are introduced. The free parameters are compression hyperparameters and empirically selected weighting rules: Kronecker rank and split size, lambda_in, sqrt(D)+1 for embedding and head, and GS block counts. The core unexamined premise is layer-wise independence with fixed calibration activations, which avoids error-propagation modeling.

free parameters (4)
  • Kronecker rank r and split size q = r=3,q=4 (25%); r=8,q=5 (37.5%); r=2,q=1 (50%)
    Chosen by hand to hit target compression ratios; affects approximation quality and is not derived from the model.
  • balancing coefficient lambda_in = ||X_out W_out||_F^2 / ||X_in W_in||_F^2, or 1
    Authors state this ratio improves results versus lambda_in=1 (Section 3.3). It depends on calibration activations and is a hand-set weighting choice.
  • embedding and head weighting matrix = sqrt(D)+1 (log(D)+1 also tested)
    Empirically selected in Section 5.4: 'we found that sqrt(D)+1 gave the best results'; tuned on the same benchmarks used for reporting.
  • GS block counts (k_L, k_R) = square 4,2; rectangular 4,8; embedding/head 1,4
    Hand-chosen to match target compression ratios, as described in Appendix D.
assumptions (5)
  • standard math Rotational invariance of transformer blocks: applying orthogonal Q to output weights and Q^T to input weights leaves outputs identical, including RMSNorm scaling (Eq. 2).
    Uses orthogonality and Frobenius-norm invariance; exact for the idealized block structure. LayerNorm-to-RMSNorm conversion is inherited from Ashkboos et al. (2024).
  • domain assumption Each transformer layer can be treated as independent during compression; calibration inputs X_in^l and X_out^l are fixed from the original model and not updated after earlier layers are compressed.
    Section 3.3 states the per-layer problem becomes independent by leaving skip rotations uncompressed; error propagation through the network is ignored.
  • domain assumption Calibration set of 128 WikiText2 sequences of length 2048 is representative of the data distribution for optimizing rotations.
    Used in Sections 6.1 and 6.2; no analysis of sensitivity to calibration size or domain shift is provided.
  • domain assumption Best approximation in the GS matrix class can be computed by SVD-based projection from Gorbunov et al. (2024), and weighted variants converge via alternating least squares.
    Assumed from prior work; the appendix gives iterative algorithms but no convergence guarantee.
  • domain assumption LayerNorm can be converted to RMSNorm without changing the model (Ashkboos et al., 2024).
    Invoked in Section 3.1; central to applying a single orthogonal Q per block.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations." pith.science (2026). https://pith.science/paper/GJPLQIFE

@misc{pith2026250602818,
  author       = {Pith},
  title        = {Pith review of: ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJPLQIFE}},
  note         = {Machine review of arXiv:2506.02818}
}
read the original abstract

Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT

Figures

Figures reproduced from arXiv: 2506.02818 by the authors.

Figure 1
Figure 1. Illustration of compressibility of different layers of OPT-125m with and without applying orthogonal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the process of compression of a single transformer layer. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 17 canonical work pages

  1. [1]

    V Abronin, A Naumov, D Mazur, D Bystrov, K Tsarova, Ar Melnikov, I Oseledets, Sergey Dolgov, R Brasher, and Michael Perelshtein. 2024. Tqcompressor: improving tensor decomposition methods in neural networks via permutations. arXiv preprint arXiv:2401.16367

    work page Pith review arXiv 2024

  2. [2]

    Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. 2019. Online embedding compression for text classification using low rank matrix factorization. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 6196--6203

    work page 2019

  3. [3]

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024

    arXiv 2024

  4. [4]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439

    2020

  5. [5]

    Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. 2018. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. Advances in Neural Information Processing Systems, 31

    work page 2018

  6. [6]

    Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. 2021. Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems, 34:29321--29334

    work page 2021

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  8. [8]

    Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher R \'e . 2022. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, pages 4690--4721. PMLR

    work page 2022

Show all 36 references
  1. [9]

    Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. 2021. Kronecker decomposition for gpt compression. arXiv preprint arXiv:2110.08152

    2021 arXiv

  2. [10]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  3. [11]

    Shangqian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2024 b . Disp-llm: Dimension-independent structural pruning for large language models. Advances in Neural Information Processing Systems, 37:72219--72244

    2024

  4. [12]

    Gene H Golub and Charles F Van Loan. 2013. Matrix computations. JHU press

    2013

  5. [13]

    Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva, Aibek Alanov, Alexey Naumov, and Maxim Rakhuba. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/7f0f24deb34c21ee590d8cece365710b-Paper-Conference.pdf Group and shuffle: Efficient structured orthogonal parametriza...

    2024

  6. [14]

    Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan Oseledets. 2020. Tensorized embedding layers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4847--4860

    2020

  7. [15]

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112

    2022 arXiv

  8. [16]

    Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Kehai Chen, and Min Zhang. 2024. Adaptive feature-based low-rank compression of large language models via bayesian optimization. In Findings of the Association for Computational Linguistics: EMNLP ...

    2024

  9. [17]

    PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. 2021. HPC resources of the higher school of economics. In Journal of Physics: Conference Series, volume 1740, page 012050

    2021

  10. [18]

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 20336--20350. PMLR

    2023

  11. [19]

    Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2024. Modegpt: Modular decomposition for large language model compression. arXiv preprint arXiv:2408.09632

    2024 arXiv

  12. [20]

    Vasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md Akmal Haidar, and Mehdi Rezagholizadeh. 2020. Improving word embedding factorization for compression using distilled nonlinear neural decomposition. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages...

    2020

  13. [21]

    Robert W Lissitz, Peter H Sch \"o nemann, and James C Lingoes. 1976. A solution to the weighted procrustes problem in which the transformation is in agreement with the loss function. Psychometrika, 41:547--550

    1976

  14. [22]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  15. [23]

    Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J Goldsmith, and Mert Pilanci. 2024. Compressing large language models using low rank and low precision decomposition. arXiv preprint arXiv:2405.18886

    2024 arXiv

  16. [24]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  17. [25]

    Peter H Sch \"o nemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1--10

    1966

  18. [26]

    Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. 2023. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558

    2023 arXiv

  19. [27]

    Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025

    2024 arXiv

  20. [28]

    Marzieh S Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. 2021. Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation. arXiv preprint arXiv:2109.06243

    2021 arXiv

  21. [29]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  22. [30]

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2024. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378

    2024 arXiv

  23. [31]

    Thomas Wolf. 2020. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2020 arXiv

  24. [32]

    Mingxue Xu, Yao Lei Xu, and Danilo P Mandic. 2023. Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition. arXiv preprint arXiv:2307.00526

    2023 arXiv

  25. [33]

    Hao Yu and Jianxin Wu. 2023. Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11007--11015

    2023

  26. [34]

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821

    2023 arXiv

  27. [35]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  28. [36]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.