Pith. sign in

REVIEW 4 major objections 5 minor 30 references

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Kronecker-factored Fisher matrix, not its diagonal, gives optimal rank-r SVD compression of a layer under maximum-likelihood loss.

desk verdict A sound idea in need of serious cleanup: Kronecker-factored Fisher weighting for LLM compression is a plausible extension of FWSVD, but the empirical comparison is undercut by an unfair baseline and the fast multiplication formulas are dimensionally wrong as written. read the letter →

arxiv 2505.17974 v1 pith:JAO7KWUS submitted 2025-05-23 cs.LG cs.AI

Viktoriia Chekalina , Daniil Moskovskiy , Daria Cherniuk , Maxim Kurkin , Andrey Kuznetsov , Evgeny Frolov This is my paper · ORCID
classification cs.LGcs.AI
keywords Kronecker-factoredFisherinformationlow-rankcompressionlargelanguagemodelsFisher-weightedSVDpost-traininggeneralizedoff-diagonalparametercorrelationsLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large-model compression by SVD normally decides which parameter directions matter from the diagonal of the Fisher information matrix, which treats each weight independently. This paper proposes to factor the full Fisher information into a Kronecker product of two smaller matrices and to use the Cholesky factors of those matrices as reweighting transformations on both sides of the SVD. Its central theorem states that when the loss is maximum-likelihood and the Fisher information really is such a Kronecker product, the resulting generalized SVD gives the optimal rank-r compression. The authors add a scalable algorithm that computes the two factors from gradients, reducing the per-layer cost from quartic to cubic in the weight-matrix dimension, and report consistent gains over diagonal-Fisher and activation-based baselines on GLUE, WikiText-2, PTB, and MMLU, with the largest improvements at aggressive compression rates.

What carries the argument

The load-bearing object is the rank-1 Kronecker product approximation of the observed Fisher information, $I_F \approx A\otimes B$, computed from per-batch gradient matrices $G_i$. Its Cholesky factors $L_A$ and $L_B$ define the transformation $\widehat W_r = L_B^{-T} \tilde W_r L_A^{-1}$, in which the truncated SVD is taken of $\tilde W = L_B^T W^* L_A$; the inverse Cholesky factors act as sensitivity reweighting maps that make an ordinary SVD optimal for the Fisher-weighted objective. To obtain $A$ and $B$ cheaply, the algorithm forms the permuted Fisher matrix $\tilde I_F = \sum_i G_i \otimes G_i$ and computes its leading singular triplet with Lanczos iteration, implementing each left product as $\sum_i \mathrm{vec}(G_i^\top Z G_i)$, so the dominant cost is cubic in the weight-matrix dimension rather than quartic.

What would settle it

Measure the normalized Frobenius residual $\|I_F - A\otimes B\|_F / \|I_F\|_F$ of the Kronecker fit on the actual calibration gradients for every compressed layer. If GFWSVD still shows its reported gains on layers where this residual is large, then the theorem's Kronecker premise is not what drives the improvement; alternatively, run GFWSVD against FWSVD on a synthetic weight matrix with a deliberately non-Kronecker Fisher matrix, since the theorem predicts GFWSVD should win only in the Kronecker case.

Watch

Extended reading notes

Core claim

Under the paper's three conditions - MLE loss, empirical Fisher information of the form $I_F = A\otimes B$, and weights drawn from a matrix-variate normal centered at the optimal weights - the best rank-r compression of a layer weight matrix $W^*$ is $\widehat W_r = L_B^{-T} \tilde W_r L_A^{-1}$, where $\tilde W_r$ is the truncated SVD of $\tilde W = L_B^T W^* L_A$ and $A = L_A L_A^T$, $B = L_B L_B^T$ are Cholesky factorizations. In other words, the Fisher-weighted truncation problem becomes an ordinary SVD after a change of coordinates defined by the Cholesky factors. The paper proves that the diagonal-Fisher method FWSVD is the special case where one factor is diagonal and the other is identity, and it gives a Kronecker-factorization algorithm whose matrix-vector products are evaluated as sums of gradient-matrix multiplications, reducing the per-layer cost from $O(m^2 n^2)$ to $O(m n^2 + m^2 n)$. Empirical results on BERT and LLaMA-2 show that this method matches or beats SVD, FWSVD, ASVD, and SVD-LLM on most settings, with the clearest margins at low ranks.

Load-bearing premise

The load-bearing premise is that each layer's full Fisher information matrix is exactly a single Kronecker product of a row-factor and a column-factor, with no leftover correlations; the paper's own limitation section says this exact factorization may not hold in practice, and if it fails the reweighting is optimal for the wrong objective.

Editorial extensions

If this is right

  • Within the theorem's scope, no other rank-r factorization of a layer can beat the Cholesky-SVD construction, so the method is optimal rather than heuristic for a single linear layer.
  • FWSVD is recovered as the diagonal limit of the same construction, so any empirical gain of GFWSVD over FWSVD is attributable specifically to off-diagonal parameter correlations.
  • The cubic-time factorization brings full-Fisher compression into the regime of modern transformer layers with dimensions on the order of $10^3$.
  • The method needs only gradients from calibration data, so it applies post-training without retraining, and in the LLM experiments it is combined with per-layer importance-based rank allocation.
  • At the most aggressive tested compression (20 percent of parameters removed), GFWSVD is the only method that keeps MMLU accuracy at 0.32 and PTB perplexity at 50.5, while the diagonal-Fisher baseline collapses to 1523 perplexity, showing that off-diagonal sensitivity matters most when capacity is scarce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test would be to replace the single Kronecker product with a sum of several Kronecker products; the optimal solution would then involve a coupled SVD rather than a single truncation.
  • The same Cholesky-reweighted coordinates could plausibly carry over to quantization or pruning, since they rank directions by Fisher sensitivity rather than by Euclidean length.
  • The reported gains may depend on the calibration data; comparing GFWSVD against FWSVD with different calibration sample sizes would isolate the value of the off-diagonal signal.
  • Because the theorem covers a single linear layer, the multi-layer transformer results rest on a heuristic extension; modeling cross-layer dependencies, which the paper names as a limitation, is the most direct route to further gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GFWSVD, a post-training low-rank compression method for transformer weight matrices that replaces the diagonal Fisher approximation of FWSVD with a Kronecker-factored empirical Fisher information matrix. The central theoretical result (Theorem 1) shows that, under an MLE loss, an exact Kronecker-structured Fisher matrix, and matrix-variate normal weights, the optimal rank-r reconstruction is obtained by a whitened SVD of the weight matrix using Cholesky factors of the Kronecker factors. To make the method tractable, the authors derive a fast matrix-vector product for the permuted Fisher matrix and report experiments on BERT (GLUE) and LLaMA-2-7B (WikiText-2, PTB, MMLU), claiming consistent improvements over FWSVD, ASVD, and SVD-LLM. The paper also proves that FWSVD is a special case of GFWSVD when the Fisher approximation is taken to be I_m ⊗ D.

Significance. The conceptual contribution is valuable: connecting Kronecker-factored Fisher information to weighted SVD for compression, and showing that FWSVD is a limiting case, is a clean and publishable idea. The proof of Theorem 1 is straightforward and correct under its explicit idealizations, and the reduction of Kronecker decomposition to fast matrix-matrix products is the right approach for scalability. The empirical evaluation covers both encoder and decoder models and multiple downstream tasks, which is appropriate for the claimed breadth. However, the significance is conditional on fixing the dimensional inconsistencies in the core algorithm and on demonstrating that the empirical gains are not artifacts of the rank-allocation protocol or of the idealized Kronecker assumption.

major comments (4)
  1. [Section 4, Algorithm 1, Eqs. (13) and (15)] The dimensions of the permuted Fisher matrix are inconsistent across the paper. Eq. (13) defines \tilde I_F = (1/|D|) Σ G_i⊗G_i with G_i ∈ R^{n×m}, so \tilde I_F ∈ R^{n^2×m^2}, but Section 4.2 states that the matrix \tilde J is in R^{m^2×n^2}; Algorithm 1 then reshapes the length-n^2 left singular vector b into B ∈ R^{m×m} and the length-m^2 right singular vector a into A ∈ R^{n×n}, which is possible only when n = m. This must be corrected and the derivation of the permuted-Fisher identity made consistent before the scalable algorithm can be reproduced.
  2. [Section 4.1, Eqs. (14)–(15) and Eq. (30)] The Kronecker-product identity stated before Eq. (15) is incorrect. The correct identity is (K⊗L) vec(C) = vec(L C K^T), so (G_i⊗G_i) vec(Z) equals vec(G_i Z G_i^T) with Z ∈ R^{m×m}, not vec(G_i^T Z G_i) with the Z ∈ R^{n×n} introduced in Eq. (14). Correspondingly, the right multiplication formula in Eq. (30) should be Σ vec(G_i^T Z G_i), not Σ vec(G_i Z G_i^T). As written, both equations are dimensionally invalid for rectangular weight matrices and would prevent a reader from implementing the proposed fast algorithm.
  3. [Theorem 1, Eq. (12), and Section 6] Theorem 1's optimality guarantee requires Condition 2, the exact Kronecker structure I_F = A⊗B, but the method in practice solves Eq. (12), a rank-1 Kronecker approximation, and the Limitations section concedes that 'this assumes exact factorization, which may not hold in practice.' The paper provides no experiment measuring the approximation error, e.g., ||I_F − A⊗B||_F/||I_F||_F or a spectral proxy, so the reported gains over diagonal-Fisher baselines have no direct theoretical support under the actual approximation error.
  4. [Section 5.2 and Table 3] The LLaMA comparison is potentially confounded by an asymmetry in rank allocation. For GFWSVD the authors state that they adopted the ASVD per-layer importance scores, but for the FWSVD baseline no per-layer rank-allocation protocol is described; if FWSVD used uniform ranks or another fixed schedule while GFWSVD used a sensitivity-based schedule, the consistent improvements in Table 3 may be due to the allocation scheme rather than to the Kronecker-Fisher weighting. The paper should specify the allocation for all methods or run an ablation with a common allocation protocol.
minor comments (5)
  1. [Abstract] The sentence 'at a 20 compression rate on the MMLU benchmark, our method outperforms FWSVD ... by 5 percent, SVD-LLM by 3 percent, and ASVD by 6 percent compression rate' is grammatically garbled; it should refer to 'a 20% compression rate' and report accuracy differences (percentage points), not 'compression rate' as the outcome.
  2. [Section 5.1, Figure 2] The text says 'Table 2 reports the empirical decomposition times for \tilde I_F', but Table 2 in the manuscript is the GLUE performance table; the reference should point to Figure 2 itself or to a separate table of runtime results.
  3. [Section 5] The code link is a placeholder ('GitHub: link'); the final version should provide the actual repository URL and ideally a version tag for reproducibility.
  4. [Section 4, first paragraph] The sentence 'define G_i ∈ R^{n×m} as a weight gradients L(θ)|_{θ=W} on the i-th batch' has a grammatical error and imprecise notation; it should say 'as the gradients of L(θ) with respect to θ evaluated at θ = W'.
  5. [Section 3.2 and Theorem 1] Condition 3 states W ~ MN(W*, B^{-1}, A^{-1}), but the proof and Eq. (9) use I_F = A⊗B with A = L_A L_A^T and B = L_B L_B^T; the ordering of the Kronecker factors relative to the MVN covariance convention in Section 3.2 is easy to mix up and should be clarified with a short note that the algebra is invariant to the exchange.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is conditional and externally evaluated; the Kronecker factors are estimated from calibration gradients, not fitted to the target metrics.

full rationale

GFWSVD's core derivation is a conditional optimality theorem, not a fitted prediction. Given the explicit assumption IF = A⊗B (Theorem 1, Condition 2), the proof reduces the Fisher-weighted quadratic objective (Eq. 8) to ||L_B^T(W*−W)L_A||_F^2 (Eq. 9), and the optimal rank-r solution by standard SVD of L_B^T W* L_A is exact. The Kronecker factors A and B are estimated from calibration gradients by minimizing ||IF−A⊗B||_F (Eq. 12) and are not tuned to GLUE, MMLU, or perplexity; the evaluation metrics are external to the fitting objective, so no fitted-input-called-prediction pattern is present. The FWSVD-as-special-case claim in Appendix A is a straightforward algebraic reduction of FWSVD's diagonal objective to a diagonal Kronecker factor; it does not smuggle in an unverified premise. There are no load-bearing self-citations and no uniqueness theorem imported from the authors' prior work. The limitations passage (Section 6) concedes that the exact Kronecker factorization 'may not hold in practice'; this is an acknowledged approximation-error risk, not a circular step, since the method's output is not defined in terms of its target metric. The dimensional inconsistencies in Eqs. 14–15 and Eq. 30 (Z size and transposition) are reproducibility or correctness issues outside circularity analysis. Verdict: no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The derivation itself is parameter-free: it uses standard Cholesky and SVD manipulations. The listed free parameters are experimental configuration choices that affect the reported gains. The axioms are idealizations inherited from the FWSVD and KFAC literature; the most fragile is the exact Kronecker structure of the FIM.

free parameters (3)
  • Regularization coefficient α = not reported
    Section 6 states Y ← Y + α diag(Y) when Kronecker factors are singular; α is chosen by hand, not specified, and affects the factor matrices and downstream compression.
  • Per-layer compression ranks for LLaMA = layer-wise ranks not reported in full
    Section 5.2 says ASVD-style per-layer importance scores are used to allocate ranks for GFWSVD; the rank schedule is a chosen configuration that materially affects results and is not disclosed.
  • Calibration sample size from FineWeb = 1024
    Section 5.2. A hand-chosen calibration set size that determines the FIM estimate; not swept or justified.
assumptions (5)
  • domain assumption The task loss function derives from an MLE problem, so the Hessian coincides with the Fisher information.
    Theorem 1 condition 1; Section 3.3. Real LLM losses are not MLE for a single linear layer, so this is an idealization.
  • ad hoc to paper The empirical Fisher information has exact Kronecker product structure IF = A⊗B.
    Theorem 1 condition 2 and Eq. 12 in Section 4. The paper computes a rank-1 Kronecker approximation, which is not exact; the optimality proof depends on exactness.
  • ad hoc to paper Weights are drawn from a Matrix-Variate Normal with covariance B^{-1}⊗A^{-1}.
    Theorem 1 condition 3 and Section 3.2. No evidence that trained LLM weights follow this distribution.
  • domain assumption Layer independence: the full-network FIM is block-diagonal per layer.
    Section 2 states the method adopts the same assumption as related work. Cross-layer correlations are ignored, also acknowledged in Section 6.
  • ad hoc to paper The rank-1 Kronecker approximation captures the important structure of the FIM.
    Sections 4 and 6. The method uses only the leading singular triplet; higher-rank Kronecker series are left to future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models." pith.science (2026). https://pith.science/paper/JAO7KWUS

@misc{pith2026250517974,
  author       = {Pith},
  title        = {Pith review of: Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAO7KWUS}},
  note         = {Machine review of arXiv:2505.17974}
}
read the original abstract

The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks. However, leveraging the full observed Fisher information is too expensive for large models, so most methods rely on simple diagonal approximations. While efficient, this approach ignores parameter correlations, often resulting in reduced performance on downstream tasks. In this work, we mitigate these limitations and propose Generalized Fisher-Weighted SVD (GFWSVD), a post-training LLM compression technique that accounts for both diagonal and off-diagonal elements of the Fisher information matrix, providing a more accurate reflection of parameter importance. To make the method tractable, we introduce a scalable adaptation of the Kronecker-factored approximation algorithm for the observed Fisher information. We demonstrate the effectiveness of our method on LLM compression, showing improvements over existing compression baselines. For example, at a 20 compression rate on the MMLU benchmark, our method outperforms FWSVD, which is based on a diagonal approximation of the Fisher information, by 5 percent, SVD-LLM by 3 percent, and ASVD by 6 percent compression rate.

Figures

Figures reproduced from arXiv: 2505.17974 by the authors.

Figure 1
Figure 1. Generalization of the Weighted SVD frameworks. For standard SVD, the transformation [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Empirical runtime for comput￾ing the Kronecker decomposition of the Fisher matrix for weight matrices of vary￾ing sizes. Although the proposed method exhibits cubic complex￾ity with respect to the dimensions of the linear layer’s weight matrix, its empirical runtime grows more slowly than that of the standard matrix-vector product, which scales quartically. In large language models, where m and n are typically on th… view at source ↗
Figure 3
Figure 3. Macro-averaged GLUE performance of bert-base-uncased model for different com￾pression ranks. 80 85 90 95 Compression Rate (%) 0.25 0.28 0.30 0.33 0.35 0.38 0.40 0.43 0.45 MMLU Avg Accuracy MMLU Full Model ASVD SVD-LLM FWSVD GFWSVD (Ours) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 22 canonical work pages

  1. [1]

    Singular value decomposition (svd) and generalized singular value decomposition

    Herv \'e Abdi. Singular value decomposition (svd) and generalized singular value decomposition. Encyclopedia of measurement and statistics, 907 0 (912): 0 44, 2007

    work page 2007

  2. [2]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Takeru Yanase, Toshihiko Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623--2631. ACM, 2019

    work page 2019

  3. [3]

    A generalized least-square matrix decomposition

    Genevera I Allen, Logan Grosenick, and Jonathan Taylor. A generalized least-square matrix decomposition. Journal of the American Statistical Association, 109 0 (505): 0 145--159, 2014

    work page 2014

  4. [4]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, N...

    2019

  5. [5]

    R. A. Fisher. On the Mathematical Foundations of Theoretical Statistics, pages 11--44. Springer New York, New York, NY, 1992. ISBN 978-1-4612-0919-5. doi:10.1007/978-1-4612-0919-5_2. URL https://doi.org/10.1007/978-1-4612-0919-5_2

    doi:10.1007/978-1-4612-0919-5_2 1992

  6. [6]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=rJl-b3RcF7

    work page 2019

  7. [7]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine L...

    work page 2023

  8. [8]

    Matrix computations

    Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013

    2013

Show all 30 references
  1. [9]

    A kronecker-factored approximate fisher matrix for convolution layers

    Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573--582. PMLR, 2016

    2016

  2. [10]

    Matrix variate distributions

    Arjun K Gupta and Daya K Nagar. Matrix variate distributions. Chapman and Hall/CRC, 2018

    2018

  3. [11]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview...

    2021

  4. [12]

    Language model compression with weighted low-rank factorization, 2022

    Yen - Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization, 2022. URL https://openreview.net/forum?id=uPv9Y3gmAI5

    2022

  5. [13]

    Fedfisher: Leveraging fisher information for one-shot federated learning

    Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedfisher: Leveraging fisher information for one-shot federated learning. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau...

    2024

  6. [14]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, et al. Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences, volume 114, pages 3521--3526, 2017

    2017

  7. [15]

    Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition

    Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Val \'e rie Gar \`e s, Mounir Haddou, and Quang Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition. ESAIM: Proceedings and Surveys, 73: 0 ...

    2023

  8. [16]

    An iteration method for the solution of the eigenvalue problem of linear differential and integral operators

    Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards, 45: 0 255--282, 1950

    1950

  9. [17]

    Dynamic fisher-weighted model merging via B ayesian optimization

    Sanwoo Lee, Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai, and Yunfang Wu. Dynamic fisher-weighted model merging via B ayesian optimization. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of t...

    2025

  10. [18]

    Adasvd: Adaptive singular value decomposition for large language models

    Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Adasvd: Adaptive singular value decomposition for large language models. arXiv e-prints, pages arXiv--2502, 2025

    2025

  11. [19]

    Approximation with kronecker products

    Charles Van Loan and Nikos Pitsianis. Approximation with kronecker products. 1992

    1992

  12. [20]

    Large language model compression via the nested activation-aware decomposition

    Jun Lu, Tianyi Xu, Bill Ding, David Li, and Yu Kang. Large language model compression via the nested activation-aware decomposition. arXiv preprint arXiv:2503.17101, 2025

    2025 arXiv

  13. [21]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Comput. Linguistics, 19 0 (2): 0 313--330, 1993

    1993

  14. [22]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openr...

    2017

  15. [23]

    Raffel, Leandro von Werra, and Thomas Wolf

    Guilherme Penedo, Hynek Kydl \' cek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela...

    2024

  16. [24]

    Trade-offs of diagonal fisher information matrix estimators

    Alexander Soen and Ke Sun. Trade-offs of diagonal fisher information matrix estimators. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Con...

    2024

  17. [25]

    Skfac: Training neural networks with faster kronecker-factored approximate curvature

    Zedong Tang, Fenlong Jiang, Maoguo Gong, Hao Li, Yue Wu, Fan Yu, Zidong Wang, and Min Wang. Skfac: Training neural networks with faster kronecker-factored approximate curvature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13479--...

    2021

  18. [26]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  19. [27]

    Svd-llm v2: Optimizing singular value truncation for large language model compression

    Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-llm v2: Optimizing singular value truncation for large language model compression. arXiv preprint arXiv:2503.12340, 2025 a

    2025 arXiv

  20. [28]

    SVD - LLM : Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD - LLM : Truncation-aware singular value decomposition for large language model compression. In International Conference on Learning Representations (ICLR), 2025 b . URL https://openreview.net/forum?id=LNYIUouhdt

    2025

  21. [29]

    Woodland

    Xiaodong Wu, Wenyi Yu, Chao Zhang, and Philip C. Woodland. An improved empirical fisher approximation for natural gradient descent. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural I...

    2024

  22. [30]

    ASVD: activation-aware singular value decomposition for compressing large language models

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: activation-aware singular value decomposition for compressing large language models. CoRR, abs/2312.05821, 2023. doi:10.48550/ARXIV.2312.05821. URL https://doi.org/10.48550/arXiv.2312.05821

    2023 doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.