Pith. sign in

REVIEW 2 major objections 5 minor 51 references

LatentLLM: Attention-Aware Joint Tensor Compression

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A pretrained transformer can be compressed into a latent-attention model in one shot with little accuracy loss.

desk verdict A clean mathematical core with a useful junction-matrix trick, but the headline claim for joint attention-aware compression is not yet isolated from the block-identity effect. read the letter →

arxiv 2505.18413 v1 pith:W77LJ2OT submitted 2025-05-23 cs.LG cs.AIcs.CLcs.CV

Toshiaki Koike-Akino , Xiangyu Chen , Jing Liu , Ye Wang , Pu (Perry) Wang , Matthew Brand This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CV
keywords modelcompressionlow-ranktensordecompositionmulti-headlatentattentionattention-awareSVDroot-covariancepre-conditioningzero-shotLLMvision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a pretrained transformer's paired projections—query with key, value with output, and MLP up with down—can be jointly factorized into shared low-rank latent projections, converting standard multi-head attention into a latent-attention model in a single training-free pass. The goal is to cut parameter count, KV-cache memory, and inference FLOPs while keeping output quality close to the original. The reported results show near-lossless compression at 10% size reduction (OPT-1.3B scores 14.5 perplexity on WikiText-2 versus 14.6 uncompressed) and preserve most ScienceQA reasoning accuracy on a 7B vision-language model even at 50% compression. If correct, this gives deployed models a cheap post-hoc efficiency upgrade without fine-tuning or task-specific data.

What carries the argument

The central object is the joint tensor (Tucker/HOSVD) decomposition of the per-head Gram products $G_i = C^{1/2}W_{q,i}^{\top}W_{k,i}C^{1/2}$ into shared latent planes $A_q, A_k$ and a core $H_i = A_qG_iA_k^{\top}$, where $C$ is the calibration covariance of input activations. The objective being minimized is the squared error of the pre-softmax attention map $\sum_i \|M_i - \hat{M}_i\|^2$; the alternating right-singular-vector updates in Algorithm 1 are what carry the joint QK compression. Two supporting mechanisms complete the argument: the square-root covariance pre-conditioner $P = C^{1/2}$, claimed optimal for activation-aware SVD among the variants in Table 1, and the junction matrix choice $J = V_1$, which turns the compression matrix into block-identity form and removes $r^2$ parameters from every factorized layer.

What would settle it

Compress only the query/key pairs of a model whose attention has near-hard softmax (most softmax weights near zero), then compare end-to-end perplexity against a model with diffuse attention at the same rank; if low attention-map error coincides with a large perplexity jump, the proxy is not faithful and the claimed advantage does not generalize.

Watch

Extended reading notes

Core claim

The central discovery is that the conversion from multi-head attention to latent attention can be posed as a joint tensor decomposition whose objective is the squared error of the pre-softmax attention maps summed over heads, not the usual per-weight approximation error. The paper derives an alternating SVD (a high-order SVD or Tucker decomposition) that factorizes query and key projections into shared compression matrices $A_q, A_k$ and per-head decompression matrices, and it applies the same global-loss idea—with auxiliary variables for the nonlinearity—to the MLP up/down projections. A separate structural finding is that the junction matrix $J$ left unspecified by truncated SVD can be chosen so that one factor becomes block-identity, saving $r^2$ parameters and FLOPs per layer and ensuring low-rank compression always shrinks the model. With these mechanisms, the reported OPT experiments from 125M to 13B parameters show LatentLLM outperforming local activation-aware SVD baselines at every compression level, and the LLaVa experiments show modest degradation in multi-modal reasoning down to 50% of the original size.

Load-bearing premise

The load-bearing premise is that minimizing the summed squared error of the raw, pre-softmax attention maps faithfully proxies end-to-end model quality; this premise ignores the softmax nonlinearity and the value/output pathway, and the paper itself reports that the analogous joint value-output compression failed for exactly that reason.

Editorial extensions

If this is right

  • At 10–20% size reduction, compressed OPT models match or slightly beat the uncompressed model's perplexity on WikiText-2 (e.g., 14.5 vs 14.6 for OPT-1.3B at 10%), so near-lossless, training-free compression is achievable in this regime.
  • Because the junction matrix is chosen to produce a block-identity factor, every compressed layer has strictly fewer parameters and FLOPs than the original whenever the latent rank is below the matrix dimension; reported OPT-6.7B FLOPs fall roughly linearly with compression ratio.
  • On the 7B vision-language model, ScienceQA accuracy drops from 70.0% uncompressed to 65.8% at 10% compression and 52.3% at 50%, so the method preserves a large share of multi-modal reasoning ability without fine-tuning.
  • The same joint-decomposition recipe extends to grouped query attention and to most positional encodings, and the authors state it can be applied to pruning and quantization as well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the unweighted head-summed attention-map loss is the method's main lever; a weighted loss (by head importance, softmax temperature, or downstream-layer sensitivity) would likely improve results at high compression, though the paper does not test this.
  • Editorial inference: the block-identity junction-matrix trick is independent of the joint tensor decomposition and could be dropped into any low-rank compression or adapter pipeline to turn parameter-neutral decompositions into parameter-saving ones.
  • Editorial inference: because the paper's own Remark 11 says joint value-output compression was not effective, the biggest gains are likely where query-key geometry dominates; models whose behavior is driven by the value/output pathway may see less benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes LatentLLM, a training-free method for converting pretrained multi-head attention (MHA) transformers into a reduced-dimension MLA-like form. It derives (i) a root-covariance preconditioner for activation-aware SVD, (ii) a block-identity junction matrix that reduces parameter count and FLOPs without changing the local SVD loss, and (iii) a joint tensor (Tucker/HOSVD) decomposition of query/key and up/down projections that minimizes a pre-softmax attention-map error and a decoupled MLP loss. Experiments on OPT models report perplexity improvements over plain-SVD and ASVD baselines at 10–40% size reduction, with some perplexity values below the uncompressed model, and LLaVA-7B ScienceQA accuracy that degrades less than the baselines at 10–50% compression.

Significance. The analytic derivations of the root-covariance preconditioner and the block-identity junction matrix are a useful contribution; the parameter-count argument in Sec. 3.3 is correct and generally applicable. If the empirical gains are really attributable to the joint attention-aware objective, the method would be a valuable zero-shot LLM/LMM compression technique. However, the current experiments do not isolate that mechanism from the other two ingredients, so the paper's central claim is not yet established. The internal Remark 11 also indicates that the attention-map proxy can be unreliable, which raises the stakes for the missing ablation. The consistent improvements over the local ASVD baselines are encouraging, but they are confounded with the block-identity rank savings.

major comments (2)
  1. [§5, Tables 2 and 4; §3.3] The central claim that the joint attention-aware QK decomposition improves over existing compression methods is not isolated in the reported experiments. Every LatentLLM result combines the joint objective of Eq. (11) with the root-covariance preconditioner (Sec. 3.2) and the block-identity junction matrix (Sec. 3.3). At fixed nominal compression, the block-identity form (Eq. 9) removes r^2 parameters per square layer with no change in the local SVD loss, so LatentLLM can operate at a larger average rank than a local ASVD baseline with the same parameter count. No experiment compares local ASVD with root covariance plus block identity at matched per-layer parameter counts against LatentLLM; without such an ablation, the gains in Tables 2 and 4 could be due entirely to rank allocation. Please add this matched ablation and report the per-layer ranks used by each method.
  2. [§4.1, Eq. (11), Remark 11] The objective in Eq. (11) minimizes the squared error of the pre-softmax attention map summed over heads, ignoring the softmax nonlinearity and the value/output pathway. The paper itself reports in Remark 11 that the analogous joint VO compression was not effective over split V/O compression because the loss does not deal with the nonlinear attention map, showing that this proxy is fragile. Since the joint QK objective is the paper's central technical novelty, the experiments should include evidence that reducing Eq. (11) actually tracks end-to-end perplexity (e.g., compare the selected QK decomposition against a local QK SVD chosen to match the same attention-map loss, or report a sweep over the alternating-SVD iterations with end-to-end perplexity). Without that, the attention-aware mechanism remains a hypothesis rather than a demonstrated cause of the reported gains.
minor comments (5)
  1. [Abstract] The abstract contains 'LLMs/LLMs' in the sentence about computationally/memory-efficient models; this should be 'LLMs/LMMs'.
  2. [§5, Tables 2 and 4] The table captions do not state whether the ASVD baselines use the block-identity junction matrix. This should be stated explicitly, since the block-identity transform is one of the proposed ingredients and its use, or non-use, in the baselines directly affects the interpretation of the comparisons.
  3. [§5, Experiments setup] The values of the damping factor λ, the per-layer ranks (r_q, r_k, r_o, r_v, r_u, r_d), and the calibration sequence length are not reported, despite these being free parameters of the method. The paper states only the numbers of alternating iterations (4 for UD, 8 for QK). Please provide the full configuration, ideally with a sensitivity analysis over λ and rank allocation.
  4. [Algorithm 1] The notation 'RightSingular_r' is used without defining the orientation of the returned matrix; in the surrounding equations A_q and A_k appear sometimes as r×d and sometimes as d×r. Please align the notation so that the algorithm is unambiguous.
  5. [§3.2, Remark 3] The normalization C = (XX^T + λI)/l is introduced without specifying whether l is the sequence length, the number of calibration tokens, or the number of calibration samples. Clarifying this would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LatentLLM's derivations are self-contained and its evaluations use held-out benchmarks.

full rationale

LatentLLM's central contribution is a training-free low-rank conversion of MHA to an MLA-like form. The derivation chain is self-contained: the joint QK objective in Eq. (11) minimizes the squared error of the pre-softmax attention map over calibration activations, which is a well-defined proxy optimization rather than a fitted surrogate for downstream perplexity. The HOSVD/Tucker solution in Algorithm 1 follows from the algebra of the stated loss, and the junction-matrix construction in Sec. 3.3 is an exact reparameterization (Eq. 9) that saves parameters without changing the local SVD error; it may make baselines unfair at matched compression ratios, but it is not circular. The paper does not fit a parameter to the test metric and then rename it a prediction; perplexity and ScienceQA accuracy are measured on held-out sets (WT2, PTB, C4 validation, and the ScienceQA test split). SparseLLM, the main external framework adopted for the MLP loss, is an independent citation whose method is explicitly reused, not a self-citation invoked as an unverified authority. No uniqueness theorem or ansatz is smuggled in via self-citation; the papers cited for global compression (SparseLLM, Q-VLM) are external. The only mild concerns are benchmarking rather than circularity: calibration is on C4 and one of the evaluation sets is C4 validation, and hyperparameters such as lambda, ranks, and iteration counts are not reported with a separate validation split. These affect attribution of the improvement, not the logical dependence of the claimed result on its inputs. Therefore no circular step can be exhibited, and the paper should receive a non-circular verdict.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities, mediators, or conserved quantities are introduced; 'latent structure' is a compression representation, not an invented entity. The central claim rests on one free-parameter-heavy experimental pipeline and five stated assumptions, the most fragile being the attention-map loss proxy and the transfer of calibration statistics.

free parameters (5)
  • damping factor lambda = not reported
    Added to the calibration covariance C = XX^T + lambda I in Sec. 3.2; no value given in the paper, so exact C depends on an unstated choice.
  • per-layer ranks (r_q, r_k, r_o, r_v) = not reported
    Ranks are selected per layer to hit a global compression target; the allocation rule is not specified, so the reported perplexity and FLOPs depend on an unreported rank schedule.
  • alternating SVD iterations N = 4 for UD, 8 for QK
    Set in Sec. 5 without sensitivity analysis; convergence of the HOSVD alternation is not proven.
  • MLP decoupled-loss weights alpha, beta, gamma = not reported
    Eq. (20) defines the joint UD objective with three undetermined weights; the values used in experiments are absent.
  • calibration set size and sequence length = 64 samples of 2048 tokens
    Adopted from SparseGPT and SparseLLM; a methodological choice that affects the covariance estimate and downstream results.
assumptions (5)
  • domain assumption The activation covariance C estimated from 64 calibration samples transfers to the test distribution.
    Sec. 5 uses 64 C4 segments to compute C for all OPT models; no validation that this estimate is stable across seeds.
  • domain assumption Minimizing the pre-softmax attention-map error (Eq. 11) is a valid proxy for compression quality.
    Used in Sec. 4.1; Remark 11 concedes this loss choice was ineffective for joint VO compression.
  • ad hoc to paper The alternating HOSVD procedure converges to an adequate solution in a few iterations.
    Algorithm 1 runs 4-8 iterations with no convergence guarantee or error bound.
  • standard math SVD low-rank approximation under the weighted loss is globally optimal.
    Standard Eckart-Young result used in Sec. 3.2 and Appendix B.
  • domain assumption The SparseLLM decoupled MLP loss yields an effective low-rank approximation.
    Adopted in Sec. 4.3 and Appendix H from SparseLLM [3]; assumes the auxiliary variable tricks preserve fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LatentLLM: Attention-Aware Joint Tensor Compression." pith.science (2026). https://pith.science/paper/W77LJ2OT

@misc{pith2026250518413,
  author       = {Pith},
  title        = {Pith review of: LatentLLM: Attention-Aware Joint Tensor Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W77LJ2OT}},
  note         = {Machine review of arXiv:2505.18413}
}
read the original abstract

Modern foundation models such as large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources. We propose a new framework to convert such LLMs/LMMs into a reduced-dimension latent structure. Our method extends a local activation-aware tensor decomposition to a global attention-aware joint tensor de-composition. Our framework can significantly improve the model accuracy over the existing model compression methods when reducing the latent dimension to realize computationally/memory-efficient LLMs/LLMs. We show the benefit on several benchmark including multi-modal reasoning tasks.

Figures

Figures reproduced from arXiv: 2505.18413 by the authors.

Figure 1
Figure 1. Reduced-dimension LLM/LMM with low-rank tensor [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Activation-aware compression with pre-conditioning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Tucker decomposition for joint QK compression. The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Perplexity over compression ratio for OPT models. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Perplexity vs. FLOPs of compressed OPT for 125M to 13B models. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Radar plots of ScienceQA accuracy results across different subjects, context modalities, and grades at [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of SVD, CorDA, and RootCorDA. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Joint-QKV vs split-QKV approximation. In this case, the optimal solution is an SVD of W C 1 2 . Note that this is different from QKV individual optimization: L ′ = X i∈[q,k,v] ∥WiX − BiAiX∥ 2 (51) = [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Split-head activation-aware approximation had terrible performance. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Attention-Aware vs. Activation-Aware Approximation. Loss is attention map error. Random query/key projections with Wishart [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Sparse approximation for Attention-Aware vs. Activation-Aware distillation. No markers are sparse approximation. Sparse is [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: RoPE-Attention-Aware Distillation: 10-token window. [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Random weight approximation with/without correlation. Correlation is sampled from Wishart distribution with covariance of [PITH_FULL_IMAGE:figures/full_fig_p033_13.png]
Figure 14
Figure 14. Figure 14: Low-rank plus sparse approximation does not outperform sparse-alone approximation. [PITH_FULL_IMAGE:figures/full_fig_p034_14.png]
Figure 15
Figure 15. Figure 15: Sparsification of B and A low-rank matrices [PITH_FULL_IMAGE:figures/full_fig_p035_15.png]
Figure 16
Figure 16. Figure 16: Comparison with WandA. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 27 canonical work pages

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Beyond efficiency: A systematic survey of resource-efficient large language models

    Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiay- ing Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. Beyond efficiency: A systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00625, 2024. 1

    arXiv 2024

  3. [3]

    SparseLLM: Towards global pruning of pre-trained language models

    Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. SparseLLM: Towards global pruning of pre-trained language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2, 4, 5, 7, 31, 33

    work page 2024

  4. [4]

    Sparks of artificial general intelligence: Early experiments with GPT-4

    S ´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Jo- hannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712, 2023. 1

    arXiv 2023

  5. [5]

    Palu: Compressing KV-cache with low-rank projection

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing KV-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024. 1, 2, 3

    arXiv 2024

  6. [6]

    Transformer-XL: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. 26

    arXiv 1901

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai...

    work page 2025

  8. [8]

    Exploiting linear structure within convolutional networks for efficient evaluation

    Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann Le- Cun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014. 2, 3

    work page 2014

Show all 51 references
  1. [9]

    The case for 4-bit pre- cision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit pre- cision: k-bit inference scaling laws. In International Confer- ence on Machine Learning, pages 7750–7774. PMLR, 2023. 7

    2023

  2. [10]

    SparseGPT: Massive lan- guage models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. SparseGPT: Massive lan- guage models can be accurately pruned in one-shot. In In- ternational Conference on Machine Learning, pages 10323– 10337. PMLR, 2023. 1, 3, 7, 14, 33

    2023

  3. [11]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. 1, 3, 7, 14

    2022 arXiv

  4. [12]

    Optimal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. InIEEE interna- tional conference on neural networks, pages 293–299. IEEE,

  5. [13]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2...

    arXiv

  6. [14]

    PC-LoRA: Low-rank adaptation for progressive model compression with knowledge distilla- tion

    Injoon Hwang, Haewon Park, Youngwan Lee, Jooyoung Yang, and SunJae Maeng. PC-LoRA: Low-rank adaptation for progressive model compression with knowledge distilla- tion. arXiv preprint arXiv:2406.09117, 2024. 1

    2024 arXiv

  7. [15]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 1

    2024 arXiv

  8. [16]

    GPT-4 passes the bar exam

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. Philo- sophical Transactions of the Royal Society A , 382(2270): 20230254, 2024. 1

    2024

  9. [17]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of naacL-HLT, page 2. Minneapolis, Minnesota, 2019. 26

    2019

  10. [18]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989. 1

    1989

  11. [19]

    A well-conditioned esti- mator for large-dimensional covariance matrices

    Olivier Ledoit and Michael Wolf. A well-conditioned esti- mator for large-dimensional covariance matrices. Journal of multivariate analysis, 88(2):365–411, 2004. 2

    2004

  12. [20]

    LoSparse: Structured com- pression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoSparse: Structured com- pression of large language models based on low-rank and sparse approximation. In International Conference on Ma- chine Learning, pages 20336–20350. PMLR, 2023. 2

    2023

  13. [21]

    Beyond linear approximations: A novel pruning approach for attention matrix

    Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, and Yufa Zhou. Beyond linear approximations: A novel pruning approach for attention matrix. arXiv preprint arXiv:2410.11261, 2024. 4

    2024 arXiv

  14. [22]

    MoE-LlaVa: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. MoE-LlaVa: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 1

    2024 arXiv

  15. [23]

    AWQ: Activation-aware weight quantization for on-device LLM compression and accelera- tion

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and accelera- tion. Proceedings of Machine Learning and Systems , 6:87– ...

    2024

  16. [24]

    DeepSeek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 1, 2, 3, 13

    2024 arXiv

  17. [25]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 1, 7

    2023

  18. [26]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  19. [27]

    The penn treebank: Annotating pred- icate argument structure

    Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating pred- icate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey,...

    1994

  20. [28]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. 7 11

    2016 arXiv

  21. [29]

    PyTorch: An imperative style, high-performance deep learning li- brary

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning li- brary. Advances in neural information processing systems ...

    2019

  22. [30]

    Improving language understanding by gener- ative pre-training

    Alec Radford. Improving language understanding by gener- ative pre-training. Preprint, 2018. 26

    2018

  23. [31]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 7

    2020

  24. [32]

    Compressing large language models using low rank and low precision decomposition

    Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems , 37: 88981–89018, 2024. 1, 2

    2024

  25. [33]

    Low-rank matrix factorization for deep neural network training with high- dimensional output targets

    Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high- dimensional output targets. In 2013 IEEE international con- ference on acoustics, speech and signal processing , pa...

    2013

  26. [34]

    Eigen attention: Attention in low- rank space for KV cache compression

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low- rank space for KV cache compression. arXiv preprint arXiv:2408.05646, 2024. 1, 2

    2024 arXiv

  27. [35]

    Low-rank lottery tick- ets: finding efficient low-rank neural networks via matrix differential equations

    Steffen Schotth ¨ofer, Emanuele Zangrando, Jonas Kusch, Gi- anluca Ceruti, and Francesco Tudisco. Low-rank lottery tick- ets: finding efficient low-rank neural networks via matrix differential equations. Advances in Neural Information Pro- cessing Systems, 35:20051–20063, 2022. 1

    2022

  28. [36]

    Green AI

    Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Et- zioni. Green AI. Communications of the ACM , 63(12):54– 63, 2020. 1

    2020

  29. [37]

    RoFormer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  30. [38]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023. 1, 3, 7, 14, 33

    2023 arXiv

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  32. [40]

    Q-VLM: Post-training quanti- zation for large vision-language models

    Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Q-VLM: Post-training quanti- zation for large vision-language models. arXiv preprint arXiv:2410.08119, 2024. 1, 2

    2024 arXiv

  33. [41]

    SVD-LLM: Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024. 2

    2024 arXiv

  34. [42]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 ,

    arXiv

  35. [43]

    Huggingface’s transformers: State-of-the-art natu- ral language processing

    T Wolf. Huggingface’s transformers: State-of-the-art natu- ral language processing. arXiv preprint arXiv:1910.03771,

    1910 arXiv

  36. [44]

    A survey on model com- pression and acceleration for pretrained language models

    Canwen Xu and Julian McAuley. A survey on model com- pression and acceleration for pretrained language models. In Proceedings of the AAAI Conference on Artificial Intel- ligence, pages 10566–10575, 2023. 1

    2023

  37. [45]

    CorDA: Context-oriented decomposition adaptation of large language models

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-oriented decomposition adaptation of large language models. arXiv preprint arXiv:2406.05223 , 2024. 3, 14

    2024 arXiv

  38. [46]

    ZeroQuant: Ef- ficient and affordable post-training quantization for large- scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xi- aoxia Wu, Conglong Li, and Yuxiong He. ZeroQuant: Ef- ficient and affordable post-training quantization for large- scale transformers. Advances in Neural Information Process- ing Systems, 35:27168–27183, 2022. 7

    2022

  39. [47]

    ASVD: Activation-aware singular value decomposition for compressing large language models

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023. 1, 2, 3, 14

    2023 arXiv

  40. [48]

    LLM inference unveiled: Survey and roofline model insights

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. LLM inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363 ,

    arXiv

  41. [49]

    OPT: Open pre-trained trans- former language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained trans- former language models. arXiv preprint arXiv:2205.01068,

    arXiv

  42. [50]

    C 1 2 O µ⊤C −1 2 (1 − µ⊤C +µ) 1 2 #

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguis- tics, 12:1556–1577, 2024. 1 12 A. Weight-Aware Compression A.1. Plain SVD Given a pretrained weight matri...

    2024

  43. [51]

    (193) Plugging into the loss gives: L = X i ∥Wo,iWv,i(X − µ1⊤) − ˆWo,i ˆWv,i(X − µ1⊤)∥2 (194) = X i ∥ Wo,iWv,i| {z } Gi∈Rd×d C 1 2 0 − Bo Ao,iBv,i| {z } Hi∈Rro ×rv AvC 1 2 0 ∥2

    (192) Thus the KKT condition gives: ˆbo,i = bo,i + Wo,i(Wv,iµ + bv,i) − ˆWo,i( ˆWv,iµ + ˆbv,i). (193) Plugging into the loss gives: L = X i ∥Wo,iWv,i(X − µ1⊤) − ˆWo,i ˆWv,i(X − µ1⊤)∥2 (194) = X i ∥ Wo,iWv,i| {z } Gi∈Rd×d C 1 2 0 − Bo Ao,iBv,i| {z } Hi∈Rro ×rv AvC 1 2 0 ∥2. (19...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.