Pith. sign in

REVIEW 18 cited by

SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.07378 v5 pith:RX3D7XUL submitted 2024-03-12 cs.CL cs.LG

Xin Wang , Yu Zheng , Zhongwei Wan , Mi Zhang This is my paper
classification cs.CLcs.LG
keywords compressionsvd-llmsingularmethodsdecompositiondifferentlanguagelarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening technique to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spectral Rewiring for Exploration, Purification, and Model Merging

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Subspace-Aligned Rewiring projects RL weight updates onto the base model’s SVD basis, retaining a compact rewiring matrix that preserves reasoning and improves exploration and multi-domain merging.

  2. FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlashSVD fuses low-rank SVD projections into attention and feed-forward GPU kernels so SVD-compressed transformers avoid materializing dense activations, cutting activation memory at a latency cost.

  3. ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ProcrustesGPT searches for per-layer orthogonal rotations that make pretrained LLM weights fit Kronecker or GS structured matrices, cutting 14 to 36 percent of parameters without fine-tuning.

  4. Multi-objective Large Language Model Alignment with Hierarchical Experts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HoE claims to align a single LLM to any preference vector over multiple objectives using training-free LoRA experts, lightweight trained routers, and nearest-neighbor preference routing.

  5. LatentLLM: Attention-Aware Joint Tensor Compression

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LatentLLM compresses pretrained LLMs and multimodal models with attention-aware joint low-rank tensor decomposition, outperforming SVD-based baselines on OPT perplexity and LLaVA ScienceQA.

  6. Break Through the Compression Bottleneck: From Theory to Practice

    cs.CL 2026-05 reject novelty 5.0 of 10

    The paper asserts a first proof that low-rank decomposition and quantization are non-orthogonal tools for LLM compression, recommends low-rank-first ordering, and adds a diagonal scaling fix (DAM) that reduces the com...

  7. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A single per-layer error tolerance, instead of a per-layer rank budget, improves zero-shot low-rank compression of LLMs and VLMs empirically, but the theorem that this uniform tolerance is Pareto-optimal is proven onl...

  8. PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptation

    cs.CV 2025-09 reject novelty 5.0 of 10

    Fine-tuning a pretrained video diffusion model with LoRA rank 16 on about 1,000 synthetic videos produces panoramic video with good seam closure, but the claim that rank must exceed 8 degrees of freedom is not proven.

  9. Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoning

    cs.CL 2025-08 reject novelty 5.0 of 10

    ASCoT claims later reasoning errors are more harmful than early ones and uses a position-weighted verifier to prune and correct CoT steps, but its key evidence is internally inconsistent.

  10. On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Compression methods can be seen as information projections, and iterative soft rank reduction, not the projection choice, dominates performance after fine-tuning, with a convergence proof for iterative singular value ...

  11. DipSVD: Dual-importance Protected SVD for Efficient LLM Compression

    cs.LG 2025-06 reject novelty 5.0 of 10

    DipSVD combines channel-weighted whitening with layer-wise compression ratios and reports better perplexity and accuracy than existing SVD-based LLM compression methods.

  12. Lag-Relative Sparse Attention In Long Context Training

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Training a language model with LagKV's static sparse attention mask improves its robustness to KV-cache compression at 2x and 4x rates, while claims of overall average gains are not consistently supported.

  13. TuneComp: Joint Fine-tuning and Compression for Large Foundation Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Jointly fine-tuning and compressing a ViT into pruned low-rank factors with progressive distillation achieves better accuracy for smaller parameter counts than sequential fine-tune-then-compress pipelines on CIFAR-100.

  14. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

  15. MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs

    cs.LG 2025-07 conditional novelty 4.0 of 10

    MGAA allocates low-rank compression budget by sublayer input-output cosine similarity and by matrix energy retention, improving compressed LLM perplexity and reasoning accuracy.

  16. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Sub-MoE compresses MoE LLMs by K-means clustering of experts plus frequency-weighted merging of right singular vectors after a shared SVD, and claims 96 and 86 percent retained accuracy at 25 and 50 percent expert red...

  17. Accelerating Attention with Basis Decomposition

    cs.LG 2025-10 reject novelty 3.0 of 10

    A low-rank factorization of attention projection matrices (basis plus coefficients) gives modest FLOP savings in exact arithmetic, but the claimed losslessness and novelty are not supported.

  18. PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint

    cs.LG 2025-09 conditional novelty 3.0 of 10

    PHLoRA extracts LoRA-compatible adapters from full-rank fine-tuned models via truncated SVD of the weight delta, matching full-rank performance on several benchmarks with no gradients or training data.

Pith tools