Pith. sign in

REVIEW 3 cited by

Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.02723 v1 pith:TF2O6VET submitted 2025-02-04 cs.LG

Qinsi Wang , Jinghan Ke , Masayoshi Tomizuka , Yiran Chen , Kurt Keutzer , Chenfeng Xu This is my paper · ORCID
classification cs.LG
keywords compressionactivationsaddressdobi-svdoptimalsvd-basedweightactivation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We provide a new LLM-compression solution via SVD, unlocking new possibilities for LLM compression beyond quantization and pruning. We point out that the optimal use of SVD lies in truncating activations, rather than merely using activations as an optimization distance. Building on this principle, we address three critical challenges in SVD-based LLM compression: including (1) How can we determine the optimal activation truncation position for each weight matrix in LLMs? (2) How can we efficiently reconstruct the weight matrices based on truncated activations? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose Dobi-SVD, which establishes a new, principled approach to SVD-based LLM compression.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Performant Unified GPU Kernels for Portable Singular Value Computation Across Hardware and Precision

    cs.DC 2025-08 conditional novelty 6.0 of 10

    A unified Julia implementation of two-stage QR SVD achieves near-cuSOLVER performance across four GPU vendors and three precisions, including firsts for Apple Metal and half precision.

  2. LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Loss-aware rank allocation plus residual-stream output correction yields substantially lower WikiText-2 perplexity than prior SVD LLM compressors at 60% compression.

  3. PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint

    cs.LG 2025-09 conditional novelty 3.0 of 10

    PHLoRA extracts LoRA-compatible adapters from full-rank fine-tuned models via truncated SVD of the weight delta, matching full-rank performance on several benchmarks with no gradients or training data.

Pith tools