Pith. sign in

REVIEW 25 cited by

SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.03078 v1 pith:2Y5MH2Q6 submitted 2023-06-05 cs.CL cs.LG

Tim Dettmers , Ruslan Svirschevski , Vage Egiazarian , Denis Kuznedelev , Elias Frantar , Saleh Ashkboos , Alexander Borzunov , Torsten Hoefler
show 1 more author
Dan Alistarh
This is my paper · ORCID
classification cs.CLcs.LG
keywords llmsspqraccuracycompressionparameterquantizationbitsweights
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities. By compressing such LLMs via quantization to 3-4 bits per parameter, they can fit into memory-limited devices such as laptops and mobile phones, enabling personalized use. However, quantization down to 3-4 bits per parameter usually leads to moderate-to-high accuracy losses, especially for smaller models in the 1-10B parameter range, which are well-suited for edge deployments. To address this accuracy issue, we introduce the Sparse-Quantized Representation (SpQR), a new compressed format and quantization technique which enables for the first time near-lossless compression of LLMs across model scales, while reaching similar compression levels to previous methods. SpQR works by identifying and isolating outlier weights, which cause particularly-large quantization errors, and storing them in higher precision, while compressing all other weights to 3-4 bits, and achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B parameter LLM on a single 24 GB consumer GPU without any performance degradation at 15% speedup thus making powerful LLMs available to consumer without any downsides. SpQR comes with efficient algorithms for both encoding weights into its format, as well as decoding them efficiently at runtime. Specifically, we provide an efficient GPU inference algorithm for SpQR which yields faster inference than 16-bit baselines at similar accuracy, while enabling memory compression gains of more than 4x.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 26 citations worldwide. Full citation record

  1. Fast Entropy Decoding for Sparse MVM on GPUs

    cs.PF 2026-03 conditional novelty 7.0 of 10

    Encoding CSR sparse matrices with dtANS, a decoupled segment-parallel tANS variant, compresses them up to 11.77x over cuSPARSE formats and accelerates GPU SpMVM up to 3.48x on large matrices.

  2. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  4. Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference

    cs.AR 2026-08 conditional novelty 6.0 of 10

    AdaMX puts per-block format and enhancement metadata into spare MX exponent bits, recovering 83% of MXFP4's commonsense accuracy loss at the same 4-bit storage budget.

  5. PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PagedWeight pages quantized MoE expert weights on and off the GPU at runtime, releasing memory to the KV cache while using sensitivity, routing, and prompt signals to choose which experts to shrink.

  6. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  7. Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

    cs.AR 2026-02 conditional novelty 6.0 of 10

    Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.

  8. GeLaCo: An Evolutionary Approach to Layer Compression

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.

  9. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models

    cs.SE 2025-07 conditional novelty 6.0 of 10

    4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.

  10. Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models

    cs.SD 2025-07 conditional novelty 6.0 of 10

    A benchmark of eight post-training quantization methods on Whisper and Moonshine edge speech models across seven datasets, finding 8-bit is safe and 3-bit weights are viable for larger models with advanced methods like SpQR.

  11. GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GPTailor searches over layer removal, layer selection, and layer merging across fine-tuned model variants to produce smaller LLMs that retain more benchmark performance than single-model pruning.

  12. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  13. Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    UPQ, a progressive FP16-to-INT4-to-INT2 pipeline with teacher-student distillation, is the first to quantize open-source instruction-tuned LLMs to 2-bit without proprietary post-training data.

  14. Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ODLRI initializes the low-rank component using activation-outlier channels, improving low-bit compression of large language models over the CALDERA baseline.

  15. TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks

    cs.LG 2025-05 conditional novelty 6.0 of 10

    TSENOR computes transposable N:M masks up to hundreds of times faster than prior solvers by combining entropy-regularized optimal transport with a greedy plus local search rounding.

  16. DarwinLM: Evolutionary Structured Pruning of Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    DarwinLM uses evolutionary search with training-aware offspring selection to prune LLMs, beating ShearedLlama with 5x less post-training data.

  17. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  18. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  19. Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices

    cs.DC 2025-12 reject novelty 5.0 of 10

    This preprint claims large latency speedups for collaborative lossless edge LLM inference via interleaved pipeline parallelism with SSD offloading, but the abstract and body describe different systems (LOIP vs. LIME) ...

  20. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A single per-layer error tolerance, instead of a per-layer rank budget, improves zero-shot low-rank compression of LLMs and VLMs empirically, but the theorem that this uniform tolerance is Pareto-optimal is proven onl...

  21. PoTPTQ: A Two-step Power-of-Two Post-training for LLMs

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A two-step power-of-two quantization method improves low-bit LLaMA perplexity and dequantization speed, but reporting inconsistencies and missing code temper the result.

  22. Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.

  23. TROJAN-GUARD: Hardware Trojans Detection Using GNN in RTL Designs

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A 2-layer GCN on RTL dataflow graphs detects hardware trojans with 98.66% precision and 92.30% recall, but the evaluation rests on only 51 designs with no released code or data.

  24. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  25. Survey of Specialized Large Language Model

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A survey of 24 specialized LLMs (2022-2025) claims a shift from domain fine-tuning to native architectures, but the synthesis is undermined by citation errors and selection bias.

Pith tools