Pith. sign in

REVIEW 18 cited by

SqueezeLLM: Dense-and-Sparse Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.07629 v4 pith:QEL4HKLM submitted 2023-06-13 cs.CL cs.LG

Sehoon Kim , Coleman Hooper , Amir Gholami , Zhen Dong , Xiuyu Li , Sheng Shen , Michael W. Mahoney , Kurt Keutzer This is my paper · ORCID
classification cs.CLcs.LG
keywords quantizationmodelsinferencememorysqueezellmbaselinecompareddense-and-sparse
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challenge due to their unprecedented resource requirements. This has forced existing deployment frameworks to use multi-GPU inference pipelines, which are often complex and costly, or to use smaller and less performant models. In this work, we demonstrate that the main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, specifically for single batch inference. While quantization has emerged as a promising solution by representing weights with reduced precision, previous efforts have often resulted in notable performance degradation. To address this, we introduce SqueezeLLM, a post-training quantization framework that not only enables lossless compression to ultra-low precisions of up to 3-bit, but also achieves higher quantization performance under the same memory constraint. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format. When applied to the LLaMA models, our 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory requirement. Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup compared to the baseline. Our code is available at https://github.com/SqueezeAILab/SqueezeLLM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

    cs.LG 2025-11 conditional novelty 7.0 of 10

    A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.

  2. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  3. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  4. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  5. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Near-memory DeQuantization Blocks in the HBM base die dequantize weight-only quantized LLM weights on the load path, cutting GPU dequant overhead and yielding up to 7.08× mpGEMM speedup and 2.20× decode throughput in ...

  6. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  7. Information-Bottleneck Driven Binary Neural Network for Change Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    BiCD is a 1-bit change detection network whose auxiliary IB-style losses improve F1 by about 1 to 3 points over other binary networks, with no extra inference cost.

  8. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  9. NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

    cs.LG 2025-05 conditional novelty 6.0 of 10

    NSNQuant applies a Normalize-Shift-Normalize transform plus a Hadamard rotation to make KV cache channels match a standard normal distribution, so one codebook trained on random noise can quantize them without calibration.

  10. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  11. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  12. Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLMs tend to assign high probability to target keywords in the first two generation steps; a plugin using those keywords with parallel constrained generation gives comparable short-text quality and large speedups on s...

  13. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  14. BAQ: Efficient Bit Allocation Quantization for Large Language Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    BAQ assigns per-column bitwidths to LLM weights by solving a convex quantization-loss problem, and reports large 2-bit perplexity gains over GPTQ.

  15. Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

    cs.LG 2025-05 conditional novelty 4.0 of 10

    ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.

  16. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  17. On multi-token prediction for efficient LLM inference

    cs.CL 2025-02 conditional novelty 4.0 of 10

    NTP-trained LLMs can predict a second token by marginalizing over first-token probabilities, but adapting them with extra heads only partially closes the gap to that baseline.

  18. Edge Intelligence in Civil Aviation: Paradigms, Techniques, and Applications

    cs.AI 2026-07 conditional novelty 3.0 of 10

    A survey of edge-AI techniques and four cloud–edge–device deployment paradigms, mapped to civil aviation applications and future trends.

Pith tools