Pith. sign in

REVIEW 25 cited by

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2211.10438 v7 pith:OPYM6Q6K submitted 2022-11-18 cs.CL cs.AIcs.LG

Guangxuan Xiao , Ji Lin , Mickael Seznec , Hao Wu , Julien Demouth , Song Han This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords quantizationsmoothquantllmsactivationsmodelsweightsaccuracyactivation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 99 citations worldwide. Full citation record

  1. Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Quantizing LLMs selectively forgets memorized text faster than capability, but 1B-scale 4-bit models still extract ~72% of memorized sequences, so quantization is not a privacy defense.

  2. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  3. FP4 All the Way: Fully Quantized Training of LLMs

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A 7-billion parameter LLM trained almost entirely in 4-bit float precision matches a standard 16-bit baseline after a brief quantization-aware fine-tuning phase.

  4. QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Quantization leaves refusal and multiple-choice bias checks flat while open-ended stereotype endorsement remains high (~24–27% under an independent judge), a gap standard safety evaluations miss.

  5. On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Serving frontier MoE and multimodal models on Ascend 910 via vLLM-Ascend is feasible but dominated by engineering cost from incomplete operators, fragile parallelism, kernel faults, and weak observability.

  6. Price of metric universality in vector quantization is at most 0.11 bit

    cs.IT 2026-02 conditional novelty 6.0 of 10

    A universal vector-quantization codebook exists that is within 0.11 bit/coordinate of covariance-adaptive waterfilling simultaneously for all input covariances, for Gaussian weights.

  7. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    cs.LG 2026-01 conditional novelty 6.0 of 10

    By appending quantized residual channels to NVFP4 activations and duplicating the matching weights, ARCQuant reaches W4A8-level accuracy while keeping a single unified 4-bit GEMM.

  8. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 reject novelty 6.0 of 10

    TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...

  9. Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models

    cs.SD 2025-07 conditional novelty 6.0 of 10

    A benchmark of eight post-training quantization methods on Whisper and Moonshine edge speech models across seven datasets, finding 8-bit is safe and 3-bit weights are viable for larger models with advanced methods like SpQR.

  10. SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding

    cs.DC 2025-06 conditional novelty 6.0 of 10

    SwiftSpec uses asynchronous, disaggregated speculative decoding with parallel tree generation and fused kernels to speed up LLM decoding by 1.75x on average over baselines, reaching 348 tokens/s for Llama3-70B on 8 H800 GPUs.

  11. PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PCDVQ compresses LLM weights to 2 bits by quantizing vector directions and magnitudes separately with distribution-matched codebooks, reporting modest zero-shot accuracy gains over prior vector quantization baselines.

  12. Recipes for Pre-training LLMs with MXFP8

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MXFP8-E4M3 with round-up scale computation matches BF16 pre-training accuracy for LLMs up to 8B parameters, enabling 2x faster training on Blackwell.

  13. Refining Datapath for Microscaling ViTs

    cs.AR 2025-05 conditional novelty 6.0 of 10

    MXInt-based datapath designs put all ViT nonlinear operators on an FPGA at 2-5 bit mantissas, with under 1% ImageNet accuracy loss on DeiT models.

  14. BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    BASE-Q combines bias correction and asymmetric scaling under fixed rotations to improve 4-bit weight-activation quantization, narrowing the accuracy gap to full precision by up to 50.5% over prior rotation-based methods.

  15. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  16. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  17. Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

    cs.CL 2026-02 unverdicted novelty 5.0 of 10

    Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...

  18. I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation

    cs.CV 2025-09 conditional novelty 5.0 of 10

    I-Segmenter is an integer-only Vision Transformer for semantic segmentation that keeps mIoU within roughly 5 points of the FP32 baseline while cutting model size by up to 3.8x.

  19. Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Ternary language models trained on 1.2 trillion tokens continue to improve, and a new GPU kernel speeds up their inference up to 5x end-to-end.

  20. Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning

    cs.CL 2025-02 conditional novelty 5.0 of 10

    On-device LLMs reach about half the AMEGA clinical-reasoning score of large cloud models, with Med42 and Aloe highest (about 490/1000) and Phi-3 Mini the best accuracy-per-memory trade-off.

  21. LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

    cs.LG 2026-08 reject novelty 4.0 of 10

    A modified WAIT scheduler with online arrival-rate estimation matches or improves throughput over Sarathi-Serve, ORCA, and vLLM in low-shift bursty workloads.

  22. Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs

    cs.LG 2025-05 reject novelty 4.0 of 10

    Adaptively quantizing parts of an LLM's layers to FP4 can improve win rates and trading yields in latency-sensitive agent tasks, but the reported gains come from choosing the best compression level after seeing test results.

  23. Edge Intelligence in Civil Aviation: Paradigms, Techniques, and Applications

    cs.AI 2026-07 conditional novelty 3.0 of 10

    A survey of edge-AI techniques and four cloud–edge–device deployment paradigms, mapped to civil aviation applications and future trends.

  24. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

  25. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools