Pith. sign in

REVIEW 25 cited by

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.04532 v3 pith:VQERXH7F submitted 2024-05-07 cs.CL cs.AIcs.LGcs.PF

Yujun Lin , Haotian Tang , Shang Yang , Zhekai Zhang , Guangxuan Xiao , Chuang Gan , Song Han This is my paper
classification cs.CLcs.AIcs.LGcs.PF
keywords quantizationqserveservinga100inferenceint4l40saccelerate
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Quantization can accelerate large language model (LLM) inference. Going beyond INT8 quantization, the research community is actively exploring even lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization techniques only accelerate low-batch, edge LLM inference, failing to deliver performance gains in large-batch, cloud-based LLM serving. We uncover a critical issue: existing INT4 quantization methods suffer from significant runtime overhead (20-90%) when dequantizing either weights or partial sums on GPUs. To address this challenge, we introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache. QoQ stands for quattuor-octo-quattuor, which represents 4-8-4 in Latin. QoQ is implemented by the QServe inference library that achieves measured speedup. The key insight driving QServe is that the efficiency of LLM serving on GPUs is critically influenced by operations on low-throughput CUDA cores. Building upon this insight, in QoQ algorithm, we introduce progressive quantization that can allow low dequantization overhead in W4A8 GEMM. Additionally, we develop SmoothAttention to effectively mitigate the accuracy degradation incurred by 4-bit KV quantization. In the QServe system, we perform compute-aware weight reordering and take advantage of register-level parallelism to reduce dequantization latency. We also make fused attention memory-bound, harnessing the performance gain brought by KV4 quantization. As a result, QServe improves the maximum achievable serving throughput of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen1.5-72B by 2.4x on A100, 3.5x on L40S, compared to TensorRT-LLM. Remarkably, QServe on L40S GPU can achieve even higher throughput than TensorRT-LLM on A100. Thus, QServe effectively reduces the dollar cost of LLM serving by 3x. Code is available at https://github.com/mit-han-lab/omniserve.

Discussion (0). Sign in to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Apps

    cs.NI 2025-10 conditional novelty 7.0 of 10

    Commercial AI video chat apps differ by 4× in video bitrate, 10× in framerate, and from zero to 10+ minutes of visual memory, with none replying in under 1.5 seconds.

  2. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  3. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    LightRot uses grouped local rotation plus outlier alignment to make 4-bit LLaMA inference accurate and cheap, claiming 27.4 TOPS/W on a 28nm accelerator.

  4. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...

  5. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  6. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  7. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Near-memory DeQuantization Blocks in the HBM base die dequantize weight-only quantized LLM weights on the load path, cutting GPU dequant overhead and yielding up to 7.08× mpGEMM speedup and 2.20× decode throughput in ...

  8. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    cs.LG 2026-01 conditional novelty 6.0 of 10

    By appending quantized residual channels to NVFP4 activations and duplicating the matching weights, ARCQuant reaches W4A8-level accuracy while keeping a single unified 4-bit GEMM.

  9. SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A training-free framework jointly quantizes a VLA model to 4 bits and prunes visual tokens, recovering or exceeding full-precision success rates at 1.93x speedup.

  10. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  11. Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective

    cs.PF 2025-08 conditional novelty 6.0 of 10

    No single LLM quantization method dominates performance, energy, and quality; the best choice depends on task, request length, load, parallelism, and GPU type.

  12. Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference

    cs.AR 2025-07 conditional novelty 6.0 of 10

    A 28nm hybrid systolic array accelerator with MXINT4 quantization and fused RMSNorm/RoPE units reports 247/117 token/s/mm2 running RetNet 1.3B, claiming over 2.45x/13.5x area-efficiency gains over prior edge LLM accelerators.

  13. Kinetics: Rethinking Test-Time Scaling Laws

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.

  14. Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    UniQuanF unifies uniform and binary-coding quantization, adding a learnable affine transform before binary-code mapping, and proves the two-step process collapses to one BCQ inference step at deployment.

  15. Recipes for Pre-training LLMs with MXFP8

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MXFP8-E4M3 with round-up scale computation matches BF16 pre-training accuracy for LLMs up to 8B parameters, enabling 2x faster training on Blackwell.

  16. Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EAGLE-2 loses most of its speedup on 4-bit quantized LLMs, and a hierarchical draft-then-sequence scheme restores 1.31x speedup over EAGLE-2 on W4A16 Llama-3-70B.

  17. PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    PM-KVQ introduces progressive bit-width shrinking, per-block memory allocation, and positional-interpolation calibration to make 2-bit KV cache quantization nearly lossless on long-CoT LLMs.

  18. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  19. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  20. Examining the Efficacy of Graph Neural Network Message-Passing in Regression Contexts

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Across four NAS/DNN-predictor regression benchmarks, GEN (deep graph convolution) achieves the best average rank over 11 GNN message-passing layers, though attention GATv2 wins on the largest graphs.

  21. APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A bit-sliced Tensor Core scheme with a bipolar integer encoding and adaptive kernel autotuning reports 1.65-3.99x speedups for ultra-low-bit GPTQ-quantized LLM inference on three NVIDIA GPUs.

  22. LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation

    cs.LG 2025-05 reject novelty 5.0 of 10

    LCD clusters LLM weights into tiny codebooks under a Hessian-guided objective and uses lookup-table inference to reach 2-3 bits, with reported speedups up to 6.2x.

  23. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  24. Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

    cs.CL 2025-05 reject novelty 4.0 of 10

    The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.

  25. Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs

    cs.LG 2025-05 reject novelty 4.0 of 10

    Adaptively quantizing parts of an LLM's layers to FP4 can improve win rates and trading yields in latency-sensitive agent tasks, but the reported gains come from choosing the best compression level after seeing test results.

Pith tools