REVIEW 8 cited by
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference. W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation. To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations. QQQ employs adaptive smoothing and Hessian-based compensation, significantly enhancing the performance of quantized models without extensive training. Furthermore, we meticulously engineer W4A8 GEMM kernels to increase inference speed. Our specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve impressive speed increases of 3.67$\times$ and 3.29 $\times$ over FP16 GEMM. Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 $\times$, 2.10$\times$, and 1.25$\times$ compared to FP16, W8A8, and W4A16, respectively.
Forward citations
Cited by 8 Pith papers
-
Studying quantization trade-offs for efficient inference deployment in machine translation
Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.
-
LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.
-
Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design
EAGLE-2 loses most of its speedup on 4-bit quantized LLMs, and a hierarchical draft-then-sequence scheme restores 1.31x speedup over EAGLE-2 on W4A16 Llama-3-70B.
-
Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.
-
Baichuan-M2: Scaling Medical Capability with Large Verifier System
Baichuan-M2, a 32B medical LLM trained with a patient simulator and a clinical rubric generator as RL verifiers, reports state-of-the-art HealthBench scores (60.1 overall, 34.7 hard), ahead of all open-source models.
-
CogDDN: A Cognitive Demand-Driven Navigation with Decision Optimization and Dual-Process Thinking
CogDDN uses a fast heuristic VLM paired with a slow analytic reflection process and a growing knowledge base to navigate to objects that implicitly satisfy a user's demand, with large reported gains on AI2Thor DDN benchmarks.
-
Is (Selective) Round-To-Nearest Quantization All You Need?
Selective round-to-nearest quantization, combined with Marlin-based kernels, matches or beats GPTQ, AWQ, and BitsAndBytes on accuracy and speed in most tested settings.
-
Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.
Discussion (0). Continue with ORCID to comment.