Pith. sign in

REVIEW 24 cited by

LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.17888 v1 pith:IGKD77JM submitted 2023-05-29 cs.CL

Zechun Liu , Barlas Oguz , Changsheng Zhao , Ernie Chang , Pierre Stock , Yashar Mehdad , Yangyang Shi , Raghuraman Krishnamoorthi
show 1 more author
Vikas Chandra
This is my paper · ORCID
classification cs.CL
keywords quantizationmethodsdownlargemodelmodelstrainingaware
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  2. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  3. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  4. SiLQ: Simple Large Language Model Quantization-Aware Training

    cs.LG 2025-07 conditional novelty 6.0 of 10

    SiLQ fine-tunes 8B-parameter LLMs with quantized weights, activations, and cache for a small fraction of extra training tokens, matching or beating leading post-training quantization methods.

  5. GeLaCo: An Evolutionary Approach to Layer Compression

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.

  6. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models

    cs.SE 2025-07 conditional novelty 6.0 of 10

    4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.

  7. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  8. Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    UPQ, a progressive FP16-to-INT4-to-INT2 pipeline with teacher-student distillation, is the first to quantize open-source instruction-tuned LLMs to 2-bit without proprietary post-training data.

  9. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  10. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  11. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  12. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  13. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  14. LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

    cs.CV 2025-09 conditional novelty 5.0 of 10

    LUQ assigns ultra-low bit widths to transformer layers whose output activations have low entropy, cutting multimodal LLM memory by up to 40 percent with modest benchmark loss.

  15. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  16. P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs

    cs.CL 2025-07 reject novelty 5.0 of 10

    P-CoT prompting improves many LLM results on PhonologyBench tasks, but it does not consistently beat baselines across all models and tasks as the paper claims.

  17. LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation

    cs.LG 2025-05 reject novelty 5.0 of 10

    LCD clusters LLM weights into tiny codebooks under a Hessian-guided objective and uses lookup-table inference to reach 2-3 bits, with reported speedups up to 6.2x.

  18. Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.

  19. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  20. Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks

    cs.LG 2025-05 conditional novelty 4.0 of 10

    The paper makes the case that tensorized neural networks offer valuable compression, scaling, and interpretability advantages that the deep learning community has not yet fully exploited.

  21. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  22. Large Language models for Time Series Analysis: Techniques, Applications, and Challenges

    cs.LG 2025-05 reject novelty 3.0 of 10

    A review of LLM-based time series analysis that proposes several taxonomies, but is undermined by citation errors and a lack of systematic methodology.

  23. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

  24. RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models

    cs.RO 2025-06

Pith tools