Pith. sign in

REVIEW 24 cited by

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.00456 v2 pith:WABLRAOR submitted 2024-03-30 cs.LG

Saleh Ashkboos , Amirkeivan Mohtashami , Maximilian L. Croci , Bo Li , Pashmina Cameron , Martin Jaggi , Dan Alistarh , Torsten Hoefler
show 1 more author
James Hensman
This is my paper · ORCID
classification cs.LG
keywords quarotllmsquantizationwithoutactivationsbitscachehidden
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MOSAIC: Masked Outsourcing of Secure AI Computations

    cs.CR 2026-07 conditional novelty 7.0 of 10

    A secure approximate matrix-multiplication outsourcing protocol with O((m+n)l) client overhead that keeps 70B transformer perplexity within ~1% of BF16 at sigma=0.5 under claimed 140-bit LWE/LPN parameters.

  2. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  3. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  4. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    LightRot uses grouped local rotation plus outlier alignment to make 4-bit LLaMA inference accurate and cheap, claiming 27.4 TOPS/W on a 28nm accelerator.

  5. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  6. Price of metric universality in vector quantization is at most 0.11 bit

    cs.IT 2026-02 conditional novelty 6.0 of 10

    A universal vector-quantization codebook exists that is within 0.11 bit/coordinate of covariance-adaptive waterfilling simultaneously for all input covariances, for Gaussian weights.

  7. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 reject novelty 6.0 of 10

    TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...

  8. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  9. Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A magnitude-compensation step that rescales the weights before a pruned layer, combined with iterative pruning, consistently improves perplexity and question-answering accuracy of layer-pruned LLMs.

  10. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  11. PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PCDVQ compresses LLM weights to 2 bits by quantizing vector directions and magnitudes separately with distribution-matched codebooks, reporting modest zero-shot accuracy gains over prior vector quantization baselines.

  12. Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    UniQuanF unifies uniform and binary-coding quantization, adding a learnable affine transform before binary-code mapping, and proves the two-step process collapses to one BCQ inference step at deployment.

  13. Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Q-VDiT quantizes video diffusion transformers to 3-4 bit weights by adding a learned rank-1 error correction (TQE) and a temporal distribution distillation loss (TMD), nearly doubling VBench scene consistency at W3A6 ...

  14. BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    BASE-Q combines bias correction and asymmetric scaling under fixed rotations to improve 4-bit weight-activation quantization, narrowing the accuracy gap to full precision by up to 50.5% over prior rotation-based methods.

  15. Saliency-Aware Quantized Imitation Learning for Efficient Robotic Control

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A saliency-weighted quantization-aware training method lets 4-bit quantized imitation-learning policies match full-precision success rates across robot manipulation, driving, and control benchmarks.

  16. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  17. Scaling Law for Quantization-Aware Training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.

  18. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  19. any4: Learned 4-bit Numeric Representation for LLMs

    cs.LG 2025-07 conditional novelty 5.0 of 10

    any4 learns a per-row 16-value codebook for 4-bit LLM weight quantization via activation-weighted k-means, beating int4/fp4/nf4 on perplexity and matching preprocessing methods like AWQ and GPTQ.

  20. Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Ternary language models trained on 1.2 trillion tokens continue to improve, and a new GPU kernel speeds up their inference up to 5x end-to-end.

  21. Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Quaff shows that activation outlier channels keep their spatial positions during LLM fine-tuning, and exploits this stability to cut fine-tuning memory and latency with INT8 quantization while matching or beating full...

  22. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.

  23. Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

    cs.CL 2025-05 reject novelty 4.0 of 10

    The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.

  24. Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

    cs.LG 2025-05 conditional novelty 4.0 of 10

    ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.

Pith tools