Pith. sign in

REVIEW 46 cited by

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.13137 v3 pith:DR3U7MOI submitted 2023-08-25 cs.LG cs.CL

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

Wenqi Shao , Mengzhao Chen , Zhaoyang Zhang , Peng Xu , Lirui Zhao , Zhiqian Li , Kaipeng Zhang , Peng Gao
show 2 more authors
Yu Qiao Ping Luo
This is my paper
classification cs.LG cs.CL
keywords quantizationomniquantlanguagememorymodelsperformanceactivationcalibrated
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) have revolutionized natural language processing tasks. However, their practical deployment is hindered by their immense memory and computation requirements. Although recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM, they hand-craft quantization parameters, leading to low performance, especially in extremely low-bit quantization. To tackle this issue, we introduce an Omnidirectionally calibrated Quantization (\textbf{OmniQuant}) technique for LLMs, which achieves good performance in diverse quantization settings while maintaining the computational efficiency of PTQ by efficiently optimizing various quantization parameters. OmniQuant comprises two innovative components including Learnable Weight Clipping (LWC) and Learnable Equivalent Transformation (LET). LWC modulates the extreme values of weights by optimizing the clipping threshold. Meanwhile, LET tackles activation outliers by shifting the challenge of quantization from activations to weights. Operating within a differentiable framework using block-wise error minimization, OmniQuant can optimize the quantization process efficiently for both weight-only and weight-activation quantization. For instance, the LLaMA-2 model family size 7-70B can be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using 128 samples. Extensive experiments validate OmniQuant's superior performance across diverse quantization configurations such as W4A4 (4-bit weight, 4-bit activation), W6A6, W4A16, W3A16, and W2A16. Additionally, OmniQuant demonstrates effectiveness in instruction-tuned models and delivers notable improvements in inference speed and memory reduction on real devices. Codes are available at \url{https://github.com/OpenGVLab/OmniQuant}.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 46 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts

    cs.LG 2026-05 unverdicted novelty 8.0

    HodgeCover isolates the harmonic kernel of a simplicial Laplacian on an expert 2-complex to identify irreducible merge cycles and selects experts for aggressive compression, matching or exceeding baselines on open-wei...

  2. AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

    cs.LG 2026-05 unverdicted novelty 7.0

    AAAC uses two adaptive 64-byte codebooks per layer for 4-bit LLM weight quantization, choosing the optimal one per group to minimize activation-weighted error with zero storage overhead and fast runtime.

  3. AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

    cs.LG 2026-05 unverdicted novelty 7.0

    AAAC learns two 64-byte codebooks per layer for 4-bit LLM weights and lets each group pick the one minimizing activation-weighted reconstruction error, storing the choice at zero extra cost.

  4. SpinQuant: LLM quantization with learned rotations

    cs.LG 2024-05 conditional novelty 7.0

    SpinQuant learns optimal rotations to enable accurate 4-bit quantization of LLM weights, activations, and KV cache, reducing the zero-shot gap to full precision to 2.9 points on LLaMA-2 7B.

  5. KronQ: LLM Quantization via Kronecker-Factored Hessian

    cs.LG 2026-07 accept novelty 6.5

    Kronecker-factored Hessian PTQ with bidirectional incoherence and joint-trace mixed precision yields stable 2-bit LLM weights where activation-only methods fail.

  6. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    cs.AR 2026-07 conditional novelty 6.0

    Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...

  7. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    cs.DC 2026-07 conditional novelty 6.0

    Tensor-granularity hybrid CPU-GPU offloading with static placement plus load-aware dynamic transfer yields up to 1.94× prefill and 3.29× decode throughput for local LLMs on consumer devices.

  8. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

    cs.AR 2026-07 conditional novelty 6.0

    Near-memory DeQuantization Blocks in the HBM base die dequantize weight-only quantized LLM weights on the load path, cutting GPU dequant overhead and yielding up to 7.08× mpGEMM speedup and 2.20× decode throughput in ...

  9. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    cs.LG 2026-06 unverdicted novelty 6.0

    SharQ combines input-adaptive N:M sparsity and FP4 quantization via sparse backbone plus dense residual, recovering 43-63% of the NVFP4-to-FP16 accuracy gap on Llama and Qwen models without calibration or retraining.

  10. HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models

    cs.LG 2026-06 unverdicted novelty 6.0

    HyperQuant unifies Hadamard transform, optimal lattice quantization, and entropy coding to outperform prior schemes on LLM weight and KV cache quantization down to 1.7 bits per scalar while preserving quality on a 19B...

  11. FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models

    cs.LG 2026-06 unverdicted novelty 6.0

    FAIR-Calib is a frontier-aware instability-reweighted calibration framework for PTQ of dLLMs that minimizes reweighted hidden-state MSE to reduce frontier decision flips.

  12. FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models

    cs.LG 2026-06 conditional novelty 6.0

    A two-stage PTQ method for diffusion LLMs that reweights calibration toward fragile 'write-frontier' positions improves W4A4 accuracy over existing baselines.

  13. Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference

    cs.LG 2026-06 unverdicted novelty 6.0

    Qift defines a fixed no-zero W2 level set for rotated weights that improves W2A4 perplexity and accuracy on LLaMA-2-7B and LLaMA-3.1-8B over the standard {-2,-1,0,1} set.

  14. LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models

    cs.LG 2026-05 unverdicted novelty 6.0

    LASER introduces curvature-weighted SVD from second-order loss approximation and loss-aware rank allocation to compress VLMs, reporting over 2.3x decoding speedup under low-precision settings.

  15. DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

    cs.LG 2026-05 unverdicted novelty 6.0

    DREAM-S combines neural architecture search, target-aware supernet training, and attention-entropy-guided distillation to accelerate speculative decoding in VLMs, reporting up to 3.85x speedup over standard methods.

  16. ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

    cs.LG 2026-05 unverdicted novelty 6.0

    ProjQ constrains post-training quantization noise to a low-rank manifold through orthogonal subspace projection, enabling better compensation by LoRA adapters and preserving greater model plasticity than standard PTQ.

  17. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

    cs.LG 2026-05 unverdicted novelty 6.0

    GEMQ applies global LP-based expert importance estimation and router fine-tuning within progressive quantization to cut memory and speed inference in MoE LLMs with little accuracy loss.

  18. Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    SplitQ improves low-bit PTQ for VLMs by isolating modality-specific outlier channels via MOCD and applying dual-branch adaptive calibration via ACC, outperforming prior methods on six datasets across W4A8 to W3A2 settings.

  19. OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

    cs.LG 2026-05 unverdicted novelty 6.0

    OSCAR achieves near-BF16 accuracy for 2-bit KV cache quantization by using offline spectral covariance-aware rotations aligned with attention, plus a custom deployable INT2 kernel compatible with paged serving.

  20. Search Your Block Floating Point Scales!

    cs.LG 2026-05 unverdicted novelty 6.0

    ScaleSearch optimizes block floating point scales via fine-grained search to cut quantization error by 27% for NVFP4, improving PTQ by up to 15 points on MATH500 for Qwen3-8B and attention PPL by 0.77 on Llama 3.1 70B.

  21. Theory-optimal Quantization Based on Flatness

    cs.LG 2026-05 unverdicted novelty 6.0

    The paper introduces the Flatness metric, derives a theory-optimal quantization solution, and presents BDQ that uses bidirectional diagonal transformations to reduce outlier impact, achieving under 1% drop at W4A4 on ...

  22. OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

    cs.LG 2026-05 unverdicted novelty 6.0

    OSAQ uses the low-rank structure of the Hessian to construct a closed-form additive weight transformation that suppresses outliers without changing task loss, enabling better low-bit LLM quantization.

  23. OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

    cs.LG 2026-05 unverdicted novelty 6.0

    OSAQ suppresses weight outliers in LLMs via a closed-form additive transformation from the Hessian's stable null space, improving 2-bit quantization perplexity by over 40% versus vanilla GPTQ with no inference overhead.

  24. BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment

    cs.LG 2026-04 unverdicted novelty 6.0

    BitRL enables on-device RL agents via 1-bit quantized language models, delivering 10-16x memory reduction and 3-5x energy efficiency gains with 85-98% retained performance.

  25. RAT: RunAnyThing via Fully Automated Environment Configuration

    cs.SE 2026-04 unverdicted novelty 6.0

    RAT is a new language-agnostic system that automates environment configuration for any code repository and improves setup success rate by 29.6% over baselines on the new RATBench benchmark.

  26. BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation

    cs.NE 2026-04 unverdicted novelty 6.0

    BiSpikCLM is the first fully binary spiking MatMul-free causal language model that matches ANN performance on generation tasks using only 4-6 percent of the compute via softmax-free spiking attention and spike-aware d...

  27. Spike-driven Large Language Model

    cs.NE 2026-04 unverdicted novelty 6.0

    SDLLM is a spike-driven LLM that uses gamma-SQP two-step encoding, bidirectional symmetric quantization, and membrane potential clipping to achieve 7x lower energy consumption and 4.2% higher accuracy than prior spike...

  28. Rethinking Residual Errors in Compensation-based LLM Quantization

    cs.LG 2026-04 conditional novelty 6.0

    Redefining residual errors to include compensation-aware discrepancies and realigning calibration to full-precision outputs improves GPTQ and GPTAQ performance on LLMs.

  29. FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling

    cs.LG 2026-04 unverdicted novelty 6.0

    Sol-RL decouples FP4-based candidate exploration from BF16 policy optimization in diffusion RL, delivering up to 4.64x faster convergence with maintained or superior alignment performance on models like FLUX.1 and SD3.5.

  30. LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIM

    cs.AR 2026-04 conditional novelty 6.0

    LOCALUT delivers 1.82x geometric mean speedup for quantized DNN inference on real UPMEM DRAM-PIM devices by using operation-packed LUTs with canonicalization, reordering, and slice streaming.

  31. Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

    cs.AR 2026-02 conditional novelty 6.0

    Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.

  32. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 conditional novelty 6.0

    TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...

  33. SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models

    cs.CV 2025-09 conditional novelty 6.0

    A training-free framework jointly quantizes a VLA model to 4 bits and prunes visual tokens, recovering or exceeding full-precision success rates at 1.93x speedup.

  34. Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models

    cs.CL 2025-08 conditional novelty 6.0

    A progressive training scheme with binary-aware initialization and dual-scaling allows pre-trained LLMs to be converted to high-performance 1-bit models without training from scratch.

  35. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  36. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    cs.DC 2026-07 conditional novelty 5.0

    Tensor-granularity hybrid CPU-GPU offloading with static placement and load-aware dynamic transfer improves local LLM prefill and decode throughput on consumer devices.

  37. Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models

    cs.AI 2026-06 unverdicted novelty 5.0

    SAGE-PTQ is a graph-guided ultra-low-bit PTQ framework that achieves 1.03 average weight bits and 0.004 scaling bits per matrix on LLMs while reporting lower perplexity and memory use than BiLLM and PB-LLM.

  38. MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models

    cs.CV 2026-06 unverdicted novelty 5.0

    MorphoQuant proposes DABC and MDQFO for 4-bit quantization of omni-modal LLMs, claiming superior performance over SOTA W4A4 methods and even W4A16 baselines on benchmarks like ScienceQA.

  39. Break Through the Compression Bottleneck: From Theory to Practice

    cs.CL 2026-05 reject novelty 5.0

    The paper asserts a first proof that low-rank decomposition and quantization are non-orthogonal tools for LLM compression, recommends low-rank-first ordering, and adds a diagonal scaling fix (DAM) that reduces the com...

  40. DA-PTQ: Drift-Aware Post-Training Quantization for Efficient Vision-Language-Action Models

    cs.RO 2026-04 unverdicted novelty 5.0

    DA-PTQ quantizes VLAs by compensating cross-space distortions and allocating mixed precision to minimize motion errors and kinematic drift in trajectories.

  41. BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

    cs.LG 2026-04 unverdicted novelty 5.0

    BWTA achieves near full-precision accuracy on BERT and LLMs using binary weights and ternary activations, with 16-24x kernel speedups via specialized CUDA kernels.

  42. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 reject novelty 5.0

    A per-layer, per-task bit-allocation rule based on hidden-activation entropy and variance reportedly matches full-precision TriviaQA accuracy within about one point — but the paper's tables and abstract are internally...

  43. Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference

    cs.AR 2025-09 unverdicted novelty 5.0

    PLENA introduces a co-designed system with three optimization pathways for long-context agentic LLM inference, claiming up to 2.23x throughput over A100 and 4.04x energy efficiency.

  44. RAT: RunAnyThing via Fully Automated Environment Configuration

    cs.SE 2026-04 conditional novelty 4.0

    Three VQ-based QAT techniques (cosine-similarity STE, hard-attention VQ, mixed VQ/LQ NAS) are proposed; they simplify training but do not consistently outperform existing methods.

  45. Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks

    cs.LG 2026-04 unverdicted novelty 4.0

    Three incremental VQ techniques using cosine similarity and NAS for neural net weight compression yield design insights without consistent accuracy gains over existing approaches.

  46. ECG Foundation Models and Medical LLMs for Agentic Cardiovascular Intelligence at the Edge: A Review and Outlook

    eess.SP 2026-04 unverdicted novelty 3.0

    ECG foundation models for signal interpretation and medical LLMs for reasoning can be integrated into agentic systems for real-time cardiovascular intelligence on edge devices.