pith. sign in

arxiv: 2605.26175 · v1 · pith:K3TY5CJInew · submitted 2026-05-25 · 💻 cs.LG · cs.AI

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

Ke Li , Dong An , Xiaoling Zang , Can Ye , Liang Xie , Qibo Qiu , Chen Shen , Xiaofei He
show 1 more author
Wenxiao Wang
This is my paper

Pith reviewed 2026-06-29 23:02 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords activation quantizationLLM quantizationpost-training quantizationdistribution shapingorthogonal transformationlow-bit inferencequantization error
0
0 comments X

The pith

InfoQuant reshapes LLM activations via orthogonal transformation so low-bit uniform quantizers incur less error.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper treats activation quantization not as a problem of suppressing outliers but as one of deliberately designing distributions that match what a low-bit uniform quantizer can represent well. Information-theoretic analysis identifies the target shape: a smaller overall numerical range combined with enough spread among the values inside that range. Guided by this criterion, the method applies a Peak Suppression Orthogonal Transformation that reduces extreme values while preserving dispersion, plus an adaptive rule for choosing which tokens to treat as outliers during the transformation. The resulting distributions produce lower quantization error than prior post-training or training-based baselines across several LLM families.

Core claim

Quantization error drops when activations are transformed so their joint distribution has both a reduced numerical range and sufficient dispersion within that range; the Peak Suppression Orthogonal Transformation reliably produces such distributions in a train-free manner, and adaptive outlier-token selection stabilizes the transformation across models.

What carries the argument

Peak Suppression Orthogonal Transformation (PSOT), an orthogonal matrix applied to activation tensors that suppresses peaks while keeping values spread inside the reduced range.

If this is right

  • 4-bit activation and KV-cache quantization preserves nearly all original model accuracy on average.
  • The performance gap between 4-bit and full-precision LLaMA-2 13B shrinks by 42 percent relative to prior state-of-the-art post-training methods.
  • The same distribution-shaping step works across multiple LLM families without any model retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same range-plus-dispersion criterion might be used to design transformations for weight-only or mixed-precision quantization.
  • Pre-computing the orthogonal matrix once per layer could allow hardware to apply the reshaping with negligible extra latency at inference time.
  • If the criterion proves stable, it could serve as a diagnostic for whether a given quantization scheme is likely to work before any accuracy measurement.

Load-bearing premise

The information-theoretic rule that smaller range plus maintained dispersion inside the range is the right target for a low-bit uniform quantizer, and that PSOT plus adaptive selection can achieve it without changing model behavior.

What would settle it

A new LLM family where applying PSOT and adaptive selection produces no reduction in the gap to floating-point accuracy compared with the previous best post-training method.

Figures

Figures reproduced from arXiv: 2605.26175 by Can Ye, Chen Shen, Dong An, Ke Li, Liang Xie, Qibo Qiu, Wenxiao Wang, Xiaofei He, Xiaoling Zang.

Figure 1
Figure 1. Figure 1: Activation distributions of the LLaMA-2 7B, layer4 q/k/v_proj input under three transfor￾mations: the original activations (left), Hadamard rotation (center), and the learned rotation from PSOT (right). Compared with the original and Hadamard-rotated activations, PSOT produces a more quantization-friendly distri￾bution with a narrower numerical range and larger normalized dispersion. Here, bn denotes the s… view at source ↗
Figure 2
Figure 2. Figure 2: Distributional effect of quantizing the LLaMA-2 7B, layer4 q/k/v_proj input. Top: KL divergence between activation histograms before and after quantization, evaluated over different quanti￾zation step sizes s and dispersion values bn with 15,000 histogram bins. Center/Bottom: activation histograms before and after quantization for low-error and high￾error cases, respectively. Low-bit quantization is most d… view at source ↗
Figure 3
Figure 3. Figure 3: Adaptive threshold selection for ASOT. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of memory overhead in quan￾tization and transform FLOPs during inference across different methods. During inference, FlatQuant per￾forms transformations on three activation values online and dynamically, while other methods apply the fast Hadamard transform only to the KV-cache efficiently. 5.2 Ablation Study Module-wise Impact [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overall rotation diagram of INFOQUANT*. ROPE ROPE softmax Embedding Multi-Head Self-Attention Feed-forward Network output logits Act Activation Quant Hadamard Merge and quantized weights Original activation Quantized activation block diagonal matrix [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overall rotation diagram of INFOQUANT. Model PSOT LAC Time Memory Time Memory LLaMA-2 7B ∼6.0 ∼6 ∼3.0 ∼10 LLaMA-2 13B ∼7.5 ∼7 ∼2.0 ∼12 LLaMA-2 70B ∼13.0 ∼11 ∼3.5 ∼23 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The rotated activation distribution of different layers in LLaMA-2 7B with Hadamard and PSOT. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The rotated activation distribution of different layers in LLaMA-3 8B with Hadamard and PSOT. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
read the original abstract

Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at [https://github.com/LLIKKE/InfoQuant](https://github.com/LLIKKE/InfoQuant)

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript proposes InfoQuant, a train-free post-training quantization method for low-bit LLM activations. It recasts activation transformation as quantizer-facing distribution design and uses an information-theoretic analysis to argue that quantization-friendly activations should have a smaller numerical range combined with sufficient dispersion within that range. The method introduces Peak Suppression Orthogonal Transformation (PSOT) together with adaptive outlier-token selection to produce such distributions. Empirical claims state that InfoQuant outperforms prior PTQ and end-to-end training baselines across LLM families, preserving 97% of floating-point accuracy on average under W4A4KV4 and reducing the LLaMA-2 13B performance gap by 42% relative to the previous state of the art.

Significance. If the empirical results prove robust, the work supplies a principled, train-free route to activation quantization that focuses on explicit distribution shaping rather than outlier suppression alone. The code release and the attempt to ground the transformation in an information-theoretic criterion are strengths that could aid reproducibility and further development in efficient LLM inference.

minor comments (3)
  1. The abstract states strong average gains but does not list the exact models, datasets, or number of runs; adding these details would strengthen the summary.
  2. Notation for the PSOT transformation and the adaptive outlier selection criterion should be introduced with explicit equations or pseudocode in the main text for clarity.
  3. Figure captions and table headers would benefit from stating the exact bit-width configuration (e.g., W4A4KV4) and the metric used for the reported accuracy percentages.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of InfoQuant and the recommendation for minor revision. The provided summary correctly captures the core contribution: recasting activation transformation as distribution design via an information-theoretic lens, with PSOT and adaptive outlier selection yielding strong W4A4KV4 results. No specific major comments appear in the report.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper's central chain begins with an information-theoretic analysis that independently derives the target properties (smaller numerical range + sufficient intra-range dispersion) for low quantization error under uniform discretization. PSOT and adaptive outlier selection are then introduced as a train-free procedure to realize those properties. No equations reduce the reported accuracy gains or distribution targets to quantities defined by the method itself, fitted parameters renamed as predictions, or self-citation chains. The performance claims rest on external benchmarks rather than internal redefinitions, satisfying the criteria for a self-contained, non-circular derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the domain assumption that the identified distribution properties (smaller range plus dispersion) are sufficient for low quantization error and that the introduced transformation reliably produces them; no free parameters or invented physical entities are described in the abstract.

axioms (1)
  • domain assumption Quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range
    This is the explicit conclusion of the information-theoretic analysis that guides the design of PSOT.
invented entities (1)
  • Peak Suppression Orthogonal Transformation (PSOT) no independent evidence
    purpose: To reshape activations into distributions with smaller range and better dispersion for quantization
    New transformation introduced by the paper; no independent evidence outside the method's claimed performance is provided in the abstract.

pith-pipeline@v0.9.1-grok · 5821 in / 1414 out tokens · 57075 ms · 2026-06-29T23:02:29.215334+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset

    KURTAIL : KURTOSIS-BASED LLM QUANTIZATION. InSparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024a. SliceGPT: Compress large language models by deleting rows and columns. InThe Twelfth Inter- national C...

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  2. [2]

    Flatquant: Flatness matters for llm quantization

    Flatquant: Flatness matters for llm quantiza- tion.Preprint, arXiv:2410.09426. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foun- dation and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Albert Tseng,...

    work page arXiv 2023