Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Squeeze10-LLM claims a staged mixed-precision recipe can store LLM weights at 1.6 bits on average, cutting 16-bit storage by about 10x while keeping sub-2-bit accuracy near full precision on larger models.

desk verdict A plausible staged quantization method is undermined by a headline 1.6-bit/10x claim that its own appendix contradicts, plus sloppy baseline tables. read the letter →

arxiv 2507.18073 v1 pith:RMZWACZW submitted 2025-07-24 cs.LG

Qingcheng Zhu , Yangyang Ren , Linlin Yang , Mingbao Lin , Yanjing Li , Sheng Xu , Zichao Feng , Haodong Zhu
show 4 more authors
Yuguang Yang Juan Zhang Runqi Wang Baochang Zhang
This is my paper
classification cs.LG
keywords post-trainingquantizationmixed-precisionweight-onlybinarizationlargelanguagemodelsactivationrobustnessmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Squeeze10-LLM, a post-training quantization method that stores 80% of a model's weights in 1 bit and the remaining 20% in 4 bits, averaging 1.6 bits per weight and compressing a 16-bit model's weights by about 10x. Its central claim is that this extreme compression need not cause the usual accuracy collapse if two conditions hold: weight salience is measured by how much binarizing a weight would change output activation ranges (PBAR), and every layer's quantization is supervised by the original full-precision activations rather than by the quantized network's drifting activations (FIAS). On LLaMA and LLaMA2 models from 7B to 70B, the method reports the best accuracy among sub-2-bit weight-only quantization methods, raising the average over six zero-shot tasks from about 43% to 56% on LLaMA2-7B and coming within about 3% of full precision on LLaMA2-70B. A reader should care because, if the result holds, a 10x reduction in weight storage could be obtained with post-training quantization alone and no retraining.

What carries the argument

The load-bearing object is PBAR, a per-weight salience metric $M=V+\lambda B$ that fuses two signals: the usual Hessian-based importance $V$, and a post-binarization activation-range term $B_{ij}=\|\hat Y_{:,i}\|_\infty - \|\hat Y_{:,i}\|_{\min}$, computed from the layer output $\hat Y = X(Q(W;i;j))^T$ when weight $(i,j)$ alone is quantized to 1 bit. It selects the 20% of weights that keep 4-bit precision, so it is the mechanism that decides where the compression budget goes. The companion mechanism is FIAS, which computes Hessians and salience using the original activation matrix $X$ of the pretrained model at every layer, instead of the shifted activations produced by previously quantized layers; this stops error accumulation from misleading later salience decisions. The staged pipeline that carries the whole argument is: 4-bit uniform quantization first, then PBAR-guided selective binarization, with FIAS supervising throughout.

What would settle it

Run Eq. (7) on a single linear layer of LLaMA2-7B and time it: if each $B_{ij}$ takes one forward pass, extrapolating to all 7B weights will exceed the paper's reported single-80G-GPU budget by orders of magnitude. A second check is to compare PBAR's selected 20% of weights against a brute-force measurement of activation-range change on a small layer: if the ranking is unchanged when the $\lambda B$ term is removed, the reported accuracy gains would be expected to disappear.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that staged mixed-precision post-training quantization can reach 1.6 average bits per weight without collapsing a large language model. The pipeline first quantizes all weights uniformly to 4 bits, then computes a salience score $M = V + \lambda B$ for every weight, where $V$ is the Hessian-based salience from SparseGPT ($v_{ij}=w_{ij}^2/[H^{-1}]_{ii}^2$) and $B$ measures the change in output activation range when that single weight is binarized. The 20% of weights with the largest scores stay at 4 bits; the other 80% are binarized to 1 bit. Throughout, FIAS keeps the Hessian and salience computations on the original pretrained activations, arguing this prevents quantization errors from accumulating layer by layer. The experiments report that this configuration outperforms existing sub-2-bit methods on all six tested LLaMA and LLaMA2 models, and on LLaMA2-70B it lands within about 3 percentage points of the 16-bit full-precision model.

Load-bearing premise

The load-bearing premise is that the PBAR salience score in Eq. (7) can be computed for billion-parameter models—each entry, as written, requires a layer output with one individual weight binarized in turn, implying roughly one forward pass per weight, and the paper gives no shortcut—so if that computation is not feasible at scale, the salience ranking the method depends on collapses.

Editorial extensions

If this is right

  • A 10x weight-storage cut for 7B-70B LLaMA models becomes feasible with post-training quantization alone, with no retraining or fine-tuning.
  • PBAR-style activation-range salience, rather than Hessian magnitude alone, is what makes the 20%-at-4-bit allocation work in the paper's experiments.
  • FIAS implies that later layers should be quantized against the original activations, not against the outputs of earlier quantized layers, to prevent cumulative drift.
  • The ablation results imply the 4-bit intermediate stage is not arbitrary: mixing binarization with 2, 3, or 5-8 bit stages is worse on the tested models, so the staging itself is part of the method.
  • Because the gap to full precision shrinks as model size grows, the paper's numbers imply ultra-low-bit weight-only quantization is most viable for the largest models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is whether PBAR's activation-range term also predicts which channels to prune; the metric already measures per-weight impact on outputs and could be reused outside quantization.
  • The FIAS principle of supervising a perturbed network with the original network's activations is a general recipe that may improve other layer-wise perturbations such as iterative pruning or distillation, though the paper only tests it for quantization.
  • The paper reports storage compression, not end-to-end inference speedup; actual deployment would additionally require hardware kernels that can execute 1-bit and 4-bit weights in mixed precision, which the paper does not measure.
  • If PBAR's per-weight forward-pass cost can be approximated by a closed-form bound on activation-range change, the same salience signal could be computed for far larger models than the ones tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Squeeze10-LLM, a post-training mixed-precision quantization framework for LLMs that binarizes 80% of weights and retains 20% at 4-bit precision, claiming an average of 1.6 bits per weight and 10x compression of 16-bit weights. The method introduces two components: Post-Binarization Activation Robustness (PBAR), an activation-range-based salience metric, and Full Information Activation Supervision (FIAS), which uses original pretrained activations to guide layer-wise quantization. Experiments on LLaMA and LLaMA2 report state-of-the-art results for sub-2-bit weight-only quantization across six zero-shot tasks and three perplexity benchmarks.

Significance. If the claims held, this would be a meaningful advance in ultra-low-bit LLM deployment, since sub-2-bit weight-only quantization that stays close to full-precision accuracy would be practically valuable. The conceptual ingredients are reasonable: activation-aware salience and stabilization of supervision signals are both sensible directions. However, the central compression claim is contradicted by the paper's own storage accounting, the salience metric as written is computationally implausible for billion-parameter models, and the baseline tables contain suspicious duplicates and mislabeled rows. The accuracy gains reported for LLaMA2-13B and LLaMA2-70B are striking but need verification after correcting these issues.

major comments (4)
  1. [Abstract, Section 1, Appendix A.3 Eq. (9)] The claimed average of 1.6 bits per weight and the resulting 10x compression omit the index overhead that the paper itself acknowledges. With the correct binarization ratio of rbinary = 0.8, Eq. (9) gives Nbit = 1×0.8 + 4×0.2 + 1 = 2.6 bits per weight, i.e., about 6.15x compression rather than 10x. The appendix instead sets rbinary = 0.2, which is inconsistent with the 80% binarization stated throughout and would give 4.4 bits per weight. The title, abstract, and Section 1 must be revised to reflect the actual storage cost, or the authors must provide a concrete argument for why the index bit can be amortized or eliminated.
  2. [Section 4.2, Eq. (7)] The PBAR metric as defined requires computing a modified layer output with a single weight quantized for each matrix entry B_ij, implying roughly one forward pass per weight position. For billion-parameter LLMs this is computationally infeasible, and the paper gives no efficient algebraic form, approximation, or complexity analysis. Without a practical way to compute B, the salience-selection step is not reproducible or deployable as described. The authors must specify how PBAR is computed in practice, for example by restricting the computation to candidate weights, deriving a closed-form update, or using a stochastic approximation.
  3. [Tables 1 and 2] Several baseline rows appear duplicated across model sizes, which undermines the state-of-the-art comparison. For example, the AWQ rows for LLaMA2-7B and LLaMA2-13B in Table 1 are numerically identical, and the LLaMA2-70B GPTQ row is labeled as 16 bits while reporting a perplexity of 46.08 on WikiText2, which is inconsistent with a 16-bit model. Similar repeated values appear across Table 2 (e.g., BoolQ values of 62.17 for multiple AWQ rows). The authors need to provide corrected baseline numbers and clarify the labeling.
  4. [Section 5.4, Appendix A.2, Table 6] The key design choices — the 4-bit intermediate width (Table 3) and the balancing factor λ (Table 6) — are selected by performance on the same benchmark tasks used to report the final accuracy results. This means the reported numbers are not independent of the design search. The authors should either use a held-out validation split for hyperparameter selection or report sensitivity analyses that show the final conclusions do not depend on these choices.
minor comments (5)
  1. [Tables 3 and 5, Section 5.6] The task name is misspelled as "WinoGrange" in multiple places; it should be "WinoGrande."
  2. [Section 5.6] The ablation text says "Replacing PBAR (i.e., '-FIAS')"; this should read "Replacing FIAS." Also, the text mentions reporting perplexity and WinoGrande accuracy, but Table 5 only shows WinoGrande; either add the perplexity column or revise the text.
  3. [Appendix A.3 Eq. (9)] The definition of rbinary as "the ratio of the binarized weights, taking the value of 0.2" is inconsistent with the 80% binarization ratio used in the main text and experiments. The notation should be corrected and the equation should be clearly labeled as an upper bound including index overhead.
  4. [Checklist, Section 7] The checklist claims results are reproducible with negligible variance by setting a random seed, but no seed value or variance statistics are reported anywhere in the paper. Please provide the specific seeds or at least a statement of observed variance across runs.
  5. [Section 4.2] There is a typo: "effets" should be "effects."

Circularity Check

1 steps flagged · score 4.0 of 10

Squeeze10-LLM's headline gains are mildly circular: lambda, the 4-bit intermediate width, and the 20% salient proportion are selected on the same benchmarks used to report final accuracy, while the rest of the derivation is self-contained.

  1. fitted input called prediction [Section 5.4 (Table 3), Section 5.5 (Table 4), Appendix A.2 (Table 6)]
    "Table 3 presents the impact of mixing binarization with different high-bit quantization levels on LLaMA2-7B. Specifically, we examine model accuracy under the condition that 20% of the salient weights are retained within the 2-bit to 8-bit range. Interestingly, we find that ... the combination of binarization and 4-bit quantization yields the highest performance. Table 6 analyzes the perplexity on WikiText2 datasets across different values of lambda of Eq. (8). For LLaMA2-7B, the best performance is achieved when lambda = 3e-4. This value is also adopted for the quantization of other models."

    The final accuracy and perplexity numbers in Tables 1 and 2 are reported after lambda, the intermediate bit-width, and the salient-weight proportion were selected on those very benchmarks. Appendix A.2 chooses lambda by minimizing WikiText2 perplexity on LLaMA2-7B; Section 5.4 chooses 4-bit because it maximizes zero-shot accuracy on LLaMA2-7B; Section 5.5 chooses 20% from a proportional sweep on LLaMA2-70B. The reported WikiText2 value 9.96 and the LLaMA2-7B zero-shot accuracies are therefore not out-of-sample predictions of a fixed method; they are the optimized results of a search over exactly the metrics later used to claim state-of-the-art performance. This makes the headline gains partly forced by the selection procedure rather than independently derived from the PBAR/FIAS mechanism.

full rationale

Aside from the hyperparameter-selection issue, the derivation chain is self-contained. PBAR is defined directly from a Hessian-based salience term plus a post-binarization activation-range term (Eqs. 5, 7, 8), and FIAS is a stated algorithmic choice to use original pretrained activations for supervision; neither reduces to the final accuracy numbers by construction. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in through the authors' prior work. The bookkeeping inconsistency between the abstract's 1.6-bit average (80% 1-bit + 20% 4-bit) and Appendix A.3's Eq. (9), which adds one index bit and states rbinary = 0.2, is a real correctness/consistency problem, but it is not an instance of circular derivation, so it does not raise the circularity score. The score of 4 reflects that the most directly reported gains on LLaMA2-7B's WikiText2 perplexity and zero-shot accuracy were obtained by selecting lambda, the 4-bit intermediate width, and the 20% proportion on those same tasks, so those particular numbers are partly fitted rather than independently predicted.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method rests on one imported criterion (Hessian salience), one standard domain assumption (layer-wise PTQ), and three empirical postulates introduced for this paper (activation-range salience, 4-bit intermediate optimality, original-activation supervision). There are no new physical entities; the free parameters are lambda, the intermediate bit-width, and the salient-weight proportion, all tuned on the same benchmark suite used for the headline results.

free parameters (3)
  • lambda (PBAR balancing factor) = 3e-4
    Balances Hessian salience and activation-range salience in Eq. (8); selected by WikiText2 perplexity on LLaMA2-7B in Appendix A.2, Table 6.
  • intermediate bit-width = 4 bits
    Chosen after comparing 2-bit to 8-bit intermediate quantization on LLaMA2-7B (Table 3); the best-performing value is adopted and then presented as a design insight.
  • salient weight proportion = 20%
    Selected from a performance sweep on LLaMA2-70B (Table 4) to meet the 1.6-bit average; higher proportions improve accuracy, so the choice is driven by the bit budget plus observed accuracy.
assumptions (5)
  • standard math Hessian-based salience from SparseGPT/GPTQ is a valid measure of weight importance.
    Eq. (5)-(6) import the Hessian criterion from prior work without re-deriving it; the paper relies on it to rank salient weights.
  • domain assumption Layer-wise quantization with calibration activations is a valid approximation of full-network quantization.
    The staged PTQ processes each layer independently using calibration activations (Section 4); no joint error model is given, and FIAS only mitigates, not removes, cross-layer shifts.
  • ad hoc to paper Activation range expansion after binarizing a single weight is a reliable proxy for end-task accuracy loss.
    This is the core intuition of PBAR in Section 4.2; the paper provides no derivation or out-of-sample validation that the B matrix correlates with performance.
  • ad hoc to paper 4-bit is the optimal intermediate representation for staged binarization.
    Chosen empirically from Table 3 and KL divergence in Figure 4; no theoretical argument is given for why 4-bit beats 5-8 bit intermediates.
  • ad hoc to paper Using original pretrained activations for all layers prevents cumulative activation shifts and improves quantization.
    FIAS premise in Section 4.3; supported only by the paper's own ablation and visualizations, not by an independent principle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method." pith.science (2026). https://pith.science/paper/RMZWACZW

@misc{pith2026250718073,
  author       = {Pith},
  title        = {Pith review of: Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMZWACZW}},
  note         = {Machine review of arXiv:2507.18073}
}
read the original abstract

Deploying large language models (LLMs) is challenging due to their massive parameters and high computational costs. Ultra low-bit quantization can significantly reduce storage and accelerate inference, but extreme compression (i.e., mean bit-width <= 2) often leads to severe performance degradation. To address this, we propose Squeeze10-LLM, effectively "squeezing" 16-bit LLMs' weights by 10 times. Specifically, Squeeze10-LLM is a staged mixed-precision post-training quantization (PTQ) framework and achieves an average of 1.6 bits per weight by quantizing 80% of the weights to 1 bit and 20% to 4 bits. We introduce Squeeze10LLM with two key innovations: Post-Binarization Activation Robustness (PBAR) and Full Information Activation Supervision (FIAS). PBAR is a refined weight significance metric that accounts for the impact of quantization on activations, improving accuracy in low-bit settings. FIAS is a strategy that preserves full activation information during quantization to mitigate cumulative error propagation across layers. Experiments on LLaMA and LLaMA2 show that Squeeze10-LLM achieves state-of-the-art performance for sub-2bit weight-only quantization, improving average accuracy from 43% to 56% on six zero-shot classification tasks--a significant boost over existing PTQ methods. Our code will be released upon publication.

Figures

Figures reproduced from arXiv: 2507.18073 by the authors.

Figure 1
Figure 1. Accuracy comparisons of LLaMA2-13B on 6 zero-shot classification tasks [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of Squeeze10-LLM. Squeeze10-LLM is the Mixed-Precision Quantization Frame [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the structural diagrams of Full Information Activation Supervision (FIAS) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of the frequency density distributions of the activation output from the output [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression

    cs.LG 2026-07 conditional novelty 6.0 of 10

    BiSCo-LLM achieves near-FP16 accuracy on Qwen3-8B at ~2 bits/weight using codebook-free binary spherical codes with residual coding and category-wise recovery distillation.

Reference graph

Works this paper leans on

52 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Piqa: Reasoning about physical common- sense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  3. [3]

    Quip: 2-bit quantization of large language models with guarantees

    Jerry Chee, Yaohui Cai, V olodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396–4429, 2023

    work page 2023

  4. [4]

    Progressive mixed-precision decoding for efficient llm inference

    Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, and Stylianos I Venieris. Progressive mixed-precision decoding for efficient llm inference. arXiv preprint arXiv:2410.13461, 2024

    arXiv 2024

  5. [5]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    arXiv 1905

  6. [6]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  7. [7]

    Optimal brain compression: A framework for accurate post-training quantization and pruning

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475–4488, 2022

    work page 2022

  8. [8]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

Show all 52 references
  1. [9]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  2. [10]

    Aptq: Attention-aware post- training mixed-precision quantization for large language models

    Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. Aptq: Attention-aware post- training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pages 1–6, 2024

    2024

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Compressing large language models by joint sparsification and quantization

    Jinyang Guo, Jianyu Wu, Zining Wang, Jiaheng Liu, Ge Yang, Yifu Ding, Ruihao Gong, Haotong Qin, and Xianglong Liu. Compressing large language models by joint sparsification and quantization. In Forty-first International Conference on Machine Learning, 2024

    2024

  5. [13]

    Slim-llm: Salience-driven mixed-precision quantization for large language models

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917, 2024

    2024 arXiv

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 10

    2024 arXiv

  8. [16]

    Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models

    Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13355–13364, 2024

    2024

  9. [17]

    Llm-mq: Mixed-precision quantization for efficient llm deployment

    Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed-precision quantization for efficient llm deployment. In NeurIPS 2023 Efficient Natural Language and Speech Processing Workshop, pages 1–5, 2023

    2023

  10. [18]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  11. [19]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  12. [20]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  13. [21]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  14. [22]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330, 1993

    1993

  15. [23]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  16. [24]

    Adaptive quantization error reconstruction for llms with mixed precision

    Lin Ou, Jinpeng Xia, Yuewei Zhang, Chuzhan Hao, and Hao Henry Wang. Adaptive quantization error reconstruction for llms with mixed precision. In First Conference on Language Modeling, 2024

    2024

  17. [25]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  18. [26]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  19. [27]

    Pb-llm: Partially binarized large language models

    Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034, 2023

    2023 arXiv

  20. [28]

    Omniquant: Omnidirectionally calibrated quantization for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023

    2023 arXiv

  21. [29]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  22. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  23. [31]

    Outlier suppression: Pushing the limit of low-bit transformer language models

    Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022

    2022

  24. [32]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  25. [33]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183, 2022

    2022

  26. [34]

    Rptq: Reorder-based post-training quantization for large language models

    Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023. 11

    2023 arXiv

  27. [35]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  28. [36]

    Mixllm: Llm quantization with global mixed-precision between output-features and highly-efficient system design

    Zhen Zheng, Xiaonan Song, and Chuanjie Liu. Mixllm: Llm quantization with global mixed-precision between output-features and highly-efficient system design. arXiv preprint arXiv:2412.14590, 2024. 12 A Appendix A.1 The analysis of selecting high bit precision For intermediate b...

    2024 arXiv

  29. [37]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Our abstract and introduction include the three main techniques and experi- mental results presented in the paper. Gui...

  30. [38]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The appendix discusses the limitations of our method, including the additional memory overhead introduced by the mask. Guidelines: • The answer NA me...

  31. [39]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 14 Answer: [NA] Justification: We do not have theoretical contributions in this work, where our contributions are validat...

  32. [40]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  33. [41]

    The code will be released soon

    Open access to data and code 15 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The implementation of ou...

  34. [42]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: All parameter settings and ex...

  35. [43]

    The results are fully reproducible by setting the random seed, with negligible variance

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: The performance of all quantized models is evaluat...

  36. [44]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We have provided that i...

  37. [45]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have reviewed that and claim we conform that Code of Ethics. Guidelines: • T...

  38. [46]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There are not direct paths to any negative applications. Guidelines: • The answer NA means that there is n...

  39. [47]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does ...

  40. [48]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We used widely a...

  41. [49]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: No new assets. Guidelines: • The answer NA means that the paper does not release new assets. • Researchers should com...

  42. [50]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  43. [51]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  44. [52]

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.