Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Quantitative Analysis of Performance Drop in DeepSeek Model Quantization

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A dynamic mix of 3-, 4-, and 6-bit weights lets 671B DeepSeek models keep FP8-level scores while fitting on one machine.

desk verdict Useful first full-scale quantization benchmark for DeepSeek-R1/V3 with released DQ3_K_M models, but the 'vs FP8' headline is undercut by uncontrolled API baselines. read the letter →

arxiv 2505.02390 v2 pith:ABD62YPS submitted 2025-05-05 cs.LG cs.AI

Enbo Zhao , Yi Shen , Shuming Shi , Jieyun Huang , Zhihao Chen , Ning Wang , Siqi Xiao , Jian Zhang
show 2 more authors
Kai Wang Shiguo Lian
This is my paper · ORCID
classification cs.LGcs.AI
keywords deepseekquantizationpost-trainingdynamicbit-widthallocation3-bitmixture-of-expertssingle-machinedeploymentmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeepSeek's 671B models are too large for a single 8-GPU machine in FP8, so this technical report asks what happens when they are quantized. It evaluates 2-, 3-, 4-, and 8-bit weight-only post-training quantization across math, code, and general-knowledge benchmarks, reporting that 4-bit quantization loses little accuracy versus FP8 while fitting on standard NVIDIA hardware. The paper then proposes DQ3_K_M, a dynamic 3-bit scheme that assigns higher precision to a few sensitive layers and lower precision elsewhere, shrinking the model to about 281 GB. On DeepSeek-V3-0324, DQ3_K_M achieves near-lossless compression with an average accuracy drop of 0.30% versus FP8, and it matches or beats the uniform 3-bit variant while approaching 4-bit accuracy. This matters because it gives practitioners a concrete memory-accuracy point for local deployment of full-parameter DeepSeek models.

What carries the argument

The central mechanism is dynamic bit-width allocation: a per-module precision schedule that assigns more bits to layers the authors identify as sensitive. In DQ3_K_M, the first two feed-forward down-projection expert layers receive six-bit quantization, every fourth subsequent layer receives four bits, and the remainder receives three bits, yielding a parameter mix of 75.9% 3-bit, 20.7% 4-bit, and 3.4% 6-bit and an effective average of 3.59 bits. The schedule is designed around the observation that down-projection layers concentrate 'super weights' that are disproportionately important, so protecting a few such layers should recover much of the accuracy lost by uniform 3-bit quantization. The mechanism carries the paper's central trade-off: it cuts the 671B model to about 281 GB, smaller than a uniform 4-bit version, while keeping scores close to 4-bit and FP8 levels.

What would settle it

Re-run DQ3_K_M and uniform 4-bit quantization from the same FP8 checkpoints under identical decoding (temperature 0.6, top-p 0.95, same sampling counts), then re-run DQ3_K_M with the six-bit and four-bit layers moved to later positions while keeping the 75.9/20.7/3.4 percent mix unchanged; if permuted schedules score the same, the specific layer choice is not what preserves accuracy, and if matched local FP8 runs shift the baseline, the reported drop percentages must be recalculated.

Watch

Extended reading notes

Core claim

The central discovery reported here is that 4-bit weight-only quantization is a low-loss deployment point for the full DeepSeek R1 and V3 families: across nine math, code, and general-knowledge benchmarks, the Q4_K_M variant stays within about 0.7-1.4% of the FP8 API baseline on average while cutting the 670 GB weight footprint enough to fit an 8-GPU NVIDIA machine. Beyond that, the paper introduces DQ3_K_M, a dynamic 3-bit scheme that outperforms the standard 3-bit variant on DeepSeek-R1 and DeepSeek-V3 and is comparable to 4-bit quantization on most tasks. On DeepSeek-V3-0324, DQ3_K_M achieves near-lossless compression, with an average accuracy drop of 0.30% versus FP8, and it still runs on Huawei Ascend 910B nodes where uniform 4-bit exceeds memory. The paper also reports that uniform 2-bit quantization degrades sharply, especially on knowledge-heavy benchmarks, so the practical frontier it identifies is 3-4 bits with architecture-aware precision allocation.

Load-bearing premise

The load-bearing premise is that the external FP8 API results are a trustworthy in-place baseline and that the DQ3_K_M layer schedule, first two expert down-projection layers at six bits and every fourth layer at four bits, is what produces its advantage, since the schedule is introduced as a heuristic without an ablation study.

Editorial extensions

If this is right

  • Standard 4-bit quantization becomes the default deployment point for full-parameter DeepSeek on 8-GPU NVIDIA nodes: near-FP8 scores at roughly 377 GB of weights instead of more than 670 GB.
  • DQ3_K_M provides a better memory-accuracy option than uniform 3-bit: adding a few sensitive high-precision layers recovers most of the accuracy gap to 4-bit at 281 GB.
  • Huawei Ascend 910B and similar 64 GB-per-device machines can host DQ3_K_M, which the paper argues is not true for uniform 4-bit quantization.
  • Very low-bit uniform quantization at 2 bits is not a practical point for these models, with large drops on C-Eval and other knowledge tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the schedule's layer choice is doing the work, similar gains should appear on other large mixture-of-experts models; the paper does not test that transfer, so it remains an open prediction.
  • A matched local FP8 baseline under the same decoding could shift the reported drop percentages by fractions of a point, but it would not change the ordering unless the schedule ablation fails.
  • Combining dynamic weight quantization with quantized KV caches or activations is the natural next step, since this report only quantizes weights.
  • The 3.59 average-bit model sits slightly below the nominal '3-bit' label, so the useful takeaway is the actual size-accuracy curve rather than the integer bit width.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This technical report evaluates post-training quantization of DeepSeek-R1, DeepSeek-V3, DeepSeek-V3-0324, and DeepSeek-R1-distill-Qwen-32B across 2, 3, 4, and 8-bit configurations on reasoning and general-knowledge benchmarks. The authors find that 4-bit quantization retains most performance relative to FP8 and propose DQ3_K_M, a hybrid 3-bit scheme that applies q6_k to early ffn_down_exps layers and inserts q4_k every fourth layer. They report that DQ3_K_M outperforms llama.cpp's Q3_K_M and matches Q4_K_M on many tasks, with a claimed near-lossless average drop of 0.30% on DeepSeek-V3-0324, and they release the quantized models. The paper includes resource-consumption tables and deployment recommendations for NVIDIA and Huawei hardware.

Significance. If the claims are established, the paper provides a useful practical contribution: a large-scale, multi-bitwidth quantization study of 671B-parameter models, an open-sourced 3-bit deployment recipe, and a concrete demonstration that dynamic precision allocation can improve over uniform 3-bit quantization. The authors are transparent about using local harnesses for quantized models and report standard deviations for sampled benchmarks. However, the central comparison against FP8 rests on externally obtained API baselines with unstated decoding protocols, and the DQ3_K_M schedule is a hand-selected allocation without any ablation or selection protocol. These issues directly affect the headline claims and need to be resolved before the conclusions can be accepted.

major comments (3)
  1. [Section 4.2, Section 4.3, Tables 2-4] The FP8 baselines used for the 'Accuracy drop' rows are not measured under the same decoding configuration as the quantized models. Section 4.2 specifies generation settings (max 32,768 tokens, temperature 0.6, top-p 0.95, repeated sampling for small benchmarks) only for 'all quantized models'; the FP8 columns are taken from DeepSeek's reported numbers or from external APIs (DeepSeek Official API, Tencent API) with no stated generation settings. Moreover, Section 4.3 states that the V3 baseline was switched to Tencent's API because the official API changed, so the reference model version is not constant across tables. Consequently, the accuracy-drop numbers, including the 0.30% near-lossless result for DQ3_K_M on V3-0324 in Table 4, conflate quantization loss with API version and decoding choices. This undermines the abstract's claim of 'little performance degradation versus FP8.'
  2. [Section 3, Appendix A.1] The DQ3_K_M precision schedule—q6_k on the first two ffn_down_exps layers, q3_k with q4_k inserted every fourth layer, yielding a 75.9/20.7/3.4% split—is presented without an ablation, a search procedure, or a validation protocol. Because the same benchmarks are used both to motivate the design and to report the final comparison, the advantage over Q3_K_M could reflect benchmark-time selection rather than a general design principle. The paper needs at least an ablation of the schedule choices or an explicit description of how the hyperparameters were selected and on which data.
  3. [Tables 2-4, Table 8] The 'Accuracy drop' metric is not computed consistently across the tables. In Table 2, the reported drop of 0.34% for DQ3_K_M matches the Weighted avg column (85.82 vs. 85.53), not the Average column (83.48 vs. 83.03, which would give about 0.54%). In Tables 3 and 4, the drops match the unweighted Average column. Furthermore, the Table 4 Q2_K_L drop of 14.66% does not match the arithmetic of the Average column (approximately 14.06%). Since the abstract relies on these drops, the paper should define the metric unambiguously and verify all computed values; a sensitivity analysis over the hand-assigned weights in Table 8 would also be appropriate.
minor comments (5)
  1. [Section 4.3] The sentence 'we substituted it with Tencent's DeepSeek V3 API in Table 2' appears to refer to Table 3, not Table 2; please correct the cross-reference.
  2. [Conclusion] The phrase 'deployment on single a single machine with 8 GPU devices' contains a duplicated article and should read 'deployment on a single machine with 8 GPU devices.'
  3. [Throughout] There are several typographical errors: 'achievied' should be 'achieved', 'assorted to' should likely be 'resorted to', and 'weighted-only PTQ' should be 'weights-only PTQ.'
  4. [Table 4] The header 'Q4 K Q3 K' for the last two columns is ambiguous; please label these columns explicitly as the fully quantized Q4_K and Q3_K variants described in Section 4.3.
  5. [Appendix A.2] The benchmark weighting scheme in Table 8 is introduced without justification; a sentence explaining the rationale for these weights, or an ablation showing that conclusions are insensitive to them, would strengthen the report.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the report is an empirical benchmark comparison against external baselines; DQ3_K_M is a hand-specified mixed-precision schedule, not a derived prediction, and no load-bearing step reduces to its own inputs.

full rationale

The paper makes no first-principles derivation. Its central method, DQ3_K_M, is presented as a heuristic mixed-precision configuration: 'Building upon the standard Q3 quantization provided in llama.cpp, we implement hybrid precision by applying q6_k or q4_k quantization to some selected modules' (Section 3). The reported claims are benchmark measurements of that configuration against external references (llama.cpp's Q3_K_M and Q4_K_M, Unsloth's UD-Q2_K_XL, and FP8 numbers from DeepSeek or Tencent APIs). No equation in the paper defines the DQ3 schedule in terms of the evaluation outcomes, and no fitted parameter is renamed as a prediction. The accuracy-drop tables are simple relative comparisons to externally sourced FP8 results, not outputs of the method itself. The absence of an ablation for the DQ3 layer schedule and the use of external APIs as FP8 baselines are methodological limitations that affect evidentiary strength, but they do not make the derivation circular: the comparison targets are independent of the method's construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no re-labeling of a known result as a new one. Therefore no circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims depend on unpaid assumptions: a hand-selected precision layout and benchmark weights, comparability of API FP8 baselines with local quantized runs, and transfer of the super-weight heuristic to DeepSeek architectures.

free parameters (2)
  • DQ3_K_M precision schedule = 75.9% q3_k, 20.7% q4_k, 3.4% q6_k; q6_k on first two ffn_down_exps; q4_k every fourth layer
    Hand-chosen hybrid-precision layout, no tuning protocol or ablation; effectiveness claims on benchmarks depend on this selection.
  • Benchmark weighting scheme = AIME 0.2; MATH/GPQA/MBPP/MBPP+/LiveCodeBench 0.5; MMLU/CMMLU/C-Eval 1.0
    Table 8 defines weights used to compute weighted averages and accuracy drops; no justification is given and the choice affects headline accuracy-drop percentages.
assumptions (3)
  • domain assumption External API FP8 scores are comparable to locally evaluated quantized scores.
    Tables 2-4 take FP8 baselines from the DeepSeek official API or Tencent API and compute accuracy drops against local runs, but decoding settings for the API are not shown and may differ.
  • domain assumption Super weights concentrated in ffn_down_proj layers (Yu et al.) are present in DeepSeek-R1/V3 and justify hybrid precision.
    Section 3 uses this to place q6_k on the first two ffn_down_exps layers; if the phenomenon does not transfer, the DQ3_K_M design basis fails.
  • domain assumption Weights-only PTQ is sufficient to evaluate the performance of quantized DeepSeek models.
    Section 2.2 restricts all evaluations to weights-only PTQ; activation quantization effects are not measured, so conclusions apply only to that paradigm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantitative Analysis of Performance Drop in DeepSeek Model Quantization." pith.science (2026). https://pith.science/paper/ABD62YPS

@misc{pith2026250502390,
  author       = {Pith},
  title        = {Pith review of: Quantitative Analysis of Performance Drop in DeepSeek Model Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ABD62YPS}},
  note         = {Machine review of arXiv:2505.02390}
}
read the original abstract

Recently, there is a high demand for deploying DeepSeek-R1 and V3 locally, possibly because the official service often suffers from being busy and some organizations have data privacy concerns. While single-machine deployment offers infrastructure simplicity, the models' 671B FP8 parameter configuration exceeds the practical memory limits of a standard 8-GPU machine. Quantization is a widely used technique that helps reduce model memory consumption. However, it is unclear what the performance of DeepSeek-R1 and V3 will be after being quantized. This technical report presents the first quantitative evaluation of multi-bitwidth quantization across the complete DeepSeek model spectrum. Key findings reveal that 4-bit quantization maintains little performance degradation versus FP8 while enabling single-machine deployment on standard NVIDIA GPU devices. We further propose DQ3_K_M, a dynamic 3-bit quantization method that significantly outperforms traditional Q3_K_M variant on various benchmarks, which is also comparable with 4-bit quantization (Q4_K_M) approach in most tasks. Moreover, DQ3_K_M supports single-machine deployment configurations for both NVIDIA H100/A100 and Huawei 910B. Our implementation of DQ3\_K\_M is released at https://github.com/UnicomAI/DeepSeek-Eval, containing optimized 3-bit quantized variants of both DeepSeek-R1 and DeepSeek-V3.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 linked inside Pith

  1. [4]

    Quantizing large language models for code generation: A differentiated replication.arXiv preprint arXiv:2503.07103,

    Alessandro Giagnorio, Antonio Mastropaolo, Saima Afrin, Massimiliano Di Penta, and Gabriele Bavota. Quantizing large language models for code generation: A differentiated replication.arXiv preprint arXiv:2503.07103,

    arXiv

  2. [6]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    arXiv 2009

  3. [8]

    O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?arXiv preprint arXiv:2411.16489,

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?arXiv preprint arXiv:2411.16489,

    arXiv

  4. [10]

    Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212,

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212,

    arXiv

  5. [11]

    Quantization meets reasoning: Exploring llm low-bit quantization degradation for mathematical reasoning.arXiv preprint arXiv:2501.03035,

    Zhen Li, Yupeng Su, Runming Yang, Congkai Xie, Zheng Wang, Zhongwei Xie, Ngai Wong, and Hongxia Yang. Quantization meets reasoning: Exploring llm low-bit quantization degradation for mathematical reasoning.arXiv preprint arXiv:2501.03035,

    arXiv

  6. [12]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  7. [13]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems, 36:21558–21572, 2023a. 11 Preprint. Technical Report. Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Y...

    arXiv

  8. [14]

    Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023b

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023b. MAA. American invitational mathematics examination - aime. InAmerican Invita- tional Mathematics Exa...

    arXiv 2024

Show all 21 references
  1. [15]

    Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al

    URL https://maa.org/ math-competitions/american-invitational-mathematics-examination-aime. Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction re...

    arXiv

  2. [16]

    Evaluating quantized large language models for code generation on low-resource language benchmarks.arXiv preprint arXiv:2410.14766,

    Enkhbold Nyamsuren. Evaluating quantized large language models for code generation on low-resource language benchmarks.arXiv preprint arXiv:2410.14766,

    arXiv

  3. [17]

    O1 replication journey: A strategic progress report– part 1.arXiv preprint arXiv:2410.18982,

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report– part 1.arXiv preprint arXiv:2410.18982,

    arXiv

  4. [18]

    Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550,

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550,

    arXiv

  5. [20]

    A survey on knowledge distillation of large language models.arXiv preprint arXiv:2402.13116,

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models.arXiv preprint arXiv:2402.13116,

    arXiv

  6. [21]

    The super weight in large language models

    Mengxia Yu, De Wang, Qi Shan, and Alvin Wan. The super weight in large language models. arXiv preprint arXiv:2411.07191,

    arXiv

  7. [2014]

    Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge

    Xuan Shen, Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, Cheng Lyu, Chih-hsiang Li, Xuehang Guo, Zhihao Shu, et al. Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge. arXiv preprint arXiv...

    arXiv

  8. [2019]

    Technical Report

    10 Preprint. Technical Report. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    arXiv

  9. [2021]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  10. [2022]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    arXiv

  11. [2023]

    Learned step size quantization.arXiv preprint arXiv:1902.08153,

    Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization.arXiv preprint arXiv:1902.08153,

    1902 arXiv

  12. [2024]

    Livecodebench: Holistic and contami- nation free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    arXiv

  13. [2025]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.