{"id":"7f816d48-f739-48eb-937b-37a4d7f4315f","arxiv_id":"2505.02390","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hand-tuned 3-bit quantization scheme, DQ3_K_M, matches 4-bit accuracy on DeepSeek-R1 and V3 while shrinking the models enough for single-machine deployment.","lead":"This report measures how much performance DeepSeek's largest AI models lose when compressed to 4-bit, 3-bit, and 2-bit formats, and proposes a custom 3-bit scheme called DQ3_K_M that roughly matches 4-bit accuracy while using less memory. A generalist should read it because it shows whether 671-billion-parameter reasoning models can run on a single eight-GPU machine with acceptable quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'little degradation vs FP8' and DQ3_K_M's near-lossless claim are not yet established: FP8 baselines come from external APIs with no stated decoding protocol, and the DQ3 schedule has no ablation.","rationale":"The reader's conditional verdict is appropriate. I read the paper as an engineering measurement report, not a theoretical claim; the open-sourced DQ3_K_M weights and the detailed per-module quantization table in Appendix A.1 are useful. The central claims are empirical, so controlled baselines are essential. The strongest concern is that the FP8 reference is not measured under the same protocol. The paper itself signals this risk by replacing the DeepSeek V3 API with Tencent's API after a model update; if external APIs are the only FP8 reference, the 'near-lossless' claim cannot be separated from serving-configuration effects. The DQ3 schedule is also unablated, but that primarily affects the generality and optimality of the proposed heuristic, not the raw comparison against Q3_K_M under identical local decoding. A single controlled FP8 rerun would settle the more serious issue. If the authors report API settings and reproduce the scores, the conditional verdict can stand; if the FP8 baseline moves substantially, the headline conclusions need revision.","tokens_in":11484,"tokens_out":7357,"duration_ms":92219,"concrete_test":"Re-evaluate DeepSeek-R1 and DeepSeek-V3-0324 FP8 through the official API using exactly the Section 4.2 decoding configuration: temperature 0.6, top-p 0.95, max 32,768 tokens, four samples for MATH-500/GPQA/LiveCodeBench, eight for AIME, and a single pass for MMLU/CMMLU/C-Eval, then recompute the accuracy-drop rows in Tables 2 and 4. If the FP8 averages shift by more than about one point, the 'little degradation' and 0.30% near-lossless claims are artifacts of the uncontrolled baseline; if the drops stay within the reported gaps, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the FP8 columns in Tables 2-4 are directly comparable to the local quantized runs. Section 4.2 specifies decoding settings (max 32,768 tokens, temperature 0.6, top-p 0.95, four samples for MATH/GPQA/LiveCodeBench, eight for AIME, single pass for MMLU/CMMLU/C-Eval) for 'all quantized models' only. The FP8 baselines are either numbers copied from DeepSeek's reports or calls to external APIs (DeepSeek Official API, Tencent API), with no stated generation settings. The paper even switches the V3 baseline to Tencent's API because the official API changed (Section 4.3 footnote), so the reference model variant is not constant. Thus every 'accuracy drop' row, including the 0.30% near-lossless DQ3_K_M result on V3-0324 in Table 4, confounds quantization loss with API version and decoding choices. Relatedly, the DQ3_K_M schedule (q6_k on the first two ffn_down_exps layers, q4_k inserted every fourth layer, Section 3 and Appendix A.1) is a hand-picked allocation with no ablation or selection protocol; its advantage over Q3_K_M could reflect benchmark-time selection rather than a general design principle. The FP8 comparability issue is the more load-bearing because it supports the abstract's central deployment claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report evaluates post-training quantization of DeepSeek-R1, DeepSeek-V3, DeepSeek-V3-0324, and DeepSeek-R1-distill-Qwen-32B across 2, 3, 4, and 8-bit configurations on reasoning and general-knowledge benchmarks. The authors find that 4-bit quantization retains most performance relative to FP8 and propose DQ3_K_M, a hybrid 3-bit scheme that applies q6_k to early ffn_down_exps layers and inserts q4_k every fourth layer. They report that DQ3_K_M outperforms llama.cpp's Q3_K_M and matches Q4_K_M on many tasks, with a claimed near-lossless average drop of 0.30% on DeepSeek-V3-0324, and they release the quantized models. The paper includes resource-consumption tables and deployment recommendations for NVIDIA and Huawei hardware.","tokens_in":11820,"tokens_out":5422,"duration_ms":57021,"significance":"If the claims are established, the paper provides a useful practical contribution: a large-scale, multi-bitwidth quantization study of 671B-parameter models, an open-sourced 3-bit deployment recipe, and a concrete demonstration that dynamic precision allocation can improve over uniform 3-bit quantization. The authors are transparent about using local harnesses for quantized models and report standard deviations for sampled benchmarks. However, the central comparison against FP8 rests on externally obtained API baselines with unstated decoding protocols, and the DQ3_K_M schedule is a hand-selected allocation without any ablation or selection protocol. These issues directly affect the headline claims and need to be resolved before the conclusions can be accepted.","major_comments":[{"comment":"The FP8 baselines used for the 'Accuracy drop' rows are not measured under the same decoding configuration as the quantized models. Section 4.2 specifies generation settings (max 32,768 tokens, temperature 0.6, top-p 0.95, repeated sampling for small benchmarks) only for 'all quantized models'; the FP8 columns are taken from DeepSeek's reported numbers or from external APIs (DeepSeek Official API, Tencent API) with no stated generation settings. Moreover, Section 4.3 states that the V3 baseline was switched to Tencent's API because the official API changed, so the reference model version is not constant across tables. Consequently, the accuracy-drop numbers, including the 0.30% near-lossless result for DQ3_K_M on V3-0324 in Table 4, conflate quantization loss with API version and decoding choices. This undermines the abstract's claim of 'little performance degradation versus FP8.'","section":"Section 4.2, Section 4.3, Tables 2-4"},{"comment":"The DQ3_K_M precision schedule—q6_k on the first two ffn_down_exps layers, q3_k with q4_k inserted every fourth layer, yielding a 75.9/20.7/3.4% split—is presented without an ablation, a search procedure, or a validation protocol. Because the same benchmarks are used both to motivate the design and to report the final comparison, the advantage over Q3_K_M could reflect benchmark-time selection rather than a general design principle. The paper needs at least an ablation of the schedule choices or an explicit description of how the hyperparameters were selected and on which data.","section":"Section 3, Appendix A.1"},{"comment":"The 'Accuracy drop' metric is not computed consistently across the tables. In Table 2, the reported drop of 0.34% for DQ3_K_M matches the Weighted avg column (85.82 vs. 85.53), not the Average column (83.48 vs. 83.03, which would give about 0.54%). In Tables 3 and 4, the drops match the unweighted Average column. Furthermore, the Table 4 Q2_K_L drop of 14.66% does not match the arithmetic of the Average column (approximately 14.06%). Since the abstract relies on these drops, the paper should define the metric unambiguously and verify all computed values; a sensitivity analysis over the hand-assigned weights in Table 8 would also be appropriate.","section":"Tables 2-4, Table 8"}],"minor_comments":[{"comment":"The sentence 'we substituted it with Tencent's DeepSeek V3 API in Table 2' appears to refer to Table 3, not Table 2; please correct the cross-reference.","section":"Section 4.3"},{"comment":"The phrase 'deployment on single a single machine with 8 GPU devices' contains a duplicated article and should read 'deployment on a single machine with 8 GPU devices.'","section":"Conclusion"},{"comment":"There are several typographical errors: 'achievied' should be 'achieved', 'assorted to' should likely be 'resorted to', and 'weighted-only PTQ' should be 'weights-only PTQ.'","section":"Throughout"},{"comment":"The header 'Q4 K Q3 K' for the last two columns is ambiguous; please label these columns explicitly as the fully quantized Q4_K and Q3_K variants described in Section 4.3.","section":"Table 4"},{"comment":"The benchmark weighting scheme in Table 8 is introduced without justification; a sentence explaining the rationale for these weights, or an ablation showing that conclusions are insensitive to them, would strengthen the report.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The external-API baseline issue is the main correctness risk: the paper's central deployment claim depends on comparing local quantized runs to FP8 numbers obtained under unspecified and possibly varying conditions. The DQ3_K_M schedule also needs an ablation or a clear tuning protocol. Both are fixable within the scope of a technical report, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The most useful thing in this report is the first multi-bitwidth sweep over the 671B DeepSeek-R1/V3 family, with the quantized weights released. The DQ3_K_M variant is a sensible hybrid (q6_k on first two ffn_down_exps layers, q4_k every fourth layer, 75.9/20.7/3.4 split) and the local comparisons show it beating llama.cpp's Q3_K_M and roughly matching Q4_K_M. If you deploy these models, the memory/accuracy tables are worth reading.\n\nThe main soft spot is exactly where the headline lives: the 'vs FP8' columns are not controlled. Tables 2–4 mix DeepSeek-reported numbers, official API calls, and for V3 a Tencent API substitute after the official API changed. No decoding protocol is given for any of those baselines, while the local quantized runs use temperature 0.6, top-p 0.95, and multi-sample means on reasoning tasks. That confounds quantization loss with API version and sampling choices. The 0.30% near-lossless claim on V3-0324 is therefore not yet established, though the Q4_K_M vs DQ3_K_M local comparison does support the 'comparable to 4-bit' claim.\n\nThe DQ3_K_M schedule is also hand-picked with no ablation or tuning protocol, so some of the observed advantage over Q3_K_M could be selection. That is a minor-to-moderate concern because the same schedule works across R1, V3, and V3-0324 without further per-model tuning, which gives it some plausibility as a general heuristic.\n\nMinor issues: MMLU/CMMLU/C-Eval are single-pass with no error bars; the 'Accuracy drop' row in Table 3 seems garbled (a '-' and '0' as values); and the report calls itself the 'first' full-scale evaluation without acknowledging Unsloth's dynamic quantization work on the same models, though it does cite the blog.\n\nOverall: this is a useful engineering study with honest, mostly internally consistent local measurements and released artifacts. The central deployment conclusion—DQ3_K_M enables 3-bit deployment on a single 8-GPU box at near-Q4 accuracy—holds up. The headline 'little degradation versus FP8' needs stronger baselines before I'd repeat it.\n\nRecommendation: send it to peer review as an empirical/engineering paper. It deserves referee time, with the caveat that the FP8 comparison must be cleaned up or toned down.","headline":"Useful first full-scale quantization benchmark for DeepSeek-R1/V3 with released DQ3_K_M models, but the 'vs FP8' headline is undercut by uncontrolled API baselines.","tokens_in":12357,"tokens_out":2427,"would_cite":true,"duration_ms":26337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dynamic mix of 3-, 4-, and 6-bit weights lets 671B DeepSeek models keep FP8-level scores while fitting on one machine.","keywords":["deepseek","quantization","post-training quantization","dynamic bit-width allocation","3-bit quantization","mixture-of-experts","single-machine deployment","model compression"],"falsifier":"Re-run DQ3_K_M and uniform 4-bit quantization from the same FP8 checkpoints under identical decoding (temperature 0.6, top-p 0.95, same sampling counts), then re-run DQ3_K_M with the six-bit and four-bit layers moved to later positions while keeping the 75.9/20.7/3.4 percent mix unchanged; if permuted schedules score the same, the specific layer choice is not what preserves accuracy, and if matched local FP8 runs shift the baseline, the reported drop percentages must be recalculated.","tokens_in":11336,"feed_emoji":"🧠","tokens_out":8582,"duration_ms":92032,"temperature":0.7,"pith_summary":"DeepSeek's 671B models are too large for a single 8-GPU machine in FP8, so this technical report asks what happens when they are quantized. It evaluates 2-, 3-, 4-, and 8-bit weight-only post-training quantization across math, code, and general-knowledge benchmarks, reporting that 4-bit quantization loses little accuracy versus FP8 while fitting on standard NVIDIA hardware. The paper then proposes DQ3_K_M, a dynamic 3-bit scheme that assigns higher precision to a few sensitive layers and lower precision elsewhere, shrinking the model to about 281 GB. On DeepSeek-V3-0324, DQ3_K_M achieves near-lossless compression with an average accuracy drop of 0.30% versus FP8, and it matches or beats the uniform 3-bit variant while approaching 4-bit accuracy. This matters because it gives practitioners a concrete memory-accuracy point for local deployment of full-parameter DeepSeek models.","feed_headline":"A 3-bit recipe keeps 671B DeepSeek near FP8 accuracy","feed_subtitle":"A dynamic layer mix at 281 GB scores within 0.3 percent of FP8 on V3-0324 and beats uniform 3-bit.","key_machinery":"The central mechanism is dynamic bit-width allocation: a per-module precision schedule that assigns more bits to layers the authors identify as sensitive. In DQ3_K_M, the first two feed-forward down-projection expert layers receive six-bit quantization, every fourth subsequent layer receives four bits, and the remainder receives three bits, yielding a parameter mix of 75.9% 3-bit, 20.7% 4-bit, and 3.4% 6-bit and an effective average of 3.59 bits. The schedule is designed around the observation that down-projection layers concentrate 'super weights' that are disproportionately important, so protecting a few such layers should recover much of the accuracy lost by uniform 3-bit quantization. The mechanism carries the paper's central trade-off: it cuts the 671B model to about 281 GB, smaller than a uniform 4-bit version, while keeping scores close to 4-bit and FP8 levels.","core_discovery":"The central discovery reported here is that 4-bit weight-only quantization is a low-loss deployment point for the full DeepSeek R1 and V3 families: across nine math, code, and general-knowledge benchmarks, the Q4_K_M variant stays within about 0.7-1.4% of the FP8 API baseline on average while cutting the 670 GB weight footprint enough to fit an 8-GPU NVIDIA machine. Beyond that, the paper introduces DQ3_K_M, a dynamic 3-bit scheme that outperforms the standard 3-bit variant on DeepSeek-R1 and DeepSeek-V3 and is comparable to 4-bit quantization on most tasks. On DeepSeek-V3-0324, DQ3_K_M achieves near-lossless compression, with an average accuracy drop of 0.30% versus FP8, and it still runs on Huawei Ascend 910B nodes where uniform 4-bit exceeds memory. The paper also reports that uniform 2-bit quantization degrades sharply, especially on knowledge-heavy benchmarks, so the practical frontier it identifies is 3-4 bits with architecture-aware precision allocation.","pith_inferences":["If the schedule's layer choice is doing the work, similar gains should appear on other large mixture-of-experts models; the paper does not test that transfer, so it remains an open prediction.","A matched local FP8 baseline under the same decoding could shift the reported drop percentages by fractions of a point, but it would not change the ordering unless the schedule ablation fails.","Combining dynamic weight quantization with quantized KV caches or activations is the natural next step, since this report only quantizes weights.","The 3.59 average-bit model sits slightly below the nominal '3-bit' label, so the useful takeaway is the actual size-accuracy curve rather than the integer bit width."],"forward_implications":["Standard 4-bit quantization becomes the default deployment point for full-parameter DeepSeek on 8-GPU NVIDIA nodes: near-FP8 scores at roughly 377 GB of weights instead of more than 670 GB.","DQ3_K_M provides a better memory-accuracy option than uniform 3-bit: adding a few sensitive high-precision layers recovers most of the accuracy gap to 4-bit at 281 GB.","Huawei Ascend 910B and similar 64 GB-per-device machines can host DQ3_K_M, which the paper argues is not true for uniform 4-bit quantization.","Very low-bit uniform quantization at 2 bits is not a practical point for these models, with large drops on C-Eval and other knowledge tasks."],"supporting_citations":[{"why":"Defines DeepSeek-V3's architecture, including the mixture-of-experts feed-forward layers targeted by the DQ3_K_M precision schedule.","marker":"Liu et al., 2024"},{"why":"Defines DeepSeek-R1 and supplies the official FP8 benchmark scores used as the reference baseline for accuracy-drop calculations.","marker":"Guo et al., 2025"},{"why":"Reports 'super weights' concentrated in MLP down-projection layers, motivating the high-precision allocation in DQ3_K_M.","marker":"Yu et al., 2024"},{"why":"Establishes weights-only post-training quantization as the baseline paradigm that the report's evaluations follow.","marker":"Frantar et al., 2022"},{"why":"Provides SpQR, a weights-only PTQ method whose ideas about preserving important weight structure inform dynamic precision selection.","marker":"Dettmers et al., 2023"}],"fun_headline_variants":["Dynamic 3-bit quantization matches 4-bit on DeepSeek","Near-lossless 3-bit: DeepSeek drops only 0.3% vs FP8","3-bit beats uniform, rivals 4-bit on 671B DeepSeek","Quantizing DeepSeek: 4-bit fits, 3-bit dynamic wins","DQ3_K_M: 3-bit DeepSeek within 0.3% of FP8"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the external FP8 API results are a trustworthy in-place baseline and that the DQ3_K_M layer schedule, first two expert down-projection layers at six bits and every fourth layer at four bits, is what produces its advantage, since the schedule is introduced as a heuristic without an ablation study.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic 3-bit quantization matches 4-bit on DeepSeek","Near-lossless 3-bit: DeepSeek drops only 0.3% vs FP8","3-bit beats uniform, rivals 4-bit on 671B DeepSeek","Quantizing DeepSeek: 4-bit fits, 3-bit dynamic wins","DQ3_K_M: 3-bit DeepSeek within 0.3% of FP8"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":2020,"prompt_tokens":1038,"completion_tokens":982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":871}},"tokens_in":654,"tokens_out":982,"duration_ms":10073,"temperature":1.0,"reasoning_tokens":871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:52:37.339020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run DQ3_K_M and uniform 4-bit quantization from the same FP8 checkpoints under identical decoding (temperature 0.6, top-p 0.95, same sampling counts), then re-run DQ3_K_M with the six-bit and four-bit layers moved to later positions while keeping the 75.9/20.7/3.4 percent mix unchanged; if permuted schedules score the same, the specific layer choice is not what preserves accuracy, and if matched local FP8 runs shift the baseline, the reported drop percentages must be recalculated.","supporting_citations":[],"review_version":1}