{"id":"b56c952a-f5fa-4dc7-8ebc-94329b4c7916","arxiv_id":"1908.04680","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Two-stage and gradually decreasing quantization, stochastic precision sampling, and joint teacher-student distillation each improve low-bit CNN accuracy on ImageNet and CIFAR-100, with the largest gains when combined.","lead":"This paper offers three training techniques that make low-bitwidth neural networks, which use very few bits per number, more accurate on image classification. If the gains hold, low-precision networks become more practical for phones and embedded devices without the usual accuracy penalty.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SP and combined SP+KD results use a 40-epoch schedule while baselines use 30 epochs, confounding the reported gains.","rationale":"The reader's verdict of CONDITIONAL is appropriate, but the most load-bearing issue I find is not the general sensitivity to hyperparameters or STE breakdown; it is an unequal training budget in the experiments that produce the headline numbers. In Sec. 4.2, SP experiments explicitly train for 40 epochs, while the default ImageNet protocol in Sec. 4 is 30 epochs. The baselines in Tables 2 and 9 are the standard 30-epoch numbers, and the paper reports no 40-epoch baseline. Because longer fine-tuning with later learning-rate decays typically improves quantized models, the apparent SP and SP+KD gains may be inflated. The KD-only comparisons in Table 4 use matched 30-epoch schedules and are less suspect, which is why I do not recommend rejection. However, the central claimed 73.25% versus 70.19% improvement requires a controlled comparison. This is a concrete, testable concern rather than a general appeal to hyperparameter uncertainty. I partially disagree with the reader's weakest assumption: hyperparameter sensitivity is a secondary issue; the training-budget mismatch is more immediate and more damaging to the headline claim. The recommended verdict remains CONDITIONAL: the paper should either report matched-budget baselines or explicitly justify why the 40-epoch SP schedule is an integral part of the method rather than an uncontrolled variable.","tokens_in":20906,"tokens_out":5575,"duration_ms":53583,"concrete_test":"Rerun the DoReFa-Net baseline (2-bit ResNet-50 on ImageNet, and at least one or two other baselines from Tables 2 and 9) with exactly the SP training schedule (40 epochs, learning-rate decay at epochs 25 and 35, same batch size and optimizer) but without any SP or KD modifications. If the 40-epoch baseline reaches roughly 72.2% to 73.2% top-1 accuracy, the reported SP and SP+KD gains are largely a training-budget artifact. If it stays near 70.2%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 (Effect of stochastic precision) sets a 40-epoch training schedule for SP experiments: the stochastic ratio is linearly decayed to 0 at the 20-th epoch, and the learning rate is decayed at the 25-th and 35-th epochs. In contrast, the general implementation details in Sec. 4 state that ImageNet experiments 'train a maximum 30 epochs, and decay the learning rate by 10 at the 15-th and 25-th epoch.' The baseline numbers in Tables 2 and 9 (e.g., DoReFa-Net 2-bit ResNet-50 at 70.19) are identical to those in Table 4, which uses the default 30-epoch schedule. No re-run of the baseline with the 40-epoch SP schedule is reported. Thus the improvement attributed to SP (70.19 to 72.23 in Table 2) and to SP+KD (70.19 to 73.25 in Table 9) may be partly or entirely due to the additional 10 epochs of training and the later learning-rate decays, rather than to the proposed stochastic-precision mechanism. The central claim that 'all our low-precision models surpass the corresponding baselines' is therefore not established on a like-for-like training budget for the SP and combined methods.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three training strategies for low-bitwidth CNNs: (i) progressive quantization, with a two-stage scheme (quantize weights first, then activations) and a progressive precision scheme (32→8→4→2 bits during training); (ii) stochastic precision (SP), which randomly quantizes fragments of the network while keeping the rest full-precision, with the stochastic ratio decaying to zero; and (iii) joint knowledge distillation (KD), in which a full-precision teacher and a low-precision student are trained together using both KL-divergence and attention-transfer losses. Experiments on CIFAR-100 and ImageNet with DoReFa-Net, LQ-Net, BiReal-Net, and GroupNet report consistent accuracy improvements over the corresponding baselines, culminating in combined SP+KD results such as 73.25% top-1 for 2-bit ResNet-50 versus a 70.19% baseline.","tokens_in":21178,"tokens_out":2250,"duration_ms":23212,"significance":"If the reported gains hold under controlled comparisons, the paper would make a useful practical contribution: it offers simple, quantizer-agnostic training recipes that improve low-bitword accuracy across several architectures and quantization schemes, and the combination of stochastic precision with joint distillation appears complementary rather than redundant. The paper also ships a fairly large set of experiments with repeated runs and standard deviations for most of the headline tables, which is a strength. However, the central quantitative claim is weakened by a training-budget confound in the SP and SP+KD experiments, by apparent typographical inconsistencies in Table 1, and by the absence of sensitivity analysis for the key hyperparameters (δ, β, γ, and the precision schedule). The contribution is incremental over the authors' CVPR 2018 conference paper, but the new one-stage SP formulation and the joint-teacher distillation study constitute a reasonable extension for a journal version.","major_comments":[{"comment":"The SP experiments use a 40-epoch schedule (learning-rate decays at epochs 25 and 35), while the baselines are trained for 30 epochs with decays at epochs 15 and 25, as stated in Sec. 4. The baseline numbers in Table 2 and Table 9 are identical to those in Table 4, which uses the default 30-epoch schedule. No baseline re-run under the 40-epoch schedule is reported, so the gains attributed to SP (e.g., 70.19→72.23 in Table 2) and to SP+KD (70.19→73.25 in Table 9) are confounded by the additional 10 epochs of training and the later learning-rate decays. The authors should either re-run the baselines under the SP schedule or provide evidence, such as a 40-epoch baseline curve, that the extra epochs alone do not explain the improvement. Without this, the central claim that 'all our low-precision models surpass the corresponding baselines' is not established on a like-for-like training budget.","section":"Sec. 4.2 and Table 2"},{"comment":"The top-5 accuracy entries for the 4W,4A ResNet-50* row are internally inconsistent: the Baseline is listed as 75.70, which is below the Top-1 value of 75.11, and is much lower than the Baseline+TS value of 91.93. Similarly, the 2W,2A ResNet-50* Baseline top-5 is 70.00, which is only slightly above the top-1 of 67.68 and inconsistent with the other methods' top-5 values (86.90–87.03). These entries appear to contain typographical errors, but they also affect the comparison: the reported gains of TS, PP, and TS+PP over Baseline in these rows are partly an artifact of incorrect baseline numbers. The authors should correct these values and state whether the corrected numbers change the conclusions in Sec. 4.1.","section":"Table 1"},{"comment":"The ablation study in Table 8 shows that attention transfer alone gives 71.51 and posterior alone gives 71.40 for 2-bit ResNet-50, while joint gives 71.96; however, Tables 4 and 9 report 'with ResNet-50' values of 71.96 and 73.25, respectively. The difference between Table 4 (joint KD only) and Table 9 (SP+KD) is 1.29 points, which is plausible, but the paper does not provide an ablation isolating the contribution of SP from the contribution of the longer 40-epoch schedule in the SP+KD setting (see the first major comment). Additionally, no sensitivity analysis is reported for β=0.5, γ=50, α_2=0.5, or the stochastic-ratio decay from 0.5, even though these parameters directly control the size of the reported gains. The authors should report at least a small grid over β and γ, or a single variation of δ, to demonstrate that the improvements are not specific to one hand-tuned configuration.","section":"Sec. 4.3.1 and Tables 4, 5, 9"},{"comment":"The claim that the jointly trained full-precision teacher can improve over the pretrained baseline is based on a comparison with the '32W, 32A' row in Table 4 (e.g., ResNet-50 teacher top-1 of 75.64 baseline versus the 'with ResNet-50' student column). However, the table reports the teacher's accuracy only in the first two rows and does not show the jointly trained teacher's accuracy after training; Figure 5 is a convergence plot without final numerical values. The statement that 'the performance of the full-precision model can be slightly improved in some cases' is therefore not directly supported by a table entry. The authors should report the final teacher accuracy for the joint-training runs, or soften the claim if the evidence is only qualitative.","section":"Sec. 4.3.1 and Figure 5"},{"comment":"Table 7 reports results for the fixed-teacher setting without standard deviations, even though Tables 4–6 report three repeated runs with standard deviations. Since the comparison between fixed and joint teachers is used to support the claim that joint adaptation is better, the absence of repeated runs makes it impossible to assess whether the differences (e.g., 65.09 vs. 65.67 for 2-bit PreResNet-18) are statistically meaningful. The authors should report the same repeated-run statistics for the fixed-teacher experiments.","section":"Table 7"}],"minor_comments":[{"comment":"The KL divergence in Eq. (7) is written as p_full(x_i) log(p_full(x_i)/p_low(x_i)), but Eq. (8) and Eq. (9) refer to L_KL(p_low|p_full) and L_KL(p_full|p_low), respectively, which is inconsistent with the direction in Eq. (7). The authors should make the argument order explicit and consistent throughout Sec. 3.4.","section":"Eq. (7)"},{"comment":"Algorithm 3 uses δ_t as both a probability and a linearly decayed value, with step 12 updating δ_{t+1} = δ_t − μ; however, the text states that δ is 'linearly decreased to 0 at the 20-th epoch.' The relationship between μ and the epoch count is not defined, so the reader cannot reproduce the exact decay schedule. A sentence defining μ in terms of epochs would suffice.","section":"Algorithm 3"},{"comment":"Table 3 reports results with one decimal place (e.g., 64.8, 85.7) for GroupNet baselines, while Table 2 reports the same or similar numbers with two decimal places. This makes direct comparison across tables awkward and suggests different runs or rounding conventions.","section":"Table 3"},{"comment":"The initial learning rate of 0.1 for the from-scratch student in Sec. 4.3.2 is very high compared with the 0.005 used in fine-tuning; the authors do not comment on whether the comparison between 'from scratch' and 'fine-tuning' is affected by this difference in learning rate, as opposed to the initialization alone.","section":"Sec. 4.3.2"},{"comment":"The sentence 'we find a 3.06% relative gap between the baseline on ResNet-50' is unclear: the numbers in Table 9 show a 3.06-point absolute difference (73.25 vs. 70.19), not a relative gap. The wording should distinguish absolute from relative improvement.","section":"Sec. 4.5"},{"comment":"Figure 2 legend refers to 'stage-1' and 'stage-2' but does not explicitly define which stage corresponds to which quantization configuration; the caption should briefly restate that stage-1 quantizes weights only and stage-2 quantizes both weights and activations.","section":"Figure 2"},{"comment":"Table 8 is labeled an 'Ablation study' but the spelling 'Abalation' appears in the caption; also, the table only reports ResNet-50 results, so the claim that 'integrating both posterior-based and attention-based distillation strategies achieve the best result' is only verified for one architecture.","section":"Sec. 4.3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical comparison paper, so the main risk is not circularity but the comparability of the training budgets. The 40-epoch SP schedule confound is the single most important issue: it affects the headline numbers in Tables 2, 9, and 10, and it can be fixed by re-running baselines under the same schedule or by showing that the extra epochs alone do not close the gap. The apparent top-5 typos in Table 1 also need correction, as they currently make some progressive-quantization improvements look artificially large. If the authors provide like-for-like comparisons and a small sensitivity analysis, the paper's practical contribution would be credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere is my read of arXiv:1908.04680. The paper proposes three training strategies for low-bit CNNs: progressive quantization (two-stage and bitwidth-annealing), stochastic precision (SP), and joint teacher-student distillation (KD). The central claim is that these improve low-bit accuracy over DoReFa-Net baselines.\n\nWhat is genuinely new and good: the joint distillation scheme, where the full-precision teacher is also updated rather than frozen, is a real idea. Tables 4-6 show consistent gains across ResNet and PreResNet at 4-bit and 2-bit, and the ablation of fixed vs. joint teacher (Table 7 vs. Table 5) supports their design. The two-stage and progressive-precision schemes are also clean, sensible, and show improvements in Table 1. The paper is thorough, with ablations on SP policies, guidance signals, and teacher depth.\n\nThe soft spots matter. The stress-test note holds up: Sec. 4.2 states SP experiments train for 40 epochs with LR decays at 25 and 35, while the default ImageNet schedule is 30 epochs with decays at 15 and 25. Tables 2, 9, and 10 report SP and SP+KD against 30-epoch baselines, so the 2-3% gain for SP may be partly or entirely due to extra training time. The paper does not re-run baselines under the 40-epoch schedule. That undermines the headline SP results. The KD-alone results in Sec. 4.3 use the default 30-epoch schedule, so those comparisons are like-for-like and remain credible.\n\nOther issues: Table 1 has typos (top-5 for 4-bit ResNet-50* is 75.70, which is clearly wrong, and the 2-bit ResNet-50* baseline top-5 differs between tables). No code release, and no sensitivity analysis for the key hyperparameters (delta, beta, gamma, alpha, precision schedule). The comparisons to very close prior work like Apprentice and Polino et al. are cited but not directly benchmarked.\n\nOverall, this is a legitimate and useful paper, but the epoch mismatch is a load-bearing flaw for the SP contribution specifically. The progressive quantization and joint KD contributions look solid and deserve publication. I would send it to peer review with a major-revision recommendation: re-run baselines under matched training budgets, fix the tables, and add sensitivity analysis. The authors may well be right that SP helps, but the evidence as presented does not prove it.\n\nI would cite this paper for the KD and progressive quantization parts, and I would bring it to reading group to discuss the confounding issue.\n\nBest","headline":"Useful quantization training tricks with a real confound: SP gains may be training-budget artifacts, but KD and progressive quantization hold up.","tokens_in":21693,"tokens_out":3128,"would_cite":true,"duration_ms":31148,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-bit convolutional networks can be trained to beat their quantized baselines.","keywords":["low-bitwidth neural networks","model quantization","progressive quantization","stochastic precision","knowledge distillation","straight-through estimator","image classification","convolutional neural networks"],"falsifier":"Re-run the 2-bit ResNet-50 ImageNet experiment with the paper's exact hyperparameters but disable stochastic precision while keeping joint distillation: the paper reports 71.96% top-1 without SP and 73.25% with it, so a replication showing no statistically significant gap would falsify the claim that stochastic precision adds a real benefit; similarly, disabling the distillation losses should reproduce the 70.19% baseline if the teacher guidance is genuinely responsible for the gain.","tokens_in":20729,"feed_emoji":"🧠","tokens_out":6579,"duration_ms":67724,"temperature":0.7,"pith_summary":"This paper argues that the accuracy drop of convolutional networks with very low-bit weights and activations is mainly a training problem, not a capacity problem, and that it can be mitigated by changing how the network is optimized. It proposes three training strategies: progressive quantization, which solves easier subproblems first by quantizing weights before activations and by gradually lowering bit-width; stochastic precision, which randomly quantizes only part of the network each iteration while annealing the quantized fraction to the full model; and joint knowledge distillation, in which a full-precision teacher and a low-precision student are trained together and adapt to each other. The central evidence is that these strategies, alone or combined, lift 2-bit ResNet-50 on ImageNet from 70.19% to 73.25% top-1 accuracy over the DoReFa-Net baseline. If true, this matters because low-bit networks offer large memory and compute savings, and the proposed training techniques are orthogonal to the specific quantizer, so they could be combined with other quantization schemes.","feed_headline":"How to train 2-bit CNNs without the accuracy crash","feed_subtitle":"Progressive quantization, stochastic precision, and joint distillation lift 2-bit ResNet-50 from 70.19% to 73.25% top-1.","key_machinery":"The machinery is the training schedule itself, built on three mechanisms. First, a fixed-point quantizer of the DoReFa form, $Q(z) = \\frac{1}{2^k-1}\\mathrm{round}((2^k-1)z_r)$, is combined with a straight-through gradient approximation, $\\partial z_q/\\partial z_r \\approx 1$, so gradient-based updates can flow through a non-differentiable quantizer. Second, a binary mask over network fragments (layers or residual blocks, and optionally weights versus activations) selects which parts are quantized at each iteration, with the stochastic ratio $\\delta$ starting at 0.5 and linearly decaying to zero; this provides the progressive-relaxation effect in one stage. Third, a two-headed loss couples the low-precision student to a jointly updated full-precision teacher through KL divergence on posterior probabilities, weighted by $\\beta = 0.5$, and attention transfer on feature maps, weighted by $\\gamma = 50$. These mechanisms carry the argument by making the discrete optimization tractable and by continuously guiding the student toward the teacher during training.","core_discovery":"The paper's central claim is that the optimization difficulty of quantized neural networks, not the capacity of the low-precision representation, is the main obstacle to accurate low-bit models, and that this difficulty can be attacked with three training techniques. Progressive quantization first solves the easier problem of quantized weights with full-precision activations, then adds activation quantization, and also anneals the bit-width from 32 bits down to the target precision. Stochastic precision randomly quantizes a fraction of layers, blocks, weights, or activations per iteration while the stochastic ratio is linearly decayed to zero, easing gradient flow in a single training stage. Joint knowledge distillation trains a full-precision teacher and a low-precision student together, using both posterior KL divergence and attention-transfer losses, and reports that the student surpasses the baseline and sometimes the teacher improves as well. The paper reports that all low-precision models produced by these methods surpass their corresponding baselines, with the largest combined gain reaching 73.25% top-1 accuracy for 2-bit ResNet-50 on ImageNet.","pith_inferences":["A natural extension, not pursued in the paper, would be to replace the hand-set stochastic ratio decay and distillation weights with learned or per-layer schedules, since the paper gives no sensitivity analysis for these hyperparameters and the gains might be further improved or made more robust.","Beyond image classification, one could test the same three training strategies on low-bit object detection or semantic segmentation, where the optimization difficulty is typically greater and the reported benefits might be even more pronounced.","The observation that the jointly updated teacher improves suggests that mutual distillation could serve as a general regularizer for full-precision training, independent of quantization.","Because the methods are quantizer-agnostic, a testable extension is to pair them with modern learned-step-size quantizers or with mixed-precision allocation, potentially recovering additional accuracy at the same average bit-width."],"forward_implications":["2-bit ResNet-50 on ImageNet can reach 73.25% top-1 accuracy with stochastic precision plus joint distillation, compared with 70.19% for the DoReFa-Net baseline, closing most of the gap to the 75.64% full-precision model.","The three training strategies are complementary: two-stage optimization, progressive precision, stochastic precision, and knowledge distillation can be combined, and each combination in the paper improves over the corresponding baseline.","Joint distillation helps more when the low-precision student is harder to train, since learning from scratch shows larger relative gains than fine-tuning, and quantized ResNet benefits more than quantized PreResNet.","The training techniques are orthogonal to the quantizer: stochastic precision improves DoReFa-Net, LQ-Net, BiReal-Net, and GroupNet baselines, suggesting the methods could be transferred to other quantization designs.","The full-precision teacher can also be slightly improved by joint training with the student, indicating that the mutual adaptation acts as a regularizer rather than a one-way transfer."],"supporting_citations":[{"why":"Supplies the DoReFa quantization function and the baseline training setup that all proposed strategies build on and compare against.","marker":"[12]"},{"why":"Introduces the straight-through estimator used in Eq. (4) to approximate gradients through the non-differentiable quantizer.","marker":"[38]"},{"why":"Is the preliminary conference version that introduced progressive quantization and knowledge distillation for low-bit networks, extended here into the unified training framework.","marker":"[23]"},{"why":"Provides the posterior-probability knowledge distillation scheme whose KL divergence is used as one of the two guidance losses.","marker":"[19]"},{"why":"Supplies the attention-transfer method used in Eq. (6) to align normalized feature-map attention between teacher and student.","marker":"[21]"},{"why":"Furnishes the incremental quantization idea that motivates the progressive and stochastic precision training schemes.","marker":"[9]"},{"why":"Provides the LQ-Net non-uniform quantizer baseline that is improved when combined with the proposed stochastic precision strategy.","marker":"[36]"}],"fun_headline_variants":["Progressive quantization lifts 2-bit ResNet to 73.25%","Low-bit CNN training: three tricks to beat the quantizer","Stochastic precision eases low-bit CNN training","Joint distillation rescues low-bit CNN accuracy","Train low-bit CNNs with progressive quantization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the straight-through estimator supplies genuinely useful gradient directions at very low precision, and that the hand-set schedule of stochastic ratios and distillation weights keeps working for other architectures, datasets, and quantization schemes.","fun_headline_variants_meta":{"raw":{"variants":["Progressive quantization lifts 2-bit ResNet to 73.25%","Low-bit CNN training: three tricks to beat the quantizer","Stochastic precision eases low-bit CNN training","Joint distillation rescues low-bit CNN accuracy","Train low-bit CNNs with progressive quantization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001005,"raw_usage":{"total_tokens":4293,"prompt_tokens":1031,"completion_tokens":3262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":3184}},"tokens_in":647,"tokens_out":3262,"duration_ms":25672,"temperature":1.0,"reasoning_tokens":3184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:03:31.765102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 2-bit ResNet-50 ImageNet experiment with the paper's exact hyperparameters but disable stochastic precision while keeping joint distillation: the paper reports 71.96% top-1 without SP and 73.25% with it, so a replication showing no statistically significant gap would falsify the claim that stochastic precision adds a real benefit; similarly, disabling the distillation losses should reproduce the 70.19% baseline if the teacher guidance is genuinely responsible for the gain.","supporting_citations":[{"cited_title":"Towards effective low-bitwidth convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Is the preliminary conference version that introduced progressive quantization and knowledge distillation for low-bit networks, extended here into the unified training framework."},{"cited_title":"Distilling the knowledge in a neural network,","cited_arxiv_id":null,"evidence_quote":"Provides the posterior-probability knowledge distillation scheme whose KL divergence is used as one of the two guidance losses."},{"cited_title":"Paying more attention to atten- tion: Improving the performance of convolutional neural networks via attention transfer,","cited_arxiv_id":null,"evidence_quote":"Supplies the attention-transfer method used in Eq. (6) to align normalized feature-map attention between teacher and student."},{"cited_title":"Incremental network quantization: Towards lossless cnns with low-precision weights,","cited_arxiv_id":null,"evidence_quote":"Furnishes the incremental quantization idea that motivates the progressive and stochastic precision training schemes."},{"cited_title":"Lq-nets: Learned quanti- zation for highly accurate and compact deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the LQ-Net non-uniform quantizer baseline that is improved when combined with the proposed stochastic precision strategy."}],"review_version":1}