{"id":"122ba9e9-c539-43e0-8a9e-82a3a9b2bf51","arxiv_id":"2501.07804","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A weighted sum of forward and reverse KL divergence with two temperatures is proposed as a logit-based knowledge distillation loss, with reported gains on image classification and semantic segmentation.","lead":"This paper proposes a knowledge distillation loss that combines forward and reverse KL divergence with separate temperature settings, claiming better transfer of a teacher's low-probability outputs to a student network. The authors report 1 to 3 percent accuracy gains on CIFAR-100 and ImageNet and a 4.72 point mIoU gain on Cityscapes over a no-distillation baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Forward-KL gradient derivation ignores softmax coupling; negative classes receive nonzero gradient, so the paper's motivating imbalance argument is invalid and the mechanism behind BDD's gains is unverified.","rationale":"The reader identified the softmax-coupling issue as the weakest assumption; I agree and consider it the most load-bearing concern. The paper's theoretical contribution is the claim that forward KL creates a positive/negative imbalance; if that premise is false, the method's novelty reduces to a heuristic addition of a reverse-KL term with separate temperatures. The empirical results are plausible but not independently verified (no code, average of runs), and the undefined integral in Eq. (9) and the inconsistency in temperature settings (Sec. 4.1 says τf=2, τr=8, while Table 8 shows best ResNet result at τf=8, τr=2) further reduce confidence. The concrete test above would settle whether the motivating derivation is invalid. If it is, the paper should be revised to either provide a correct mechanism or reframe the contribution as an empirical regularization technique, which would lower the novelty/claims but not necessarily reject the empirical findings. Therefore, the CONDITIONAL verdict is appropriate: accept with major revisions, not outright reject.","tokens_in":14987,"tokens_out":5602,"duration_ms":53752,"concrete_test":"Compute the gradient of the forward-KL term in Eq. (2) with respect to the student logit z_c under the softmax parameterization. For a two-class example with teacher logits (10,0), student logits (0,0), τ=1, the negative-class gradient is ∂KL_f/∂z_neg = p_S_neg − p_T_neg = 0.5, not 0 as implied by Eq. (5). If this derivative is nonzero, the paper's premise that forward KL neglects negative classes is falsified; the empirical gain of BDD must then be attributed to another mechanism, and the central claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that standard forward-KL KD ignores the teacher's extremely small negative probabilities, and that adding a weighted reverse-KL term restores balance. The supporting derivation in Sec. 3.2 (Eqs. 4–7) differentiates the loss with respect to each softmax probability p(F_S_c) as if the probabilities were independent. This is mathematically incorrect: the softmax has a shared denominator, so the loss must be differentiated with respect to the student logits z_c. Doing so gives ∂KL_f/∂z_c = (p_S_c − p_T_c)/τ. For a negative class with p_T_c ≈ 0, the gradient is p_S_c/τ, which is nonzero whenever the student assigns any probability to that class. Thus the forward-KL term does not 'ignore' negatives; it actively pushes p_S_c down. Eq. (5), which concludes the gradient vanishes as p_T→0, is an artifact of ignoring the normalization. The proposed reverse-KL term is therefore not correcting the stated deficiency. The empirical gains in Tables 1–4 may be real, but they are unexplained by the paper's theory; they could stem from the added reverse-KL term acting as a regularizer or from the temperature schedule. The paper's mechanistic claim is unverified, and the 'accumulate' variant (Eq. 9) is additionally undefined, further weakening the theoretical presentation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Balance Divergence Distillation (BDD), a logit-based knowledge distillation loss that adds a weighted reverse-KL term to the standard forward-KL term, using separate temperatures for the two terms (Eq. 8). The motivation is that forward KL neglects the teacher's extremely small negative-class probabilities, and that the reverse-KL term restores balance. The method is evaluated on CIFAR-100 and ImageNet classification and on Cityscapes semantic segmentation, reporting consistent gains of 1-3% top-1 accuracy over KD baselines and 4.72 mIoU improvement for PSPNet-R18. The paper also integrates BDD with MLD and CWD and reports ablations over the weighting coefficient and temperature settings.","tokens_in":15232,"tokens_out":5855,"duration_ms":57648,"significance":"The empirical component is a real strength: the experiments cover multiple teacher-student architectures, two classification datasets, a dense-prediction task, and integration with existing logit- and attention-based distillation methods; the gains in Tables 1-4 are consistent. If the mechanism were correctly established, BDD would be a simple and broadly applicable improvement to logit-based distillation. However, the theoretical derivation of the alleged imbalance problem is mathematically flawed, and the temperature-setting recommendation in the text is contradicted by the paper's own ablation table. The contribution therefore currently rests on an unverified mechanism, and the paper needs substantial revision before the central claim can be accepted.","major_comments":[{"comment":"The derivation treats the softmax probabilities p(F^S_c) as independent variables when differentiating the KL terms. Differentiating the forward-KL term with respect to the student logit z_j gives ∂KL_f/∂z_j = (p^S_j − p^T_j)/τ, not a quantity proportional to −p^T_j/p^S_j. For a negative class with p^T_j ≈ 0, the gradient is approximately p^S_j/τ, which is nonzero whenever the student assigns any probability to that class. The limit in Eq. (5) is an artifact of ignoring the shared softmax denominator, and the same problem affects Eq. (7). The paper's central motivation, that forward KL ignores teacher negatives, is therefore not supported by its own gradient analysis. The empirical gains may still be real, but the claimed mechanism is unverified; the authors should either redo the gradient analysis correctly or reframe the contribution as an empirical loss design and provide direct evidence (for example, gradient statistics or ablations isolating the effect of the reverse-KL term) for the proposed mechanism.","section":"3.2 (Eqs. 4-7)"},{"comment":"The 'accumulate' variant is not well defined. Equation (9) writes an integral with respect to d(F^T_c/τ_f) and d(F^S_c/τ_r), but F^T_c and F^S_c are network outputs, not integration variables, and no integration range or measure is specified. Table 8 reports results for 'accumulate' without explaining how the expression is computed in practice. Since this variant is used as evidence for the BDD+ configuration, its definition must be supplied or the variant should be removed from the tables.","section":"3.3 (Eq. 9)"},{"comment":"The text and the ablation table directly contradict each other. The text states that for ResNet teachers one should set τ_f smaller and τ_r larger, and that for VGG teachers one should set τ_f larger and τ_r smaller. Table 8 shows the opposite: for ResNet32×4→ResNet8×4, the best configurations are (8.0, 2.0) and 'accumulate' (76.21 and 76.22), while (2.0, 8.0) gives 76.02; for VGG13→VGG8, the best configuration is (2.0, 8.0) at 74.74 and the worst is (8.0, 2.0) at 73.50. The proposed temperature-ratio recommendation is contradicted by the reported numbers and needs to be corrected or explained.","section":"4.4 (Table 8)"},{"comment":"Several empirical claims lack measures of uncertainty. Tables 1 and 2 state that results are averages of five runs but report no standard deviations, and Table 3 reports no variance at all. In addition, the ImageNet KD baseline for MobileNetV2 (68.58) is below the student trained without distillation (68.87), which is a known failure mode of logit distillation and should be discussed because it affects the interpretation of the reported +3.25 gain. Adding variance estimates or at least discussing run-to-run variation would materially strengthen the central empirical claim.","section":"4.2 (Tables 1-3)"}],"minor_comments":[{"comment":"The abstract and introduction state a 4.55% mIoU improvement for the Cityscapes semantic segmentation experiments, while Table 4 and Section 4.3 report 4.72 mIoU for PSPNet-R18; these numbers should be harmonized.","section":"Abstract and Introduction"},{"comment":"There are several typos and infelicities: 'sementatic' (Section 4.3), 'Mehtod' (Table 5 caption), 'releation' (Section 1), 'ifp(F T ) > 0' (Section 3.2), and inconsistent capitalization of Hinton's name in Section 1.","section":"Throughout"},{"comment":"Equation (2) writes the KL divergence with an explicit spatial sum, but the later equations in Section 3 drop or reorder the spatial indices without comment; the notation should be made uniform so that the classification and dense-prediction cases are clearly distinguished.","section":"Eq. (2)"},{"comment":"The checkmark-style entries in Table 6 are visually ambiguous; replacing them with explicit 'yes'/'no' entries would improve readability.","section":"Table 6"},{"comment":"The caption of Figure 2 repeats almost verbatim the caption of Figure 1; the two figures should have distinct descriptive captions.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main weakness is the mismatch between the claimed theoretical motivation and the actual softmax gradient. The empirical study is fairly extensive and the proposed loss is simple and easy to implement, so the contribution is salvageable, but the revision must fix the gradient derivation or honestly reframe the contribution as empirical, and it must resolve the contradiction in the temperature recommendation. I do not see grounds for outright rejection because the loss itself is well defined and the reported gains are consistent across tasks; however, in its current form the paper does not support its central mechanistic claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BDD is a simple logit-distillation loss: forward KL plus a weighted reverse KL, each with its own temperature. The empirical story is solid: consistent 1–3% top-1 gains on CIFAR-100, believable ImageNet numbers, and a meaningful +4.72 mIoU on Cityscapes when applied to channel-wise attention. The loss is easy to bolt onto existing pipelines, and the authors show it helps when stacked on MLD. That part is genuinely useful.\n\nWhat is not new is the core loss: a weighted sum of forward and reverse KL is symmetric KL, and the authors cite BD-KD [22] which already uses symmetric KL for online distillation. The separate temperatures are a modest extension, along the lines of CTKD. Calling this 'novel' oversells it.\n\nThe bigger problem is the theory. Section 3.2 differentiates the KL loss with respect to each softmax probability as if they were independent. That is not legitimate: softmax shares a denominator. Differentiating with respect to the student logit z_c gives ∂KL_f/∂z_c = (p_S_c − p_T_c)/τ. For a negative class with p_T_c ≈ 0, the gradient is p_S_c/τ, not zero. So forward KL does not 'ignore' negatives; it actively pushes the student's probability down. The claim that the reverse term fixes a misspecified imbalance is therefore unsupported. The gains may come from the reverse KL acting as a regularizer, or from the temperature schedule, but the paper does not demonstrate that.\n\nThe temperature ablation makes matters worse. The text recommends τf small and τr large for ResNet, but Table 8 shows (τf=8, τr=2) is slightly better for ResNet, and for VGG the recommended (8,2) is the worst configuration. That internal contradiction needs to be resolved. Equation 9, the 'accumulate' variant, is never defined; Table 8 lists it but the paper doesn't say how it was computed. The abstract says 4.55% mIoU, the table says 4.72. No code is provided.\n\nThis is a paper that probably works empirically but explains itself badly. The loss is a reasonable thing to try, and the experiments are extensive enough that I'd send it to review with a clear request for major revision: fix the gradient derivation (or drop it), reconcile the temperature story with the ablation, define Eq. 9 or remove it, and release code. If those are fixed, the practical contribution is real, if modest.","headline":"A simple two-temperature symmetric-KL loss with decent empirical gains, but the paper's theoretical motivation is mathematically wrong and its temperature recommendations contradict its own ablation.","tokens_in":15778,"tokens_out":3590,"would_cite":false,"duration_ms":30962,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a weighted reverse-KL term to the standard forward-KL distillation loss lets students learn the teacher's tiny non-target probabilities, yielding 1–3% top-1 accuracy gains on CIFAR-100 and ImageNet and a 4.72…","keywords":["Knowledge distillation","Kullback-Leibler divergence","Reverse KL divergence","Soft labels","Temperature coefficient","Image classification","Semantic segmentation","Model compression"],"falsifier":"Compute the exact gradient of the forward-KL distillation term with respect to the student's raw scores (not its normalized probabilities) for a class where the teacher's probability is below $10^{-8}$ while the student still assigns noticeable probability to that class; if that gradient component is not negligible, the forward-KL loss does carry information about tiny teacher probabilities, contradicting the paper's motivating claim.","tokens_in":14713,"feed_emoji":"⚖️","tokens_out":8703,"duration_ms":76811,"temperature":0.7,"pith_summary":"The paper sets out to show that standard knowledge distillation, which minimizes forward Kullback-Leibler divergence between teacher and student softmax outputs, systematically underweights the teacher's tiny non-target probabilities, which the authors call the negative part of dark knowledge. Its proposed fix, Balance Divergence Distillation (BDD), adds a weighted reverse-KL term with its own temperature to the usual forward-KL term, so the student is pushed to match the teacher in the low-probability tail as well as the high-probability head. If the claim is right, a simple additive change to an existing loss improves logit-based distillation by 1–3% top-1 accuracy on CIFAR-100 and ImageNet, and gives a roughly 4.7 mIoU improvement for a segmentation student on Cityscapes. This matters because logit-based distillation is the cheapest form of knowledge transfer, and a drop-in loss change could improve many lightweight vision models.","feed_headline":"Reverse-KL term lifts distillation accuracy 1–3%","feed_subtitle":"The BDD loss adds a weighted reverse-KL term with its own temperature, improving CIFAR-100, ImageNet, and Cityscapes students.","key_machinery":"The load-bearing object is the two-temperature, two-direction KL sum. Forward KL, $\\mathrm{KL}_f(p(F_T/\\tau_f)\\|p(F_S/\\tau_f))$, keeps the student close to the teacher where the teacher is confident; the added reverse term, $\\alpha\\,\\mathrm{KL}_r(p(F_S/\\tau_r)\\|p(F_T/\\tau_r))$, is supposed to transfer the teacher's small probabilities. The separate temperatures $\\tau_f$ and $\\tau_r$ are the adjustable levers: a low forward temperature sharpens the positive-class signal, while a high reverse temperature spreads the student's attention over the long tail of near-zero teacher probabilities. The paper claims this decomposition counteracts the zero-avoiding property of forward KL, which otherwise lets the student ignore classes the teacher assigns essentially zero probability.","core_discovery":"The central claim is that forward-KL distillation starves the student of the teacher's tiny non-target probabilities, so the student overfits the confident classes and never models the tail. The proposed remedy is the Balance Divergence Distillation loss, $L_{\\mathrm{BDD}} = \\mathrm{KL}_f(p(F_T/\\tau_f)\\,\\|\\,p(F_S/\\tau_f)) + \\alpha\\,\\mathrm{KL}_r(p(F_S/\\tau_r)\\,\\|\\,p(F_T/\\tau_r))$, a weighted sum of forward and reverse KL terms with separate softmax temperatures. The paper argues that the reverse term, $\\mathrm{KL}(\\text{student}\\|\\text{teacher})$, forces the student to keep probability mass where the teacher is near zero, while the forward term preserves fidelity on high-probability classes. Experiments on CIFAR-100 and ImageNet show top-1 accuracy gains of roughly 1–3% over the KD baseline across many teacher–student pairs, and on Cityscapes the method improves a segmentation student's mIoU by about 4.7 points over the baseline and by 0.58 over the channel-wise distillation method it builds on.","pith_inferences":["The mechanism may not be exactly what the paper states: because the gradient derivation treats each softmax output as independent, the two-temperature structure might mainly act as a form of logit-space regularization or ensembling rather than specifically rescuing ignored negative classes; an ablation that fixes the total KL weight and varies only the temperature split would separate these explan","The same reverse-KL compensation could be applied to feature-based distillation objectives that use normalized attention maps, where long-tailed channel statistics may create a similar imbalance.","By making the reverse temperature a hyperparameter, the method effectively rescales the teacher's distribution; a natural extension would be to learn the temperature pair from a small validation set, turning a manual knob into a trained component."],"forward_implications":["Integrating BDD into an existing logit-distillation pipeline is a one-term change: compute forward KL with one temperature, reverse KL with another, and add them with weight $\\alpha$.","On CIFAR-100, BDD improves top-1 accuracy over the KD baseline by roughly 1–2% for same-architecture teacher–student pairs and 2–3% for heterogeneous pairs, and it can be stacked on top of multi-level logit distillation.","On ImageNet, the gain over the KD baseline is about 1–3% for both ResNet-style and MobileNet-style students.","On Cityscapes, applying BDD to channel-wise attention outputs raises mIoU by about 4.7 points for one segmentation student and about 2.7 points for another, beating channel-wise distillation alone.","The optimal temperature split is architecture-dependent: deeper ResNet-style students favor a lower forward temperature and higher reverse temperature, while VGG-style students favor the opposite, so the two terms should be tuned rather than set equal."],"supporting_citations":[{"why":"Supplies the base knowledge-distillation loss, forward KL between teacher and student softmax outputs, which BDD modifies.","marker":"[5]"},{"why":"States the asymmetry of KL divergence used to justify separating the forward and reverse directions.","marker":"[34]"},{"why":"Supplies the zero-avoiding property that the authors use to claim forward KL ignores tiny teacher probabilities.","marker":"[35]"},{"why":"Provides the temperature-adjustment distillation method that motivates the separate forward and reverse temperature coefficients.","marker":"[11]"},{"why":"Is the balanced logit-distillation baseline BDD is compared against on CIFAR-100 and ImageNet.","marker":"[12]"},{"why":"Is the multi-level logit method BDD is stacked on to show the new loss is compatible with other distillation losses.","marker":"[13]"},{"why":"Provides the channel-wise attention distillation setup for dense prediction that BDD adapts for semantic segmentation.","marker":"[10]"}],"fun_headline_variants":["Reverse KL term boosts distillation accuracy by 1-3%","Balance Divergence: fixing KD's blind spot in tiny probabilities","Reverse-KL correction lifts student accuracy 1-3% on vision benchmarks","BDD loss: reverse KL to capture teacher's tail probabilities","Reverse KL term lifts classification 1-3% and segmentation mIoU 4.55%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a near-zero teacher probability sends the student no learning signal, so those tail classes never get modeled; if the shared normalizer that couples all output probabilities is taken into account, that premise may fail.","fun_headline_variants_meta":{"raw":{"variants":["Reverse KL term boosts distillation accuracy by 1-3%","Balance Divergence: fixing KD's blind spot in tiny probabilities","Reverse-KL correction lifts student accuracy 1-3% on vision benchmarks","BDD loss: reverse KL to capture teacher's tail probabilities","Reverse KL term lifts classification 1-3% and segmentation mIoU 4.55%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001193,"raw_usage":{"total_tokens":4985,"prompt_tokens":1074,"completion_tokens":3911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":3813}},"tokens_in":690,"tokens_out":3911,"duration_ms":27174,"temperature":1.0,"reasoning_tokens":3813,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:35:43.571866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the exact gradient of the forward-KL distillation term with respect to the student's raw scores (not its normalized probabilities) for a class where the teacher's probability is below $10^{-8}$ while the student still assigns noticeable probability to that class; if that gradient component is not negligible, the forward-KL loss does carry information about tiny teacher probabilities, contradicting the paper's motivating claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the temperature-adjustment distillation method that motivates the separate forward and reverse temperature coefficients."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the balanced logit-distillation baseline BDD is compared against on CIFAR-100 and ImageNet."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the multi-level logit method BDD is stacked on to show the new loss is compatible with other distillation losses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the channel-wise attention distillation setup for dense prediction that BDD adapts for semantic segmentation."}],"review_version":1}