{"id":"834e3007-82f9-4e2e-a54f-43cc62eef3a5","arxiv_id":"2508.20224","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Calibrating the teacher with temperature scaling before knowledge distillation gives small student accuracy gains, but the paper's causal claim that calibration error is the key driver is not cleanly established.","lead":"This paper tests whether a teacher model's confidence calibration, not just its accuracy, determines how much a student model learns from knowledge distillation. It finds a strong correlation and shows that softening teacher predictions with temperature scaling gives small accuracy gains across image classification and detection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Teacher calibration is not isolated from distribution softness; Table 5 and Eq. (8) undermine the causal claim.","rationale":"The reader's weakest assumption—that the benefit might come from soft-target smoothing rather than calibration—is exactly the central soft spot. Table 5 provides direct internal evidence: teacher ACE increases while student accuracy improves, which the paper explains by invoking underconfidence, effectively conceding that the operative variable is not calibration error per se. The formal derivation in Eqs. (5)–(8) is not a proof: postulating p_error ≈ y is an assumption, not a consequence of any stated property of trained networks, and it ignores that temperature scaling rescales all logits, changing both the 'calibrated' and 'error' components. The empirical correlations in Figure 1 are uncontrolled for teacher accuracy or output entropy, so they do not identify a unique mechanism. These issues do not diminish the practical value of the recipe (temperature-scaling the teacher is cheap and consistently helps), nor the breadth of experiments. But the paper's title and conclusion assert a causal role for calibration, and that assertion is not yet warranted. The reader's CONDITIONAL verdict is appropriate; no change is needed from this stress-test.","tokens_in":13880,"tokens_out":4620,"duration_ms":47179,"concrete_test":"Sweep temperature T on a fixed teacher-student pair (e.g., ResNet32x4→ShuffleNetV1) from T=1 to T=6. For each T, measure teacher ACE, ECEo/ECEu, and student accuracy (mean over ≥3 runs). Plot student accuracy vs. teacher ACE across the sweep. If student accuracy keeps rising while teacher ACE rises (as Table 5 suggests for T beyond the calibrated point), the claim that reducing calibration error drives KD improvement is falsified in the paper's own setting. To complement, compute the partial correlation between teacher ACE and student accuracy for the 17 teachers in Figure 1, controlling for teacher accuracy and teacher output entropy; if the partial R^2 drops substantially, the Figure 1 correlation is confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that teacher calibration error is an important causal factor in KD—is load-bearing, and it is not established. The intervention (temperature scaling) changes both calibration and the softness of the teacher's output distribution, and the paper's own Table 5 shows a case (ResNet32x4→ShuffleNetV1) where the teacher's ACE increases from 0.081 to 0.131 while student accuracy improves from 72.69 to 73.66. The authors attribute this to increased underconfidence, but this directly contradicts a simple 'lower ACE → better student' mechanism. The analytic derivation in Eqs. (5)–(8) assumes the overconfident error component is exactly the one-hot distribution (p_err ≈ y), which is not generally true; an overconfident model may concentrate probability on a wrong class without being one-hot, and temperature scaling alters the entire distribution, not just an error component. Thus the derivation does not isolate calibration error. The observed gains may be fully explained by soft-target smoothing, a known KD mechanism, leaving the paper's headline interpretation unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the calibration error of the teacher model is an important factor in knowledge distillation (KD). It presents an observational study (Fig. 1) correlating teacher ACE with student accuracy across 17 teacher models, claims that ACE is more predictive than teacher accuracy, and proposes a simple intervention: temperature-scaling the teacher network before KD. Experiments on CIFAR-100, ImageNet, and MS-COCO show accuracy gains over vanilla KD and some state-of-the-art methods when teacher temperature scaling is applied. The paper also includes a theoretical decomposition (Sec. III-B, Eqs. 5–8) intended to explain why overconfident teachers weaken KD, and additional experiments with other calibration methods (Sec. IV-E). The central claim is that reducing teacher calibration error leads to better students, and that calibration error should be a teacher-selection criterion.","tokens_in":14125,"tokens_out":5625,"duration_ms":64673,"significance":"If the causal claim were established, the paper would provide a simple, inexpensive way to improve KD performance across tasks and a new, practically useful teacher-selection criterion. The experimental scope is broad (classification and detection, multiple teacher–student architectures, integration with feature- and logit-based KD), and the core observation that temperature scaling often helps is reproducible in principle. However, the significance is substantially undercut by identification problems: the observational correlation does not control for teacher accuracy, the mechanistic derivation is tautological under an untested approximation, and the intervention (temperature scaling) does not isolate calibration error from distribution softness. The paper's own Table 5 provides a direct counterexample to the claim that lower teacher ACE causes better students. The useful empirical finding may be that softening teacher targets improves KD, but the calibration-error interpretation is not currently supported.","major_comments":[{"comment":"The claimed correlation (R²=0.92 between teacher ACE and student accuracy) is univariate and does not control for teacher accuracy. Since teacher accuracy and calibration error are correlated in practice (as noted in ref. [4]), the observation is consistent with teacher accuracy being the underlying factor. The paper should report partial correlations controlling for teacher accuracy, or stratify teachers by accuracy. As it stands, the claim that calibration error is 'more important' than teacher accuracy is not established.","section":"Figure 1 / Section III-B"},{"comment":"The derivation assumes p_error ≈ y, i.e., the overconfident error component equals the one-hot ground-truth label. This is a strong, untested assumption; an overconfident model can concentrate its probability mass on a wrong class. With this assumption, the conclusion that overconfidence reduces the KD term is a tautology rather than a derived mechanism. The derivation also does not isolate calibration error from the softness of the teacher distribution. Table 5 (ResNet32x4→ShuffleNetV1) shows teacher ACE rising from 0.081 to 0.131 while student accuracy improves from 72.69 to 73.66; the explanation in Sec. IV-D ('slightly underconfident teacher') directly contradicts the claim that lower ACE is the causal driver.","section":"Section III-B, Eqs. (5)–(8)"},{"comment":"The text claims that 'we can exclusively reduce the overconfident calibration error while fixing the accuracy.' This is not supported: temperature scaling alters the entire probability distribution, can increase ACE (as in Table 5), and can induce underconfidence. The observed KD gains could therefore be due to softer targets—a known KD mechanism—rather than to reduced calibration error. The paper needs a control that varies softness independently of measured calibration error (e.g., comparing against simply increasing the standard KD temperature) and should report ACE/ECE for all teacher models used in Tables 1–4. Without such a control, the central causal claim is not identified.","section":"Section IV-A"},{"comment":"For ResNet32x4→ShuffleNetV2, KD+Ours gives 75.99% vs KD 76.15% (Δ = -0.16), contradicting the abstract's 'consistently achieving superior performance' and Sec. IV-C1's 'consistently yields comparable or better performance across all models.' This inconsistency should be acknowledged and explained (e.g., statistical noise, variance), or the claims should be tempered to 'on average' with per-pair statistics. Combined with Table 5, this undermines the robustness of the headline claim.","section":"Table 2, Sec. IV-C1"}],"minor_comments":[{"comment":"Typo: 'overcalibrated error' should likely read 'overconfident error.' Also, the notation p(i)_cal and p(i)_error is not defined explicitly; please clarify that these are conditional on the teacher's output.","section":"Section III-B"},{"comment":"The interaction between the calibration temperature (T=1.5) and the standard KD temperature (T=4) is not clearly described. Please specify the exact forward pass: are teacher logits divided by 1.5 before softmax, and then is the student trained with the standard KD loss using T=4? This is important for reproducibility.","section":"Section IV-A"},{"comment":"The table does not indicate the student architecture used. Please specify the student model and dataset for these calibration-method comparisons.","section":"Table 6"},{"comment":"The y-axis label 'Accuracy (%)' should be clarified to 'Student accuracy (%)' for consistency with the text.","section":"Figure 2"},{"comment":"The limitation paragraph states the study is 'limited in scope to logit distillation methods,' but Table 4 applies the method on top of ReviewKD, a feature-based method. Please reconcile this statement.","section":"Section V (Limitation)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the causal claim that teacher calibration error drives KD performance. The current evidence does not support this claim due to confounding (teacher accuracy), a tautological derivation, and direct counterexamples (Table 5, Table 2 negative delta). The empirical finding that temperature scaling often helps may still be publishable if reframed as a softening intervention, but this would require substantial new analysis and a change in the paper's framing. The authors should be asked to provide partial correlations, controls for softness, and a more cautious interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the recipe works, the explanation is shaky. The paper shows a simple thing—temperature-scale the teacher before KD—and gets consistent small gains. That's practically useful, and the breadth of experiments (17 teachers, CIFAR-100, ImageNet, COCO, plus combining with MLLD and ReviewKD) is a genuine plus. The student also ends up better calibrated, which is a nice side effect. They also openly cite [20] as the closest prior work, so the core idea isn't new; this is an extended empirical study and a recipe, not a new principle.\n\nThe soft spots: First, Figure 1 is an observational correlation over 17 teachers, and teacher accuracy is a known confound. With 17 points they could have easily run a partial correlation, but they didn't. Second, the intervention—temperature scaling—changes both calibration and distribution softness. Any improvement could just be soft-target smoothing, which is already a well-known KD mechanism. Third, the derivation in Eqs. (5)-(8) packs the conclusion into the assumption that the overconfident error component is a one-hot vector. That's not generally true, and temperature scaling doesn't separate a calibrated part from an error part.\n\nOne nuance missed by the stress-test: Table 5 actually supports the narrower overconfidence story. In the ResNet32x4→ShuffleNetV1 row, total ACE rises, but ECEo drops sharply (9.27→1.16) while ECEu rises, and the student improves. So the paper's internal mechanism is about overconfidence, not ACE overall. The problem is that the abstract, Figure 1, and the conclusion talk about ACE and 'calibration error' in general. That mismatch makes the paper look less careful than it actually is.\n\nThey also state in the Limitation that they only cover logit distillation, which is honest. No code or data is released, which limits immediate reuse, but the training details are clear enough to reproduce.\n\nVerdict: as an empirical contribution, this deserves a serious referee. The recipe is cheap and probably correct, but the causal claim needs work: partial correlations, an ablation that separates temperature from calibration (e.g., apply temperature to a teacher that is already well-calibrated, or use vector scaling with a validation set), and a derivation that doesn't assume what it's proving. If those are addressed, the paper could be a solid practical contribution.","headline":"Useful empirical recipe, but the calibration story is oversold; the gains are real, the causal claim is not.","tokens_in":14582,"tokens_out":3783,"would_cite":true,"duration_ms":43316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The calibration error of the teacher — how well its confidence matches reality — is a major driver of knowledge distillation performance.","keywords":["knowledge distillation","teacher calibration","temperature scaling","calibration error","overconfidence","adaptive calibration error","model compression","teacher selection"],"falsifier":"Train students with two teachers of equal validation accuracy but different calibration error — for example, a standard teacher and an accuracy-matched teacher trained with label smoothing or mixup. If student accuracy does not track teacher ACE when teacher accuracy is held constant, the calibration-causes-gains claim is falsified. Similarly, if temperature-scaling an already well-calibrated teacher still improves the student, then softening alone, not calibration, is the active ingredient.","tokens_in":13778,"feed_emoji":"🎓","tokens_out":5890,"duration_ms":65304,"temperature":0.7,"pith_summary":"This paper tries to establish that in knowledge distillation, what makes a teacher effective is not its raw accuracy but how well calibrated its probability outputs are. It shows that teacher accuracy explains only about 56–68% of student accuracy, while the teacher's Adaptive Calibration Error explains about 90–92%. It then demonstrates that temperature-scaling the teacher's logits — a simple calibration step that does not change the teacher's accuracy — improves student accuracy across classification and object detection, and can be layered onto state-of-the-art distillation methods. If the claim is right, it gives practitioners a cheap new lever: calibrate the teacher before distilling, and a new criterion for choosing which teacher to use.","feed_headline":"Teacher calibration error predicts student gains in distillation","feed_subtitle":"Temperature-scaling a teacher's outputs lifts student accuracy on CIFAR-100, ImageNet, and COCO.","key_machinery":"Temperature scaling applied only to the teacher: dividing the teacher's logits by T > 1 before the softmax softens the output probabilities without changing the predicted class. The paper pairs this with the Adaptive Calibration Error (ACE) metric, a classwise, equal-bin variant of expected calibration error that measures confidence–accuracy mismatch. The explanatory mechanism is a decomposition of the teacher probability as p = (1-k)p_cal + k p_err, where the overconfident error term is approximated as the one-hot label y; this makes the overconfident component reinforce standard cross-entropy and shrink the effective weight of the informative distillation signal.","core_discovery":"The paper's central claim is that the teacher's calibration error is a key determinant of how much a student learns from knowledge distillation. Using 17 pretrained teachers on CIFAR-100, the authors find that teacher accuracy is weakly correlated with student accuracy, while teacher ACE is strongly correlated (R^2 of roughly 0.90–0.92). They then show that temperature-scaling the teacher's logits, which makes the teacher less overconfident without changing its accuracy, consistently improves student performance in standard KD and in the state-of-the-art MLLD method, on CIFAR-100, ImageNet, and COCO detection. The same calibration step also makes the student itself better calibrated. The aut","pith_inferences":["A direct causal test would hold teacher accuracy fixed while varying calibration alone; temperature scaling is close but also changes distribution sharpness, so an experiment with accuracy-matched teachers trained under different calibration objectives would isolate the mechanism.","If calibration is the true driver, then teachers trained with calibration-improving regularizers such as mixup or label smoothing should produce the same distillation lift including for student accuracy; the paper's Table 6 hints at this without fully separating calibration gains from accuracy gains.","The paper's decomposition suggests a sweet spot: because true labels act as a one-hot overconfident component, a slightly underconfident teacher can counterbalance them, which may explain why the authors observe gains even when the teacher becomes slightly underconfident at high temperatures.","The paper's own limitation section restricts the claim to logit distillation; whether teacher calibration also benefits feature-based KD methods remains an open and testable question."],"forward_implications":["Teacher selection can be reframed: prefer the well-calibrated teacher over the highest-accuracy teacher, using teacher ACE as an offline, cheap selection criterion.","Any logit-based distillation pipeline can be upgraded at near-zero cost by temperature-calibrating the teacher before training, and the gain stacks with strong methods such as MLLD.","Students distilled from calibrated teachers inherit better calibration themselves, which is valuable when the compressed model is deployed in safety-sensitive settings.","The effect transfers across tasks — classification and two-stage detection — so the principle is not tied to a single architecture or dataset."],"supporting_citations":[{"why":"Supplies temperature scaling as the calibration method and documents that modern neural networks are overconfident, motivating the intervention.","marker":"[4]"},{"why":"Defines the Adaptive Calibration Error (ACE) metric used to measure teacher calibration.","marker":"[5]"},{"why":"Introduces the knowledge distillation loss and teacher–student framework that the paper modifies by calibrating the teacher.","marker":"[6]"},{"why":"Shows that a higher-accuracy teacher does not always produce a better student, motivating the search for a teacher-quality criterion such as calibration.","marker":"[1]"},{"why":"Provides the decoupled KD training scheme and baseline results used for the CIFAR-100 and ImageNet experiments.","marker":"[16]"},{"why":"State-of-the-art logit distillation method (MLLD) to which the calibration step is added, demonstrating compatibility.","marker":"[13]"},{"why":"State-of-the-art feature distillation method (ReviewKD) and the detection baseline used to test the calibration step on COCO.","marker":"[14]"},{"why":"Earlier work connecting knowledge distillation and model calibration that the paper extends across tasks and methods.","marker":"[20]"}],"fun_headline_variants":["Calibrated teachers produce better distilled students","Teacher calibration, not accuracy, drives distillation gains","Simple temperature scaling improves teacher distillation","Well-calibrated teachers improve student models in KD"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that temperature scaling improves student performance by lowering the teacher's calibration error rather than by merely softening the probability distribution; this is not fully isolated in the paper, and the authors' own Table 5 shows a case where teacher ACE rises while the student still improves.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated teachers produce better distilled students","Teacher calibration, not accuracy, drives distillation gains","Simple temperature scaling improves teacher distillation","Well-calibrated teachers improve student models in KD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2337,"prompt_tokens":658,"completion_tokens":1679,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":402,"completion_tokens_details":{"reasoning_tokens":1635}},"tokens_in":402,"tokens_out":1679,"duration_ms":16248,"temperature":1.0,"reasoning_tokens":1635,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:13:27.977394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train students with two teachers of equal validation accuracy but different calibration error — for example, a standard teacher and an accuracy-matched teacher trained with label smoothing or mixup. If student accuracy does not track teacher ACE when teacher accuracy is held constant, the calibration-causes-gains claim is falsified. Similarly, if temperature-scaling an already well-calibrated teacher still improves the student, then softening alone, not calibration, is the active ingredient.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies temperature scaling as the calibration method and documents that modern neural networks are overconfident, motivating the intervention."},{"cited_title":"Nixon, M","cited_arxiv_id":null,"evidence_quote":"Defines the Adaptive Calibration Error (ACE) metric used to measure teacher calibration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that a higher-accuracy teacher does not always produce a better student, motivating the search for a teacher-quality criterion such as calibration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the decoupled KD training scheme and baseline results used for the CIFAR-100 and ImageNet experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"State-of-the-art logit distillation method (MLLD) to which the calibration step is added, demonstrating compatibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"State-of-the-art feature distillation method (ReviewKD) and the detection baseline used to test the calibration step on COCO."},{"cited_title":"Rethinking the Knowledge Distillation From the Perspective of Model Calibration","cited_arxiv_id":"2111.01684","evidence_quote":"Earlier work connecting knowledge distillation and model calibration that the paper extends across tasks and methods."}],"review_version":1}