{"id":"34a6b516-065d-4757-9470-2bce67176ccb","arxiv_id":"2507.18366","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A LoRA-based distillation framework transfers Bayesian and prompt-ensemble uncertainty from LLM teachers into single-pass softmax or Dirichlet students, with comparable quality and large speedups.","lead":"This paper distills slow, sampling-based uncertainty estimates from LLM teachers into fast single-pass student models, using LoRA and either softmax or Dirichlet output heads. The distilled students roughly match teacher accuracy and calibration on four text classification tasks while cutting inference time by up to tens of times.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD uncertainty gains of Dirichlet students may stem from training dynamics and inductive bias rather than from distilling teacher uncertainty; a label-trained control is missing.","rationale":"The paper is a reasonable empirical contribution: the single-pass Dirichlet student is faster (Table 1) and on most datasets its accuracy, NLL, and Brier scores are at parity with, or better than, the teachers. The mathematics in Eq. (3) and the Dirichlet entropy decomposition are standard and are not the source of my concern. My concern is that the central claim, namely that uncertainty capabilities are inherited from the teacher, rests on comparisons that do not include a non-distillation control. The authors are candid that student improvements over the teacher cannot arise from knowledge transfer alone and are a side effect of early stopping and architecture (Sections 4.2 and 4.3), and for the Laplace teacher the student is initialized from the teacher's LoRA adapters (Section 3.1). These concessions are exactly why the headline claim is underdetermined: the same results could be obtained by a student that never saw teacher samples. A label-trained control and a random-init distillation control would settle the attribution. This does not undermine the practical speed/accuracy trade-off, so the conditional verdict is appropriate; the conditions should explicitly include these controls.","tokens_in":17688,"tokens_out":9771,"duration_ms":106934,"concrete_test":"Run a control: fine-tune the same Mistral-7B backbone with the same Dirichlet head and LoRA configuration directly on the Amazon training set using ground-truth labels (standard cross-entropy on the Dirichlet mean), with no teacher samples, no teacher initialization, and the same early-stopping rule. Evaluate accuracy, ECE, NLL, Brier, and OOD AUROC/W1 on the same SST2, YouTube, and Yahoo splits used in Table 4. If the control matches or exceeds the distilled Dirichlet student, the reported OOD and calibration gains cannot be attributed to uncertainty distillation. Also run one BayesPE distillation (e.g., SST2) and one Laplace distillation (e.g., Amazon) from random LoRA initialization to quantify the Section 3.1 initialization confound separately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest evidence for the central claim is Section 4.3, Table 4, where Dirichlet students distilled from the BayesPE teacher show substantially better OOD detection than the teacher (e.g., SST2 total-entropy AUROC 0.82 vs 0.40; Yahoo 1.00 vs 0.90). The paper itself concedes in Sections 4.2-4.3 that such improvements cannot arise from knowledge transfer alone and calls them a 'side effect' of early stopping and model architecture. This concession means the experiments do not isolate the distilled quantity. For the Laplace teacher, the situation is further confounded by initializing the student from the teacher's LoRA adapters (Section 3.1), and early stopping is based on training-data NLL (Section 3.3). The design lacks a control student trained on the same in-domain data with ground-truth labels and no teacher signal, and a distillation run from random LoRA initialization. Without these controls, the reported ECE, NLL, and OOD AUROC cannot be attributed to the teacher's predictive distribution; they may reflect the Dirichlet head's inductive bias or early-stopping dynamics. Since OOD inputs never appear in the distillation loss, the student's OOD behavior is not direct evidence of uncertainty transfer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes distilling sampling-based uncertainty-aware LLM teachers (BayesPE and Laplace LoRA) into single-pass students with either softmax or Dirichlet output heads, using LoRA fine-tuning and an evidential negative-log-likelihood distillation loss. Experiments on four text classification datasets compare accuracy, ECE, NLL, Brier score, OOD detection AUROC, and inference time. The central empirical claim is that Dirichlet students match or improve on the teacher's predictive accuracy and uncertainty quantification while requiring only a single forward pass, with 11x-36x speedups reported for the BayesPE teacher.","tokens_in":17928,"tokens_out":7272,"duration_ms":75850,"significance":"If the result holds, the approach is a practical step toward making Bayesian and prompt-ensemble uncertainty quantification affordable at LLM inference time, and the Dirichlet head's explicit aleatoric/epistemic decomposition is an attractive design. The paper has concrete strengths: public code, two teacher families, four datasets, three student seeds, detailed ablations on regularization and prompt weights, and an honest acknowledgment of the YouTube calibration exception and of the 'side effect' explanation for OOD gains. The main weakness is attribution: the current experiments do not cleanly separate the student's uncertainty behavior that comes from the teacher's predictive distribution from behavior induced by shared initialization, label-based early stopping, or the Dirichlet architecture. The requested control experiments are feasible and would substantially strengthen the central claim.","major_comments":[{"comment":"The Laplace-teacher experiments are confounded by the student initialization described in Section 3.1: the student copies the teacher's LoRA adapters and shares the same Mistral backbone. As a result, the student starts at the teacher's fine-tuned solution, so the near-parity in accuracy, ECE, NLL, and Brier reported in Table 3 is expected even if the distillation losses contributed little. To make the uncertainty-transfer claim load-bearing, the authors should add a control with random LoRA initialization for the student and, ideally, a student distilled from a different LoRA seed of the teacher; without this, the Laplace results cannot be attributed to the distilled predictive distribution.","section":"Section 3.1 and Tables 2-3"},{"comment":"The OOD detection evidence in Section 4.3 and Table 4 does not isolate the teacher's uncertainty. Inputs from SST2, YouTube, and Yahoo never appear in the distillation losses (Eqs. 2 and 3), so the high AUROC values of the Dirichlet student may reflect the Dirichlet head's inductive bias or early-stopping dynamics rather than transferred uncertainty; the paper itself says the student's better-than-teacher OOD behavior can be explained only as a side effect of the distillation process and model architecture. A control Dirichlet student trained on the same in-domain labels without any teacher signal, and preferably a student distilled from a teacher with permuted uncertainty estimates, is required before Table 4 can support the claim that uncertainty is inherited from the teacher.","section":"Section 4.3 and Table 4"},{"comment":"The early-stopping rule in Section 3.3 uses ground-truth labels through the training-data NLL, so model selection is not purely teacher-driven. This is a second channel, in addition to the initialization issue, through which the student's improved ECE, NLL, and OOD AUROC can arise independently of the teacher's predictive distribution. The requested label-only and random-initialization controls should use the same early-stopping rule, and the paper should report the early-stopping metric on those controls, so the contribution of checkpoint selection is quantified.","section":"Section 3.3 and Tables 2-4"}],"minor_comments":[{"comment":"The Dirichlet student's ECE of 0.097 +/- 0.023 is about three times worse than the BayesPE teacher's 0.031, while the Softmax student's ECE is 0.015; the conclusion that Dirichlet students achieve calibration comparable to the teacher should explicitly except this dataset.","section":"Table 2, YouTube row"},{"comment":"The teacher rows are single point estimates with no standard deviations, so the 'comparable' claims rest on deterministic teacher evaluations; a few repeated teacher runs or an explicit statement that the teacher metrics are deterministic under the fixed trained model would strengthen the comparison.","section":"Tables 2 and 3"},{"comment":"The 11x-36x speed-up numbers are reported only for the BayesPE teacher; the Laplace teacher's timings are omitted because of the conservative 10k-sample setting. The abstract and conclusion should state that the speed-up claim is for the BayesPE teacher.","section":"Section 4.1"},{"comment":"The student is always queried with the highest-weight prompt Q_best, a design choice that is not highlighted in the main text; given the prompt-sensitivity results in Section 4.5, this choice and its effect on the headline results should be discussed.","section":"Supplement, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the code release is a genuine strength. The key decision hinge is whether the control experiments can separate the distillation signal from the initialization and early-stopping effects; if the authors provide them and the results change, the claims will need to be reformulated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this is a solid empirical paper with a clear, narrow claim: you can distill sampling-based LLM teachers (BayesPE, Laplace LoRA) into single-pass LoRA students and get accuracy, ECE, NLL, and Brier scores that are comparable to the teacher at 11-36x lower inference cost. The Dirichlet-headed student is the interesting part because it preserves a usable epistemic/aleatoric split that the softmax student cannot. That combination — evidential head plus LoRA plus distillation from MC-sampling teachers — is genuinely new relative to the cited [23] and [25]. The experiments are well-designed: four classification datasets, two teacher types, three seeds, ablations on regularization and prompt impact, and public code. I believe the central empirical result: for in-distribution calibration and accuracy, the students match or slightly beat the teachers. The YouTube ECE exception (0.097 vs 0.031) is honestly reported and the paper flags it.\n\nThe soft spots are real but not fatal. First, the teacher numbers in Tables 2-3 have no error bars, so the parity claims are not as tight as the student numbers. Second, the OOD section is the weakest link. The Dirichlet student's AUROC gains over the BayesPE teacher (e.g., SST2 0.82 vs 0.40) cannot be attributed to knowledge transfer alone, because OOD inputs never appear in the distillation loss and the student is initialized from the teacher's LoRA adapters. The paper itself concedes these gains are a 'side effect' of early stopping and architecture. What is missing is a label-trained control student (same backbone, no teacher signal) and a distillation run from random LoRA initialization. Without those, the OOD improvements remain an interesting but unproven byproduct, not evidence of uncertainty transfer. Third, the fixed global alpha0=5K is selected after seeing results; the supplement admits there is no principled way to choose it.\n\nI would not desk-reject this. The main recipe — evidential distillation into single-pass students — is useful and the in-distribution calibration results are adequate support. The OOD claims need to be toned down or controlled. This deserves a serious referee.","headline":"A practical distillation recipe with honest limitations; the core speed/calibration result holds, but the OOD attribution is under-determined by the current experiments.","tokens_in":18487,"tokens_out":1852,"would_cite":true,"duration_ms":18614,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Dirichlet-headed student LLM can inherit a sampling-based teacher's uncertainty and run a single forward pass, matching accuracy and calibration on classification benchmarks.","keywords":["knowledge distillation","evidential deep learning","uncertainty quantification","large language models","Dirichlet distribution","LoRA","Bayesian prompt ensembles","out-of-distribution detection"],"falsifier":"Train a Dirichlet student on data with known, controllable label noise and distribution shift, and compare its estimated epistemic uncertainty against the true shift magnitude; if the student's epistemic component does not track the true shift while the teacher's does, the claim that the single-pass student preserves the uncertainty decomposition is falsified.","tokens_in":17496,"feed_emoji":"⚡","tokens_out":4346,"duration_ms":41085,"temperature":0.7,"pith_summary":"This paper tries to show that the expensive uncertainty estimates of sampling-based LLM teachers can be compressed into a single forward pass without losing predictive performance or calibration. It compares two student designs: a standard softmax head that learns only the teacher's mean probabilities, and a Dirichlet (evidential) head that also learns the shape of the teacher's predictive distribution. On four text-classification benchmarks, distilled students match or slightly beat their teachers on accuracy, calibration, NLL, and Brier score, while cutting inference time by 11x to 36x. The Dirichlet student also separates aleatoric from epistemic uncertainty and shows strong out-of-distribution detection, which the softmax student cannot do.","feed_headline":"Single-pass students match sampling-based LLM uncertainty","feed_subtitle":"Evidential Dirichlet outputs move aleatoric and epistemic uncertainty from multi-pass teachers into one pass.","key_machinery":"The Dirichlet output head parameterized by concentration parameters $\\alpha_c = 1 + \\mathrm{softplus}(z_c)$, with total evidence $\\alpha_0 = \\sum_c \\alpha_c$; the predictive mean is $\\alpha_c/\\alpha_0$ and the concentration measures epistemic confidence. A negative log-likelihood loss on teacher probability samples trains this head, so the student's Dirichlet shape, not just its mean, encodes the teacher's uncertainty. Entropy decomposition via the law of total entropy splits total, aleatoric ($\\mathbb{E}_p[H[Y|p]]$), and epistemic (mutual information) components. LoRA adapters and NLL-based early stopping carry the distillation.","core_discovery":"The central claim is that distillation based on evidential learning lets a single-pass LLM student recover both the predictions and the uncertainty decomposition of a multi-pass Bayesian or prompt-ensemble teacher. Teacher hypotheses $\\{\\theta_n\\}$ with weights $w_n$ define a Monte Carlo predictive distribution; the softmax student fits the weighted mean $\\bar{p}_{T,c}$, while the Dirichlet student fits the parameters $\\alpha$ of a Dirichlet distribution by maximizing the log-likelihood of the sampled probability vectors. The paper finds that the Dirichlet student matches teacher accuracy (and exceeds it on some datasets), reduces expected calibration error substantially in several cases, and produces higher out-of-distribution AUROC than the teacher itself, while running a single forward pass.","pith_inferences":["Going beyond the paper, a testable extension is to compare the Dirichlet student's epistemic uncertainty against a gold-standard measure, such as disagreement among independently trained models, to check whether the transferred epistemic component is faithful rather than merely well-calibrated in aggregate.","Because the teacher's sampled prompts and weights define the distillation target, and because the student is initialized from the teacher's LoRA matrices, the student may inherit teacher-specific biases; a clean test would distill from a teacher with a known analytic predictive distribution and check exact recovery.","The authors restrict experiments to classification; one could extend the evidential head to regression or sequence outputs, for instance with an evidential Gaussian output layer, to see whether single-pass epistemic uncertainty transfers in those settings as well.","The regularization study shows the optimal fixed $\\alpha_0$ varies by dataset, so a practical follow-up is a hyperprior over $\\alpha_0$ that adapts per domain without manual tuning."],"forward_implications":["Distilled students run inference in a single forward pass yet match or exceed multi-pass teachers on classification accuracy, ECE, NLL, and Brier score (Tables 2-3).","The Dirichlet student is the only variant that retains a usable epistemic/aleatoric split, enabling out-of-distribution detection with AUROC up to 1.00 on the tested shifts (Table 4).","Speed-ups of 11x to 36x over the BayesPE teacher are reported on the four datasets, with larger relative gains on larger datasets.","The distillation process itself can improve on the teacher: students beat their teacher on accuracy in some datasets, attributed to early stopping and the regularizing effect of the Dirichlet distribution.","Softmax students remain competitive when the teacher is Laplace-based, suggesting a simpler parameterization suffices when the teacher's predictive distribution is well-behaved."],"supporting_citations":[{"why":"BayesPE teacher: supplies the prompt-ensemble predictive distribution and learned prompt weights that the students are distilled from.","marker":"[21]"},{"why":"Laplace LoRA teacher: provides the Bayesian teacher over LoRA weights from which students are distilled.","marker":"[29]"},{"why":"Evidential deep learning: provides the Dirichlet output parameterization and the evidential loss used for the student.","marker":"[17]"},{"why":"BNN-plus-evidential distillation: prior work combining Bayesian teachers and Dirichlet students, which the paper extends to LoRA-fine-tuned LLMs and soft-label-only training.","marker":"[25]"},{"why":"Parallel distillation of Bayesian LLMs: the closest competing approach that distills only to softmax students and is contrasted with the evidential variant.","marker":"[23]"},{"why":"LoRA: the low-rank adaptation mechanism used to fine-tune both teachers and students efficiently.","marker":"[10]"},{"why":"Prior-network entropy decomposition: supplies the law-of-total-entropy split into aleatoric and epistemic uncertainty used to evaluate the students.","marker":"[13]"}],"fun_headline_variants":["Evidential distillation yields one-pass LLM uncertainty","Single-pass students inherit teacher uncertainty via Dirichlet","LoRA students match multi-pass uncertainty in one forward pass","Speed up LLM uncertainty with evidential distillation","Distilling multi-pass uncertainty into a single LoRA pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The teacher's Monte Carlo samples and prompt weights are assumed to fully capture its predictive uncertainty, so a student trained on only those samples, from a single prompt, can recover the same aleatoric and epistemic decomposition.","fun_headline_variants_meta":{"raw":{"variants":["Evidential distillation yields one-pass LLM uncertainty","Single-pass students inherit teacher uncertainty via Dirichlet","LoRA students match multi-pass uncertainty in one forward pass","Speed up LLM uncertainty with evidential distillation","Distilling multi-pass uncertainty into a single LoRA pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00031,"raw_usage":{"total_tokens":1709,"prompt_tokens":826,"completion_tokens":883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":808}},"tokens_in":442,"tokens_out":883,"duration_ms":7253,"temperature":1.0,"reasoning_tokens":808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:13:15.843313+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a Dirichlet student on data with known, controllable label noise and distribution shift, and compare its estimated epistemic uncertainty against the true shift magnitude; if the student's epistemic component does not track the true shift while the teacher's does, the claim that the single-pass student preserves the uncertainty decomposition is falsified.","supporting_citations":[{"cited_title":"In: Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., Sun, Y","cited_arxiv_id":null,"evidence_quote":"Laplace LoRA teacher: provides the Bayesian teacher over LoRA weights from which students are distilled."},{"cited_title":"In: Proceedings of the 32nd International Conference on Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Evidential deep learning: provides the Dirichlet output parameterization and the evidential loss used for the student."},{"cited_title":"In: Kiyavash, N., Mooij, J.M","cited_arxiv_id":null,"evidence_quote":"BNN-plus-evidential distillation: prior work combining Bayesian teachers and Dirichlet students, which the paper extends to LoRA-fine-tuned LLMs and soft-label-only training."},{"cited_title":"arXiv preprint arXiv:2505.11731 (2025)","cited_arxiv_id":null,"evidence_quote":"Parallel distillation of Bayesian LLMs: the closest competing approach that distills only to softmax students and is contrasted with the evidential variant."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LoRA: the low-rank adaptation mechanism used to fine-tune both teachers and students efficiently."},{"cited_title":"In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R","cited_arxiv_id":null,"evidence_quote":"Prior-network entropy decomposition: supplies the law-of-total-entropy split into aleatoric and epistemic uncertainty used to evaluate the students."}],"review_version":2}