{"id":"8e69622c-eb06-4322-bb3c-4dcd9af2c61a","arxiv_id":"2602.11219","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A credal concept-bottleneck model that supervises aleatoric uncertainty with annotator disagreement and epistemic uncertainty with prediction error yields near-zero correlation between the two uncertainty estimates.","lead":"This paper proposes training separate neural heads for epistemic (model uncertainty) and aleatoric (label ambiguity) uncertainty, using concept-level annotator disagreement as the aleatoric supervision target. On five text benchmarks, the two uncertainty scores become nearly uncorrelated (ρ≈0.05–0.08) while retaining accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MAQA*/AmbigQA* results may be trained-and-tested on the same examples: no train/val split is reported (Table 10), so the headline ρ(U_ale,H[p*]) and AUROC gains could be circular, not evidence of generalization.","rationale":"The reader's verdict CONDITIONAL with moderate confidence is reasonable, but the weakest assumption they cite—uncorrelated supervision targets—is explicitly acknowledged by the paper (Remark 3.2 and the footnote) and is not internally inconsistent. The deeper problem is that the empirical validation on the two 'ground-truth' datasets is under-specified to the point of potential circularity. If MAQA* has no train split, either the model was trained on the evaluation set or trained on another dataset; neither case is described. Additionally, concept supervision is not defined for QA data, raising doubt about whether the CBM was even applicable. The abstract also promises a credal SENN that never appears in the method. These issues are all checkable with code and data, so a conditional acceptance could be appropriate if the authors provide them; but as the manuscript stands, the central claim that aleatoric uncertainty tracks true ambiguity is not supported. I therefore recommend UNVERDICTED rather than outright rejection: the flaw is potentially fixable, but the current text does not allow a reader to verify the key result.","tokens_in":24894,"tokens_out":9138,"duration_ms":87789,"concrete_test":"Request the exact train/validation/test partition for MAQA* and AmbigQA* (or a 5-fold cross-validation script). Re-run the Variational Credal CBM with the model trained only on the training folds and evaluate on held-out folds. If ρ(U_ale,H[p*]) on held-out questions falls toward 0 (or training is impossible because the required concept labels are not defined for these datasets), then the ground-truth validation claim in Table 11 is circular and should be retracted. Also confirm which of the two models (CBM or SENN) produced the MAQA*/AmbigQA* rows, since the manuscript only describes the CBM.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The most load-bearing issue is the unstated experimental protocol for MAQA* and AmbigQA*. Table 10 lists no train or validation split for these datasets (dashes), yet Table 11 reports test-set metrics. Section 4.1 simply says they 'provide ground-truth answer distributions' and defers to Appendix H, which gives dataset fields but no split or training procedure. The aleatoric head is trained by direct regression to H[p*] (Eq. 9), so if the same examples are used for training and evaluation, ρ(U_ale,H[p*]) ≈ 0.42–0.45 on MAQA*/AmbigQA* is a measure of fit, not of learned generalization. Worse, the CBM architecture requires concept annotations c, and Appendix H never explains what concepts are used for open-domain QA questions or where their labels come from. This is not a small omission: the paper's claim to be 'the first validation of learned aleatoric uncertainty against true ambiguity distributions' depends entirely on these two datasets, and neither the supervision signal nor the train/test separation is specified. The theoretical Corollary 3.3 is also weaker than advertised (its proof in E.2 uses a target t_epi = ψ(err − t_ale) that does not match the actual loss in Eq. 10), but the empirical leakage risk is the more immediate threat. If the authors can supply the split and confirm no test labels were seen during training, the concern is resolved; as written, the key result is unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'structural separation' as a design principle for disentangling epistemic and aleatoric uncertainty in supervised latent-variable models: the two uncertainties are computed from disjoint parameter paths, trained with disjoint supervision signals (prediction error for epistemic, annotator entropy for aleatoric), with gradient isolation enforced by a frozen encoder, orthogonal projections, and stop-gradient operations. The principle is instantiated in a Variational Credal Concept Bottleneck Model, where an ellipsoidal credal set in logit space is parameterized by an epistemic covariance Σ_epi and an aleatoric variance σ_ale. The paper proves a gradient-separation theorem (Thm. 3.1), claims an asymptotic decorrelation corollary (Cor. 3.3), and reports across five benchmarks that the EU–AU correlation drops from ρ≈0.75–0.84 for standard methods to |ρ|≤0.08, while ρ(U_ale,H[p*]) reaches 0.42–0.74 and error-detection AUROC improves, with downstream 'quadrant routing' as a practical application.","tokens_in":25326,"tokens_out":4142,"duration_ms":38017,"significance":"If the empirical claims hold, the paper makes a useful contribution: it offers a concrete architectural recipe for obtaining uncertainty estimates that are not merely decorrelated but are supervised toward semantically distinct targets, and it provides a clean formal statement of gradient isolation. The ablations in Table 3 (trainable encoder vs. frozen encoder, shared vs. disjoint heads) directly support the claim that architectural separation is necessary for the observed decorrelation, which is a valuable and non-obvious point. The use of MAQA*/AmbigQA* as gold-standard ambiguity benchmarks, if the protocol is sound, would be a notable validation step. However, the empirical core is currently under-supported by the reported experimental protocol and is clouded by several inconsistent numbers.","major_comments":[{"comment":"The experimental protocol for MAQA* and AmbigQA* is unspecified in a load-bearing way. Table 10 lists no train or validation split for these datasets (dashes), yet Table 2 and Table 11 report test-set AUROC and ρ(U_ale,H[p*]). The aleatoric head is trained by direct regression onto H[p*] (Eq. 9). If the same examples are used for training and evaluation, the reported ρ(U_ale,H[p*])≈0.42–0.45 and the AUROC gains are measures of fit, not of learned generalization. Appendix H.3 describes the dataset fields and the co-occurrence construction of p* but does not state how examples were split, how the CBM concept annotations were defined for open-domain QA, or what training labels were used for the concept head. This must be clarified; otherwise the paper's headline claim of 'first validation of learned aleatoric uncertainty against true ambiguity distributions' is unverifiable.","section":"§4.1, Table 10, Appendix H.3"},{"comment":"The reported headline numbers are internally inconsistent. For MAQA*, Variational Credal CBM has AUROC 0.76 in Table 2 but 0.74 in Table 11. More seriously, ρ(U_ale,H[p*]) on MAQA* is 0.45 in Table 2, but §3.4 and Fig. 3(c) report ρ=0.78 with the claim that 'aleatoric uncertainty tracks annotator entropy.' These are different claims about the same quantity on the same dataset. The paper must state which number is correct, reconcile the discrepancy, and explain how the 0.78 value was computed. As written, a reader cannot determine the actual performance of the method on its key validation metric.","section":"Table 2 vs. Table 11 vs. Fig. 3"},{"comment":"The role of the decorrelation penalty is contradictory and affects the central claim. Remark 3.2 and Table 3 state that λ_d=0 by default and that structural separation alone achieves |ρ|<0.1. However, Table 9 lists λ_d=5.0 for CEBaB and MAQA, and Table 13 lists λ_d=0.1 for all datasets. The main objective Eq. (11) does not contain L_decorr, while the appendix objective Eq. (14) does, and Algorithm 1 omits it. If λ_d>0 was used in the reported experiments, then (a) the claim that the results were obtained by 'structural separation alone' is false, and (b) Theorem 3.1's clean gradient-isolation statement is violated, because L_decorr deliberately couples the heads through a shared gradient signal (Remark 3.2 concedes this). The authors must specify exactly which objective was optimized for each reported result and whether the decorrelation penalty was active.","section":"Eq. (11), Eq. (14), §3.2, Remark 3.2, Tables 9 and 13"},{"comment":"The proof of Corollary 3.3 does not match the loss actually used. Appendix E.2 defines the epistemic target as t_epi = ψ(err − σ*_ale), i.e., an error signal with the aleatoric contribution removed. But Eq. (10) defines the epistemic supervision target as φ(|p̂−c|_sg), with no subtraction of an aleatoric term. The proof's conclusion ρ(σ_epi,σ_ale)→0 therefore relies on a target that is not the one minimized in training. Additionally, condition (ii) — that prediction errors and annotator entropy are approximately uncorrelated in the data — is asserted, not tested, and condition (iii) ('orthogonal projections extract sufficiently distinct features') is not formalized. The theoretical support for the empirical decorrelation claim needs to be aligned with the actual objective and either proved under explicit, verifiable conditions or empirically checked.","section":"Appendix E.2, Eq. (10), Corollary 3.3"}],"minor_comments":[{"comment":"Algorithm 1 omits L_concept, L_orth, and the Hausdorff KL term βD_H^+, despite these appearing in the full objective Eq. (11) and in the gradient-separation proof. Please make the training algorithm consistent with the stated objective.","section":"Algorithm 1 vs. Eq. (11)"},{"comment":"Appendix G.2 describes early stopping on 'validation ρ(AU,H)', but Table 10 reports no validation set for MAQA* and AmbigQA*. Clarify what validation data (if any) was used for these datasets.","section":"Appendix G.2 vs. Table 10"},{"comment":"The ablation row 'No decorr penalty (λ_d=0)' is confusing when the default is stated to be λ_d=0 in Remark 3.2. Please define the default configuration explicitly and consistently across the main text, tables, and appendices.","section":"Table 3 caption and §4.3"},{"comment":"The discussion of limitations appropriately acknowledges that co-occurrence statistics are an approximation to true p*. It would be helpful to state explicitly that this approximation is inherited by the reported ρ(U_ale,H[p*]) values.","section":"Appendix H.3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and has a plausible, well-motivated architectural idea. The gradient-separation result and the ablations are valuable. However, the empirical core currently rests on an unstated MAQA*/AmbigQA* protocol, and the inconsistency between the reported AUROC/ρ values and the role of λ_d makes the headline claims impossible to verify as written. I would like to see a revised version that supplies the train/val/test split details, confirms that no test H[p*] values were used during training, reconciles Tables 2 and 11 with Fig. 3, and unambiguously specifies the objective used in each experiment. If the split details cannot be provided, the MAQA*/AmbigQA* results should be reframed as preliminary or removed from the main claims. The remaining issues are fixable within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a genuinely useful idea — train epistemic and aleatoric uncertainty heads on disjoint parameters, disjoint losses, and stop-gradient/frozen encoder so their training signals cannot leak. The ablation supports the claim: trainable encoder or shared heads leave EU–AU correlation at 0.42–0.68, frozen + disjoint gets it below 0.1. That is a real empirical finding, and the quadrant-routing story is a nice way to show why it matters. The paper is also honestly hedged about needing concept labels and about the co-occurrence approximation to p*.\n\nThe soft spots are not minor, though. The \"theoretical guarantee\" is almost a tautology: Corollary 3.3 assumes the supervision targets (error and annotator entropy) are approximately uncorrelated, then concludes the outputs are uncorrelated. That is a conditional statement about the datasets, not a result about the architecture producing decorrelation. Worse, the proof in E.2 uses a target t_epi = ψ(err − t_ale) that does not match the actual loss in Eq. 10. The empirical \"validity\" metrics are also partly regression-to-target: the aleatoric head is trained by least squares to H[p̂*], so ρ(U_ale, H[p*]) around 0.42–0.74 is partly fit, not independent confirmation.\n\nThe biggest concrete problem is MAQA*/AmbigQA*. Table 10 lists no train/val split, yet Table 11 reports test metrics. The appendix describes the dataset fields but never says where concepts come from or how training was separated from evaluation. If the model is trained and evaluated on the same 468/2,553 examples, the headline result is circular. I cannot tell from the manuscript, and the inconsistencies do not help: AUROC for MAQA* is 0.76 in Table 2 and 0.74 in Table 11; ρ(U_ale, H[p*]) is 0.45 in Table 2 and 0.78 in Figure 3; λ_d is 0 by default in Remark 3.2 but 5.0 in Table 9 and 0.1 in Table 13. A reader cannot reproduce this.\n\nVerdict: the architectural principle is worth taking seriously, and the CEBaB/HateXplain/GoEmotions results suggest it works when the supervision signal is genuine annotator disagreement. But the paper needs code/data, the tables reconciled, and the MAQA*/AmbigQA* protocol specified. As it stands, I would not cite the headline numbers. I would still send it to peer review — the idea is important enough that referees should see it and push for the fixes.","headline":"The architectural idea is real and worth building on, but the key validation on MAQA*/AmbigQA* is unverifiable as written and the theory is close to tautological.","tokens_in":25862,"tokens_out":3821,"would_cite":false,"duration_ms":38800,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that epistemic and aleatoric uncertainty become operationally separable when each is computed from its own parameter path with its own supervision signal, and proves a gradient-separation theorem that breaks the coupling t","keywords":["uncertainty decomposition","epistemic uncertainty","aleatoric uncertainty","credal sets","concept bottleneck models","gradient isolation","annotator disagreement","structural separation"],"falsifier":"Construct a benchmark where ambiguous questions (high annotator entropy) are also the ones the model most often answers incorrectly, train the same architecture, and check whether the epistemic–aleatoric correlation stays below 0.1; if it rises, the decorrelation is conditional on the supervision targets being uncorrelated rather than being a pure architectural guarantee.","tokens_in":24698,"feed_emoji":"🎯","tokens_out":4861,"duration_ms":47123,"temperature":0.7,"pith_summary":"The paper argues that the usual way of estimating epistemic and aleatoric uncertainty—deriving both from the same predictive distribution—is why the two estimates are strongly correlated and blurred in meaning. It proposes a design principle called structural separation: compute the two uncertainties from disjoint parameter paths trained with disjoint loss signals, so the model cannot learn them from the same gradient. Instantiating this in a credal-set concept bottleneck model, the authors prove a gradient-separation theorem and show empirically that epistemic–aleatoric correlation drops to near zero across five benchmarks while accuracy is preserved and aleatoric estimates track true ambiguity. If right, this gives a practical route to making \"the model doesn't know\" and \"the question is ambiguous\" operationally distinct.","feed_headline":"Decoupled uncertainty: model doubt vs question ambiguity","feed_subtitle":"Separate heads cut EU–AU correlation from ~0.8 to ≤0.08, keeping accuracy intact.","key_machinery":"The central object is a credal set—a set of plausible predictive distributions rather than a single point—parameterized as an ellipsoid in logit space. The epistemic head controls the ellipsoid's covariance (set size) and is supervised by rescaled prediction error plus a Hausdorff KL regularizer; the aleatoric head controls a separate variance term and is supervised directly by annotator entropy. The gradient-separation theorem rests on three enablers: a frozen encoder so no shared gradient path flows back, orthogonal projections so each head reads a distinct subspace, and stop-gradient in the epistemic loss so it cannot game the error signal. Together these make the two uncertainty estimate","core_discovery":"The central claim is Theorem 3.1: with a frozen encoder, orthogonal projections of the shared representation, and stop-gradient in the epistemic loss, the aleatoric parameters receive gradients only from the aleatoric loss and the epistemic parameters only from the epistemic loss. This structural guarantee, the paper argues, lets the two uncertainty heads decorrelate by construction rather than through post hoc decomposition. Empirically, across five ambiguity-aware benchmarks, the method reduces the Spearman correlation between epistemic and aleatoric uncertainty from roughly 0.75–0.85 for standard methods to at most 0.08, while preserving or slightly improving accuracy, improving aleatoric","pith_inferences":["As an editorial extension: the strong reported correlation between aleatoric uncertainty and true ambiguity is partly by construction, because that head is regressed directly onto annotator entropy; the more novel claim is that this can happen without reintroducing correlation with the epistemic head.","As an editorial extension: the decorrelation guarantee depends on the two supervision targets being uncorrelated in the data; if ambiguous examples are also the ones the model errs on, the method will inherit that correlation no matter how clean the gradients are.","As an editorial extension: the same recipe—frozen backbone, orthogonal heads, separate supervision—should transfer beyond concept bottleneck models to other multi-head architectures, suggesting a testable general principle for uncertainty decomposition.","As an editorial extension: the ground-truth ambiguity used for validation is itself a proxy (corpus co-occurrence), so a human-study validation with explicit ambiguity judgments would be a stronger test of whether the aleatoric head captures genuine semantic ambiguity."],"forward_implications":["Uncertainty decomposition becomes actionably distinct: high-epistemic, low-aleatoric cases can be routed to data collection, while low-epistemic, high-aleatoric cases can be routed to human review.","Error detection degrades far less in the high-ambiguity regime (roughly half the AUROC drop of baselines), because epistemic uncertainty no longer gets diluted by aleatoric noise.","Aleatoric uncertainty can be validated against ground-truth ambiguity in benchmarks, improving from roughly 0.2 to 0.42–0.74, making the decomposition testable rather than merely asserted.","The decorrelation is architectural rather than a post-processing trick, holding across random seeds and across different frozen encoders.","The method requires multi-annotator concept labels; without the aleatoric supervision signal, decorrelation remains but aleatoric validity drops substantially."],"fun_headline_variants":["Uncertainty, decoupled: separate paths for model doubt and label ambiguity","Gradient isolation yields independent epistemic and aleatoric uncertainty","Structural separation cuts EU–AU correlation to ≤0.08 without accuracy loss","Two heads, one model: supervised latent variables split uncertainty types","Epistemic and aleatoric uncertainty decoupled by design, not post hoc"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The two supervision targets—rescaled prediction error and annotator entropy—must be approximately uncorrelated in the data distribution; if they are correlated, the two uncertainty heads will output correlated estimates no matter how cleanly their gradients are separated, and the empirical decorrelation will vanish.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty, decoupled: separate paths for model doubt and label ambiguity","Gradient isolation yields independent epistemic and aleatoric uncertainty","Structural separation cuts EU–AU correlation to ≤0.08 without accuracy loss","Two heads, one model: supervised latent variables split uncertainty types","Epistemic and aleatoric uncertainty decoupled by design, not post hoc"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2721,"prompt_tokens":710,"completion_tokens":2011,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":1930}},"tokens_in":454,"tokens_out":2011,"duration_ms":12986,"temperature":1.0,"reasoning_tokens":1930,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:59:21.488110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a benchmark where ambiguous questions (high annotator entropy) are also the ones the model most often answers incorrectly, train the same architecture, and check whether the epistemic–aleatoric correlation stays below 0.1; if it rises, the decorrelation is conditional on the supervision targets being uncorrelated rather than being a pure architectural guarantee.","supporting_citations":[],"review_version":1}