{"id":"68e464df-2ff6-47a3-9d6c-19fce13b16a7","arxiv_id":"2507.08150","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CLEAR jointly calibrates aleatoric and epistemic uncertainty estimates with two parameters and produces narrower calibrated prediction intervals across 17 regression datasets.","lead":"A new calibration method, CLEAR, combines two kinds of uncertainty estimates, aleatoric (noise) and epistemic (model uncertainty), using two tunable weights instead of one, and reports narrower prediction intervals on 17 regression benchmarks. It matters because it offers a model-agnostic way to improve conditional coverage and interval sharpness for any pair of uncertainty estimators, including tree ensembles and deep ensembles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline width gains are NCIW, a test-time-calibrated metric; raw-NIW and AISL averages should be reported before claiming 28.3%/17.5% interval-width improvements.","rationale":"The reader's weakest assumption concerns residual-based aleatoric estimation, which is a real limitation for conditional-coverage claims in extrapolation. However, the paper's central advertised numbers are about width at fixed coverage, and that comparison is mediated by NCIW. The NCIW issue is therefore more directly load-bearing for the abstract's headline claim. I partially agree with the reader because the reader's rationale explicitly flags NCIW, even though the weakest_assumption field points to residual estimation. The paper does contain independent support: raw NIW and quantile loss / AISL tables in Appendix F show CLEAR is usually best, and the conformalized variant restores finite-sample marginal coverage. The issue is not that CLEAR is unhelpful, but that the specific 28.3%/17.5% numbers are NCIW, not raw width. A recomputation of raw-NIW and AISL averages is a cheap, decisive check. If the gains are similar, the concern is mostly cosmetic; if smaller, the abstract should be reworded. Either way, the reader's CONDITIONAL verdict stands, so no verdict change is needed.","tokens_in":64964,"tokens_out":9504,"duration_ms":106253,"concrete_test":"Recompute the 17-dataset average relative improvement of CLEAR over ALEATORIC and PCS using raw NIW (no ctest-cal) from Tables 15 and 17, and using Interval Score Loss / Quantile Loss from Tables 16 and 18; compare these averages with the 28.3% and 17.5% NCIW figures. If the raw-NIW average gains fall substantially below the NCIW figures, the abstract should cite the uncalibrated metric or reword 'interval width' to 'NCIW after test-time rescaling'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central numerical claim reports 28.3% and 17.5% 'interval width' improvements, but these are NCIW averages, and NCIW (Section 3.3) applies a calibration constant ctest-cal chosen on the test set to enforce PICP >= 1 - alpha. This makes the headline comparison a post-hoc rescaled width comparison rather than a strict out-of-sample predictive comparison. The paper's remark that 'methods are already calibrated before evaluation on the test set, resulting in ctest-cal approx 1' does not remove the test-label dependence: any validation-to-test calibration gap is absorbed by ctest-cal and can differentially rescale the baselines. Raw NIW tables in Appendix F.1 support the general direction, and the quantile loss / AISL results provide independent support, but those are different objectives and the advertised 28.3%/17.5% magnitudes are not recomputed without ctest-cal. The central claim's magnitude is therefore not fully established as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLEAR, a calibration method that combines an aleatoric uncertainty estimator (quantile regression on residuals) and an epistemic uncertainty estimator (PCS ensembles) by jointly calibrating two parameters, γ1 and γ2 = λγ1, so as to maintain nominal coverage while reducing interval width. Across 17 real-world regression datasets, the abstract reports average NCIW improvements of 28.3% and 17.5% over the two individually calibrated baselines, with similar gains for Deep Ensembles and Simultaneous Quantile Regression. The paper also provides asymptotic conditional coverage theory, a conformalized variant with finite-sample marginal coverage guarantees, extensive appendix tables, and a case study on the Ames Housing data.","tokens_in":65145,"tokens_out":6471,"duration_ms":82693,"significance":"If the reported gains hold as stated, the work is a useful contribution to regression UQ: it demonstrates a simple, estimator-agnostic mechanism for reweighting aleatoric and epistemic components, and it ships a large empirical study (17 datasets, 10 seeds, three model variants, separate conformalized results) with code. The paper is unusually transparent about its main limitation—reusing Dval as Dcal in the default procedure—and it explicitly separates the conformalized variant. The residual-based aleatoric estimator is a practically motivated idea, and the empirical evidence that the joint calibration improves over both single-component baselines is directionally consistent across raw NIW, quantile loss, and AISL tables. The main weakness is that the headline width improvements are computed with a test-time calibration constant (NCIW), so the abstract's 'interval width' claim is not exactly what was measured; this is fixable by reporting raw-width aggregates in the main text and tempering the wording.","major_comments":[{"comment":"The headline numbers 28.3% and 17.5% are NCIW averages, not raw interval-width improvements. NCIW applies a calibration constant ctest-cal chosen on the test set to enforce PICP ≥ 1 − α. This makes the comparison a post-hoc rescaled width comparison; a method with a validation-to-test calibration gap can be differentially rescaled, and the statement that ctest-cal ≈ 1 because methods are already calibrated does not remove the test-label dependence. The raw NIW tables in Appendix F.1 support the general direction, but the advertised magnitudes are not recomputed without ctest-cal. Please report aggregate raw NIW, quantile loss, and AISL for the PCS/ALEATORIC/ALEATORIC-R comparison in the main text and qualify the abstract accordingly.","section":"Abstract and §3.3"},{"comment":"In the default configuration the algorithm sets Dcal = Dval and optimizes λ on the same dataset, so the finite-sample conformal coverage guarantee of Lemma B.2 does not apply. The paper discloses this in Appendix B.1 and provides a conformalized version in Appendix G, which is commendable. However, the abstract's 'while maintaining nominal coverage' is stated for the standard configuration, where coverage is only empirical. Please either make the conformalized variant the primary comparison or explicitly state in the abstract and Section 4.2 that the standard variant achieves nominal coverage empirically rather than by guarantee.","section":"Algorithm 1 and Appendix B.1"},{"comment":"The residual-based aleatoric estimator is one of the paper's stated contributions, but no theoretical result connects the quantiles of Yi − f̂(Xi) to the conditional aleatoric quantiles when f̂ is not exactly the conditional median. Lemma 2.1 assumes consistency of quantile regression estimators for the conditional quantiles of Y, not for residuals of a possibly biased point predictor. If f̂ is biased or the residual distribution differs from the aleatoric noise distribution, the parameters γ1 and λ can rescale the interval but cannot repair its conditional coverage shape in extrapolation regions. Please add explicit conditions under which residual quantiles equal noise quantiles, or empirically validate the decomposition on synthetic data with known model bias and known noise.","section":"Section 2.3 and Lemma 2.1"}],"minor_comments":[{"comment":"The sentence 'CLEAR (a) was, in fact, the top-performing method on 15 of the 17 datasets' should specify which metric (NCIW, quantile loss, or both) this ranking refers to, since the ranking can differ across metrics.","section":"Section 4.2"},{"comment":"Step 3 defines γ1 as the smallest value satisfying a coverage constraint but does not specify the behavior when no finite γ1 achieves the constraint; Algorithm 2 in Appendix B.5 provides the necessary detail and should be referenced directly from Step 3.","section":"Algorithm 1"},{"comment":"The notation +∞1/10 v for infinitely wide UACQR-P intervals should be explained in the table caption rather than only in the surrounding text.","section":"Table 1"},{"comment":"Lemma B.9 is essentially a no-worse-than-baselines statement because CLEAR's grid contains the baseline parameter choices; this is fine, but the surrounding text could state more explicitly that the result does not imply a strict improvement over the baselines at the population level.","section":"Appendix B.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is largely transparent about its limitations, and the empirical appendix is unusually thorough. The main revision needed is to align the headline claims with the actual metrics: NCIW is not raw interval width, and the default Dcal = Dval procedure does not carry a finite-sample coverage guarantee. These are fixable within the manuscript's scope, so I do not recommend rejection. The paper fits the ICLR audience well; no citation or novelty concerns beyond those already raised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Your colleague should know this: CLEAR is a useful but incremental calibration method that jointly tunes gamma1 and lambda, and it ships with a large, careful benchmark. The headline 28.3%/17.5% width improvements, however, are NCIW numbers—a metric that re-calibrates widths on the test set—so the abstract overstates a strict out-of-sample width gain.\n\nWhat is genuinely new: UACQR fixes gamma1=1 and tunes only lambda; CLEAR tunes both, and the two-parameter search plus residual-based aleatoric quantiles (fitting on Yi - fhat(Xi) rather than on Y directly) is a natural but real step forward. The empirical program is extensive: 17 datasets, 10 seeds, three PCS variants, deep ensembles and SQR, plus a conformalized appendix. Code and data are public. The paper is honest—Appendix B.1 explicitly says the default Dcal=Dval reuse forfeits the finite-sample conformal guarantee, and the conformalized variant in the appendix restores it.\n\nWhere it is soft: the NCIW concern is legitimate. NCIW applies a test-set calibration constant ctest-cal to enforce PICP, so the reported width reduction is a rescaled comparison, not a pure predictive comparison. The argument that ctest-cal≈1 doesn't fully remove the dependence on test labels. Appendix F.1's raw NIW tables support the general direction, but the abstract's magnitudes aren't recomputed without calibration. That's an easy fix: lead with raw NIW or quantile loss, or report both. The Dcal=Dval default is also a real caveat; the main-text results don't carry the conformal guarantee, and while the conformalized appendix supports the same pattern, the claims in the abstract should be consistent with the theoretically clean version. Lemma B.9 is near-tautological (CLEAR's parameter space contains the single-parameter baselines), so it adds little. Finally, the conditional-coverage advantage is demonstrated in simulations but not directly measured on real datasets; the real-data evidence is marginal coverage plus width, so 'improved conditional coverage' is more inference than demonstration. None of these is a load-bearing flaw; they are all addressable.\n\nBottom line: this paper deserves a serious referee. The methodology is transparent, the benchmark is reproducible, and the two-parameter calibration is worth testing in practice. I'd send it out for review, but ask for the headline numbers to be changed to raw NIW (or accompanied by them), and for a clear statement that the default configuration trades the conformal guarantee for data efficiency.","headline":"Solid incremental calibration paper with a big reproducible benchmark; trust the direction, not the NCIW headline numbers.","tokens_in":65677,"tokens_out":3428,"would_cite":true,"duration_ms":36537,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62G05","62F40"],"pacs":[],"model":"deepseek-v4-flash","headline":"CLEAR calibrates aleatoric and epistemic uncertainty with two scaling parameters, cutting 95% interval width by 28.3% on average over single-component baselines while holding coverage.","keywords":["uncertainty quantification","prediction intervals","conformal calibration","aleatoric uncertainty","epistemic uncertainty","quantile regression","PCS framework","regression"],"falsifier":"Run a synthetic experiment with known heteroskedastic noise, deliberately use a median estimator whose bias grows with distance from the training mass, and record CLEAR's conditional coverage as a function of $x$ on a dense grid; if coverage falls below the nominal level in the high-bias regions, the residual-based aleatoric assumption is the source of the failure.","tokens_in":64710,"feed_emoji":"🎯","tokens_out":8883,"duration_ms":88459,"temperature":0.7,"pith_summary":"The paper proposes CLEAR, a calibration scheme for regression prediction intervals that treats aleatoric noise and epistemic model uncertainty as separate components, each with its own scale parameter, and learns the ratio between them from data. Its central claim is that two-parameter calibration produces intervals that are markedly narrower than individually calibrated aleatoric-only or epistemic-only baselines while keeping nominal coverage. Across 17 real datasets, CLEAR reports average normalized interval-width reductions of 28.3% and 17.5% over those two baselines, with the largest gains where the uncertainty sources are unbalanced. The practical payoff is that a practitioner does not have to fix the epistemic-aleatoric ratio by hand: the validation data decides which component dominates, and the interval widens in data-sparse regions where a purely aleatoric interval would undercover.","feed_headline":"Two-parameter calibration narrows prediction intervals by 28 percent","feed_subtitle":"A learned ratio between noise and model uncertainty cuts interval width while keeping coverage.","key_machinery":"The load-bearing object is the two-parameter interval $C(x) = [\\hat f(x) - \\gamma_1 \\hat q_{\\alpha/2}^{ale}(x) - \\gamma_2 \\hat q_{\\alpha/2}^{epi}(x), \\hat f(x) + \\gamma_1 \\hat q_{1-\\alpha/2}^{ale}(x) + \\gamma_2 \\hat q_{1-\\alpha/2}^{epi}(x)]$ with the constraint $\\gamma_2 = \\lambda \\gamma_1$. The aleatoric pieces $\\hat q^{ale}$ are fit by quantile regression on the residuals $Y_i - \\hat f(X_i)$, bagged over bootstrap samples; the epistemic pieces $\\hat q^{epi}$ are the distances from the ensemble median to the ensemble quantiles. Conformal calibration of $\\gamma_1$ enforces marginal coverage, while $\\lambda$ rebalances the two components so that scale and ratio are adjusted separately, which is what lets the method compensate when one uncertainty estimator is miscalibrated.","core_discovery":"CLEAR's central discovery is that the two sources of predictive uncertainty should be calibrated jointly, with two degrees of freedom, instead of separately or with a fixed 1:1 ratio. The interval takes the form $C(x) = [\\hat f(x) - \\gamma_1 \\hat q_{\\alpha/2}^{ale}(x) - \\gamma_2 \\hat q_{\\alpha/2}^{epi}(x), \\hat f(x) + \\gamma_1 \\hat q_{1-\\alpha/2}^{ale}(x) + \\gamma_2 \\hat q_{1-\\alpha/2}^{epi}(x)]$, with $\\gamma_2 = \\lambda \\gamma_1$; $\\gamma_1$ is fixed by the standard conformal coverage calibration on a calibration set, and $\\lambda$ is selected from a grid by minimizing quantile loss on validation data. Aleatoric quantiles are obtained by fitting quantile regression to the residuals $Y_i - \\hat f(X_i)$ and taking medians over bootstrap members, while epistemic quantiles are the ensemble perturbation quantiles. The paper argues and demonstrates empirically that this adaptive reweighting repairs conditional coverage in data-sparse regions without sacrificing sharpness: average NCIW gains of 28.3% over the aleatoric baseline and 17.5% over the epistemic baseline, and similar gains when the components are deep ensembles and simultaneous quantile regression.","pith_inferences":["A natural extension, already flagged in the paper, is to carry the same dual-parameter idea into classification, calibrating a confidence score and an ensemble-dispersion score separately.","Since $\\lambda$ is estimated on finite validation data, small calibration samples will overfit the ratio; regularizing $\\lambda$ toward 1 is a testable modification likely to help in low-data regimes.","The residual-based aleatoric step implies CLEAR is only as good as the median's shape; deliberately misspecifying the median so its bias varies with $x$ is a direct way to probe the limits of the additive interval form.","Under covariate shift, the ratio learned on the validation distribution may need re-estimation; the paper's sphere-of-increasing-radius simulation protocol is a ready-made template for testing how far $\\lambda$ carries."],"forward_implications":["CLEAR can wrap any pair of aleatoric and epistemic estimators, so the same calibration layer can sit on top of tree-based, deep ensemble, or quantile regression models.","The learned ratio $\\lambda$ gives a data-driven statement about which uncertainty source dominates, which can guide active learning and data collection decisions.","Joint calibration is asymptotically no worse than calibrating either component alone, so the second parameter does not sacrifice long-run coverage guarantees.","With a separate calibration split, CLEAR inherits the finite-sample marginal coverage guarantee of split conformal prediction.","The reported benchmark gains (top performer on 15 of 17 datasets in the main configuration) mean the improvement is not confined to one model family."],"supporting_citations":[{"why":"Supplies the conformalized quantile regression baseline and the aleatoric interval construction that CLEAR extends.","marker":"(Romano et al., 2019)"},{"why":"Defines the PCS framework whose ensemble perturbation quantiles provide CLEAR's epistemic uncertainty component.","marker":"(Yu & Kumbier, 2020)"},{"why":"Provides the PCS-UQ calibration approach and the collection of 17 regression datasets used in the benchmarks.","marker":"(Agarwal et al., 2025)"},{"why":"Provides the conformal calibration and coverage theory that underpin Step 3 of the CLEAR algorithm.","marker":"(Angelopoulos et al., 2024)"},{"why":"Supplies quantile random forests, one of the base quantile estimators used for the aleatoric residual model.","marker":"(Meinshausen, 2006)"},{"why":"Defines quantile regression and the quantile loss used both for training and for selecting $\\lambda$.","marker":"(Koenker & Bassett, 1978)"},{"why":"Defines deep ensembles, the alternative epistemic estimator on which CLEAR's generality is tested.","marker":"(Lakshminarayanan et al., 2017)"},{"why":"Defines simultaneous quantile regression, the alternative aleatoric estimator used in the generality experiments.","marker":"(Tagasovska & Lopez-Paz, 2019)"},{"why":"Introduces UACQR, the closest prior method that combines both uncertainties; CLEAR is compared against it directly.","marker":"(Rossellini et al., 2024)"}],"fun_headline_variants":["Joint calibration of noise and model uncertainty sharpens intervals","CLEAR: Two-parameter fix for prediction intervals","Learn the right ratio of aleatoric to epistemic uncertainty","Calibrate both uncertainties at once for tighter intervals","CLEAR's two dials tune intervals to hit coverage targets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing assumption is that the fitted quantiles of the residuals $Y_i - \\hat f(X_i)$ equal the quantiles of the true aleatoric noise; if the ensemble median is biased or the residual spread does not match the true noise spread, the two scaling parameters can resize the interval but not reshape it, and conditional coverage in extrapolation regions can fail.","fun_headline_variants_meta":{"raw":{"variants":["Joint calibration of noise and model uncertainty sharpens intervals","CLEAR: Two-parameter fix for prediction intervals","Learn the right ratio of aleatoric to epistemic uncertainty","Calibrate both uncertainties at once for tighter intervals","CLEAR's two dials tune intervals to hit coverage targets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3239,"prompt_tokens":1047,"completion_tokens":2192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2111}},"tokens_in":663,"tokens_out":2192,"duration_ms":17982,"temperature":1.0,"reasoning_tokens":2111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:28:01.416404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a synthetic experiment with known heteroskedastic noise, deliberately use a median estimator whose bias grows with distance from the training mass, and record CLEAR's conditional coverage as a function of $x$ on a dense grid; if coverage falls below the nominal level in the high-bias regions, the residual-based aleatoric assumption is the source of the failure.","supporting_citations":[{"cited_title":"Similarly, classical Random Forests are known to be consistent under standard assumptions (Scornet et al., 2015)","cited_arxiv_id":null,"evidence_quote":"Supplies quantile random forests, one of the base quantile estimators used for the aleatoric residual model."},{"cited_title":"relative vs. absolute uncertainty","cited_arxiv_id":null,"evidence_quote":"Introduces UACQR, the closest prior method that combines both uncertainties; CLEAR is compared against it directly."}],"review_version":1}