{"id":"e5a51389-7e6d-419e-85a3-250bfb1cdd8b","arxiv_id":"2412.13738","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of quantile regressors plus an iterative data-augmentation algorithm separates aleatoric from epistemic uncertainty in synthetic tasks.","lead":"This paper proposes a method that separates two types of uncertainty in machine learning models by repeatedly adding data in uncertain regions and tracking which uncertainties vanish. The practical hook is that knowing whether uncertainty is due to noise or missing knowledge helps decide where to spend data collection effort.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim that Algorithm 1 separates aleatoric and epistemic uncertainty rests on the unvalidated premise that, after Q progressive-sampling iterations, the total uncertainty map contains only aleatoric uncertainty; the evaluation is purely qualitative, so no evidence supports this…","rationale":"The reader's weakest_assumption correctly identifies the unvalidated premise that epistemic uncertainty vanishes with added data, which is indeed load-bearing. I partially agree but would sharpen the concern to the algorithmic termination condition and the implementation of the common-uncertainty aggregation, since these are concrete and falsifiable, and they make the central claim fail even if the general premise about vanishing epistemic uncertainty is accepted in principle. The paper's own Section 3.1 describes the exact leakage failure mode, which makes the unvalidated termination criterion especially vulnerable. The evaluation is purely qualitative, with no quantitative separation metric and no comparison to any baseline, so the claimed superiority over DE and MC dropout is unsupported. On the positive side, the paper identifies a real phenomenon and the progressive-sampling idea is plausible, but the manuscript as written does not establish the central claim. The verdict should remain REJECT.","tokens_in":6807,"tokens_out":1506,"duration_ms":15175,"concrete_test":"Re-run Algorithm 1 on the toy experiment with a ground-truth aleatoric noise map, and for each iteration, measure the fraction of the initial high-uncertainty regions that remain labeled aleatoric after adding data. If the final total-uncertainty map does not converge to the known aleatoric region within a small number of iterations, or if the XOR map misidentifies known data-poor regions, the separation claim fails. Also verify that the per-output aggregation in the pseudocode (U_j_total multiplication over all outputs) actually computes the 'common uncertainty map' described in Section 3.2; otherwise the algorithm as written does not match the proposed method.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that Algorithm 1 reliably separates aleatoric and epistemic uncertainty. This depends on the assertion, stated in Section 3.2 and embedded in the pseudocode, that after enough progressive-sampling iterations 'the total uncertainty map only contains aleatoric uncertainty' (Algorithm 1, lines before the XOR). This premise is never validated: there is no convergence argument, no stopping criterion for Q, and no quantitative metric for 'epistemic uncertainty has vanished.' In fact, the paper's own Section 3.1 describes the opposite failure mode: with insufficient data, aleatoric uncertainty leaks into epistemic estimates. After adding data only in high-uncertainty regions, the retrained model may still misestimate aleatoric noise in those regions, so the XOR between initial and final maps could misclassify persisting leakage as epistemic. The evaluation is limited to two synthetic tasks with qualitative visual results and no comparison against DE or MC dropout, so the abstract's claim of 'surpassing competing methods' is unsupported empirically. Moreover, the pseudocode contains a concrete implementation flaw: the line 'U_j_total = UE + UA' is assigned inside a loop over outputs O, overwriting the same variable on each iteration rather than computing the common region described in Section 3.2; as written, the algorithm does not implement the described multiplication of per-output uncertainty maps.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ensemble quantile regression (E-QR) approach for uncertainty estimation and separation, combined with an iterative progressive-sampling algorithm (Algorithm 1) that adds data in regions of high total uncertainty and then uses an XOR between initial and final uncertainty maps to separate aleatoric and epistemic uncertainty. The evaluation is conducted on two synthetic tasks, a toy function and a multi-joint robot arm, with qualitative visual inspection of uncertainty maps as the only form of validation, and the paper claims superiority over Deep Ensembles and Monte Carlo dropout.","tokens_in":7083,"tokens_out":3440,"duration_ms":30873,"significance":"If validated, the method would address a practically important problem: scalable uncertainty separation for large-scale data, and the use of quantile regression as the underlying uncertainty estimator is a plausible alternative to deep ensembles. However, the manuscript provides no quantitative comparisons against the claimed baselines, no error bars, no convergence guarantees for the core iterative procedure, and the algorithm as written contains a concrete implementation bug. The central claims of superiority and reliable separation are therefore not supported by the evidence presented in the manuscript.","major_comments":[{"comment":"The abstract claims that E-QR surpasses Deep Ensembles and Monte Carlo dropout, but the evaluation contains no quantitative comparison with DE or MC dropout; the only evidence is visual inspection of uncertainty maps. Because the central claim is superiority over these baselines, the absence of measurements such as calibration error, separation accuracy, AUROC, or any metric on a dataset with known ground-truth uncertainty decomposition is a load-bearing gap.","section":"Section 4, Figures 3 and 4"},{"comment":"The method's validity rests on the premise that after Q progressive-sampling iterations, the total uncertainty map contains only aleatoric uncertainty. This premise is asserted in the pseudocode comment ('After sufficient number of iterations the total uncertainty map only contains aleatoric uncertainty') but never validated by a convergence argument, a stopping criterion, or an empirical check on a benchmark with known uncertainty types. Section 3.1 itself describes how aleatoric uncertainty leaks into epistemic estimates when data is insufficient; adding data only in high-uncertainty regions could still leave the model's aleatoric fit unreliable in those regions, so the XOR could misclassify persisting leakage as epistemic.","section":"Section 3.2, Algorithm 1"},{"comment":"The loop over outputs is not implemented as described in Section 3.2: the line 'U_j_total = UE + UA' is assigned inside the loop over j, overwriting the same variable each iteration, while 'U_total_total' is used on the right-hand side before being initialized and is never reset for each iteration i. As written, the algorithm does not compute the product of per-output uncertainty maps described in the text, and it is unclear whether the intended common-region accumulation is actually what was executed to produce Figures 3 and 4.","section":"Algorithm 1 pseudocode"},{"comment":"The algorithm depends on a binarization threshold T and iteration count Q, but the text and experiments do not specify how these values are chosen; Sections 4.2 and 4.3 use one and four iterations respectively without justification, and no sensitivity analysis is provided. In addition, the experimental section omits the neural-network architecture, training hyperparameters, ensemble size, and quantile levels, which are free choices that materially affect the results and make the experiments difficult to reproduce or compare.","section":"Section 4.2, 4.3, and Algorithm 1 inputs"}],"minor_comments":[{"comment":"Typos: 'Guassian' should be 'Gaussian'; the sentence 'the fit of aleatoric uncertainty estimates can become arbitrarily' in Section 3.1 is incomplete, seemingly missing a word such as 'bad' or 'inaccurate'.","section":"Introduction and Section 3.1"},{"comment":"The final lines of Algorithm 1 are confusingly written: 'UA, DU_A, PU_A ← U_total_total' appears twice, once before and once after the XOR line, with the second assignment using the same variable names; this makes it hard to tell which quantities are the separated aleatoric and epistemic outputs.","section":"Algorithm 1 output lines"},{"comment":"Algorithm 1 says 'Train the BNN surrogate' and the text uses 'BNN' in places, but the method is an ensemble of quantile regression models (E-QR), not a Bayesian neural network; inconsistent terminology may confuse readers.","section":"Terminology"},{"comment":"The figure captions reference 'Step 1', 'Step 2', etc., but the number of steps and their correspondence to Algorithm 1 iterations are not defined; adding explicit iteration numbers or matching caption text to the algorithm would clarify the presentation.","section":"Figures 3 and 4"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft submitted before the experimental evaluation was completed: the central claim of superiority over DE and MC dropout is made without any quantitative comparison to those baselines, and Algorithm 1, as written, does not match the intended procedure in Section 3.2. The core premise that total uncertainty becomes purely aleatoric after Q iterations is unvalidated, and the paper's own Section 3.1 identifies a leakage mechanism that could undermine the XOR step. Given the scope of the current evidence, the paper is not ready for publication; a major revision with a full benchmark, corrected pseudocode, and a validation of the convergence premise would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper's core idea is worth half an hour of your time: train an ensemble quantile regression model, get separate aleatoric/epistemic maps, then iteratively add data where total uncertainty is high, and XOR the final uncertainty map with the initial one. The vanished regions are labeled epistemic, the persistent regions aleatoric. That's a genuinely new combination, and it addresses a real failure mode—aleatoric noise leaking into epistemic estimates when data is sparse.\n\nWhat the paper does well: it identifies both leakage directions, motivates the iterative sampling clearly, and demonstrates on two synthetic tasks that the procedure recovers the known ground-truth regions (injected noise vs. missing-data gaps). The writing is readable and the related work is relevant.\n\nThe soft spots are substantial. The abstract claims the method 'surpasses' Deep Ensembles and MC dropout, but the evaluation is purely visual—no numbers, no error bars, no tables, no comparison runs. The central premise, stated in Algorithm 1, is that after Q iterations the total uncertainty map contains only aleatoric uncertainty. That's never argued or tested; there is no stopping criterion, no convergence measure, and no check that the added data didn't reduce aleatoric misestimation too. Since the method labels epistemic as 'whatever disappeared,' the separation is partially self-fulfilling, though the synthetic ground truth does provide some external check.\n\nThere are also concrete errors in the pseudocode. The line 'U_j_total = UE + UA' inside the output loop overwrites a single variable and never indexes by output, so the claimed multiplication of per-output maps isn't what the code does. And the two final assignments to 'UA' overwrite the aleatoric output with the XOR result; one of those should be 'UE'. These look like typos, but they mean the algorithm as printed is not executable.\n\nOn balance: the conceptual direction has merit, but the evidence as presented doesn't support the claims. This is a paper I'd want to see again after the authors add quantitative comparisons, fix the pseudocode, and say something about when the iterative procedure actually converges. As is, I would not send it to a referee; I'd desk reject with a clear message that the idea is interesting but the validation and rigor aren't there yet.","headline":"A clever but unproven heuristic for separating aleatoric and epistemic uncertainty; the idea is worth a look, but the paper as written overclaims and the algorithm has concrete bugs.","tokens_in":7585,"tokens_out":4424,"would_cite":false,"duration_ms":38197,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of quantile regression models, combined with iterative sampling in uncertain regions, separates aleatoric from epistemic uncertainty more reliably than Deep Ensembles or Monte Carlo dropout.","keywords":["uncertainty quantification","aleatoric uncertainty","epistemic uncertainty","quantile regression","deep ensembles","progressive sampling","Bayesian optimization","active learning"],"falsifier":"Take a synthetic function with two region types, one with fixed irreducible noise and one with data simply absent, and run Algorithm 1 with a known number of iterations and threshold. If the final XOR map labels any part of the known-noise region as epistemic, or the surviving total-uncertainty map labels any part of the known-missing region as aleatoric, the separation assumption is violated. The claim would also be weakened if increasing the number of iterations causes the aleatoric uncertainty estimate to shrink substantially, since the method presumes aleatoric uncertainty is invariant to added data.","tokens_in":6612,"feed_emoji":"🎯","tokens_out":5424,"duration_ms":45129,"temperature":0.7,"pith_summary":"The paper claims that combining an ensemble of quantile regression models with iterative sampling in uncertain regions separately recovers aleatoric and epistemic uncertainty more reliably than Deep Ensembles or Monte Carlo dropout. This matters because in Bayesian optimization, active learning, and inverse design, treating irreducible noise as a knowledge gap wastes queries, while treating missing data as inherent noise corrupts the uncertainty map. The proposed algorithm adds data to regions flagged as uncertain, retrains, and compares the initial and final uncertainty maps with an XOR operation to label vanished regions as epistemic and persistent ones as aleatoric. On two synthetic benchmarks, a toy function with known noise and a multi-joint robotic arm with an excluded joint, the paper reports correct separation in one and four iterations respectively.","feed_headline":"Progressive sampling separates aleatoric from epistemic uncertainty","feed_subtitle":"Quantile-regression ensembles and iterative sampling correct both directions of uncertainty leakage.","key_machinery":"The load-bearing object is the ensemble of quantile regression models (E-QR), trained with pinball loss to predict upper and lower quantiles per output; the spread between quantiles gives aleatoric uncertainty, and the disagreement across ensemble members gives epistemic uncertainty. The load-bearing procedure is Algorithm 1, which uses progressive sampling in binarized regions of high common uncertainty, followed by retraining and an XOR comparison of normalized total-uncertainty maps to classify regions as epistemic, those that vanish, or aleatoric, those that persist.","core_discovery":"The central claim is that the two failure modes of uncertainty separation, aleatoric uncertainty leaking into epistemic estimates and epistemic uncertainty leaking into aleatoric estimates, share a single root cause: insufficient local data in the affected regions. Because each ensemble member is trained on a subsample, sparse regions cause members to overfit noise, producing an aleatoric-to-epistemic leak, or to produce unreliable quantile fits, producing an epistemic-to-aleatoric leak. The paper's resolution is Algorithm 1: build a common uncertainty map across outputs by normalizing and multiplying per-output total uncertainties, binarize that map, acquire new data in those regions from the native forward process, retrain, and iterate. Once epistemic uncertainty has vanished with the added data, an XOR between the final total-uncertainty map and the initial one isolates the epistemic regions of the original dataset, while the surviving total uncertainty is declared aleatoric.","pith_inferences":["An implication the paper leaves implicit is that Algorithm 1's cost scales with the number of retraining rounds and the amount of newly simulated data, so its practical value depends on how cheap the native forward process is; for expensive simulators, the iterative acquisition budget would need to be counted alongside the uncertainty-separation gain.","A testable extension would be to run the same progressive-sampling scheme on a real dataset with a known noise floor and track the per-iteration change in the aleatoric map; if the aleatoric estimate shrinks systematically, the XOR procedure would be biased toward labeling persistent regions as aleatoric.","The paper argues, but does not isolate in an ablation, that pinball loss makes E-QR less sensitive to initialization and learning rate than NLL-based Deep Ensembles; a controlled comparison with identical architectures and seeds would make that claimed stability measurable."],"forward_implications":["On the reported synthetic benchmarks, E-QR plus Algorithm 1 separates aleatoric from epistemic uncertainty more accurately than Deep Ensembles and MC dropout.","Progressive sampling resolves leakage in both directions: irresducible noise is no longer misreported as epistemic, and missing-data regions are no longer misreported as aleatoric.","E-QR trains quantiles in a single pinball-loss step, which the paper argues is computationally cheaper than Deep Ensembles' two-step NLL training.","The framework is scalable to large datasets, making it applicable to simulation-driven scientific and engineering tasks where uncertainty separation guides data acquisition.","In applications such as Bayesian optimization and active learning, the separated uncertainty maps let a decision-maker target epistemic regions while avoiding regions dominated by irreducible noise."],"supporting_citations":[{"why":"Defines the Deep Ensembles baseline that the paper compares against and whose uncertainty-separation limitations motivate the proposed method.","marker":"Lakshminarayanan, Pritzel, and Blundell 2017"},{"why":"Defines the MC dropout baseline, which the paper argues only predicts epistemic uncertainty and cannot separate aleatoric uncertainty.","marker":"Gal and Ghahramani 2016b"},{"why":"Introduces regression quantiles and the pinball loss that forms the core training objective of E-QR.","marker":"Koenker and Bassett 1978"},{"why":"Supplies the aleatoric-versus-epistemic distinction that the separation task is defined against.","marker":"Hüllermeier and Waegeman 2021"},{"why":"Presents a prior ensemble quantile aggregation method that captures aleatoric uncertainty, which the paper extends toward separation of both uncertainty types.","marker":"Fakoor et al. 2023"},{"why":"Presents a prior quantile-ensemble approach for epistemic uncertainty, which E-QR builds on by jointly modeling aleatoric uncertainty.","marker":"Tagasovska and Lopez-Paz 2018"},{"why":"Provides the multi-joint robotic arm benchmark used to test epistemic-to-aleatoric leakage.","marker":"Ardizzone et al. 2019"}],"fun_headline_variants":["Ensemble quantile regression splits uncertainty types","Iterative sampling corrects uncertainty leakage","Quantile ensembles separate aleatoric and epistemic errors","Progressive sampling fixes uncertainty separation","E-QR: scalable uncertainty separation via sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that after enough rounds of extra sampling in uncertain regions, the total uncertainty map contains only aleatoric uncertainty, because epistemic uncertainty disappears as data is added while aleatoric uncertainty does not; this premise is not validated on a real benchmark or with a formal argument.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble quantile regression splits uncertainty types","Iterative sampling corrects uncertainty leakage","Quantile ensembles separate aleatoric and epistemic errors","Progressive sampling fixes uncertainty separation","E-QR: scalable uncertainty separation via sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1327,"prompt_tokens":822,"completion_tokens":505,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":438,"tokens_out":505,"duration_ms":5083,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:49:16.521054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic function with two region types, one with fixed irreducible noise and one with data simply absent, and run Algorithm 1 with a known number of iterations and threshold. If the final XOR map labels any part of the known-noise region as epistemic, or the surviving total-uncertainty map labels any part of the known-missing region as aleatoric, the separation assumption is violated. The claim would also be weakened if increasing the number of iterations causes the aleatoric uncertainty estimate to shrink substantially, since the method presumes aleatoric uncertainty is invariant to added data.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Deep Ensembles baseline that the paper compares against and whose uncertainty-separation limitations motivate the proposed method."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces regression quantiles and the pinball loss that forms the core training objective of E-QR."},{"cited_title":"J.; and Tibshirani, R","cited_arxiv_id":null,"evidence_quote":"Presents a prior ensemble quantile aggregation method that captures aleatoric uncertainty, which the paper extends toward separation of both uncertainty types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-joint robotic arm benchmark used to test epistemic-to-aleatoric leakage."}],"review_version":1}