{"id":"b6b5d997-8d87-43bb-a600-d6a7bcd0fb83","arxiv_id":"2412.18980","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Deep ensemble models, particularly a heterogeneous four-member ensemble, outperform MC-dropout and Bayesian neural networks at flagging out-of-distribution fault data in bearing diagnosis, under both epistemic and aleatoric uncertainty.","lead":"This paper compares three families of uncertainty-aware deep learning models for detecting out-of-distribution faults in rotating machinery on the CWRU bearing dataset. It finds that deep ensembles, especially the heterogeneous De2, flag unseen and noisy fault data better than MC-dropout and Bayesian neural networks, and also run faster at inference.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ensemble advantage in De1/De2 is confounded by base architecture; a single-CNN baseline is missing.","rationale":"I agree with the reader that the architecture confound is the weakest load-bearing assumption. The paper is a useful empirical comparison: it tests a wide range of scenarios, introduces a reasonable threshold (tau2), and extends aleatoric uncertainty to non-Gaussian noise. The headline result—De2 achieving about 97.5% OOD detection under tau1 and 89% under tau2, versus 71% and 67% for the dropout model—is striking and, if real, practically important. However, because each uncertainty method is paired with a distinct base architecture, the result does not yet establish that ensembling (rather than the simpler CNN feature extractor, the lower capacity, or the sharp calibration of the CNN) drives the improvement. The absence of a single-CNN baseline is the single gap that, if filled, would either validate or overturn the causal interpretation. Secondary issues reinforce the conditional verdict: no error bars or repeated runs are reported, and the inference-time advantage in Table 11 is partly an artifact of using K=4 for ensembles versus K=10 for dropout and BNN; per-sample prediction times are actually similar (about 1.4 seconds per forward pass). The threshold-ordering inconsistency in Table 8 (tau2 < tau1 for several BNN rows) is minor. These do not change my assessment: the paper should be accepted conditionally, with the single-CNN control identified above required before the causal claim is endorsed.","tokens_in":21860,"tokens_out":9325,"duration_ms":87652,"concrete_test":"Train a single deterministic CNN with the exact De1 base-learner architecture (Appendix A3: input 512, one Conv+Pool block with 16 filters, kernel 3, stride 1, same padding, batch norm, max-pool stride 2; dense layer of 64 nodes with sigmoid; softmax output). Evaluate it on the six epistemic hold-out scenarios exactly as in Section 3.4: compute predictive entropy H = -sum_c p_c log p_c, calibrate tau1 and tau2 on the validation set, and report average OOD-to-UT and ID-to-UT on the test set. Compare with the De1 rows of Tables 6 and 7, running multiple seeds to bound seed-to-seed variation. If the single CNN's average OOD-to-UT is within 5 percentage points of De1, the ensemble mechanism is not the source of the reported gains; if De1 is substantially better, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that deep ensembles, especially De2, are the preferred uncertainty-aware architectures (Sections 4.1–4.3 and Conclusions)—rests on a comparison in which the uncertainty mechanism and the base architecture are varied simultaneously. Section 3.2 assigns dropout to a ConvLSTM (ConvLSTM-D), Bayesian inference to a conv stack with Bayesian dense layers (BNN), and ensembling either to four identical small CNNs (De1) or to a mixture of two CNNs and two ConvLSTM learners (De2). No non-ensemble CNN baseline is reported. This matters because De1, the smallest architecture, already outperforms the larger ConvLSTM-D and BNN on average OOD detection in Tables 6 and 7 (e.g., 93.9% versus 71.4% and 70.4% under tau1). The low thresholds for De1/De2 in Table 5 (tau1 = 0.0001 for De1 in most scenarios) suggest these models are simply very confident on ID data, which could make the IQR-based tau1 flag OOD data more easily regardless of ensembling. If a single deterministic CNN with the same architecture as a De1 base learner already achieves comparable OOD detection, then the reported superiority is not evidence for ensembling as an uncertainty mechanism; it would instead reflect a base-architecture advantage. The paper's practical recommendation therefore lacks a key experimental control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of four uncertainty-aware deep learning architectures for bearing fault diagnosis on the CWRU dataset: Monte Carlo dropout on a ConvLSTM (ConvLSTM-D), a Bayesian neural network (BNN), and two deep ensembles (De1 and De2). Six epistemic scenarios are constructed by holding out one fault class, and twelve aleatoric scenarios by adding Gaussian, Impulse, Rayleigh, and Weibull noise at three SNR levels. OOD detection is based on entropy thresholds: the IQR-based tau1 and a proposed F1-maximizing threshold tau2. The main finding is that De2 has the highest OOD detection rates and, under tau2, the lowest ID misclassification, especially under strong noise, and the shortest prediction time; the authors conclude that deep ensembles, particularly De2, are the preferred choice for uncertainty-aware fault diagnosis.","tokens_in":22120,"tokens_out":9518,"duration_ms":85983,"significance":"If the empirical ranking is trustworthy, the paper provides useful practical guidance and broadens the aleatoric-uncertainty evaluation to non-Gaussian noise types. The proposed tau2 threshold is a simple and practical alternative to tau1, and the central ranking is evaluated on a separate test set, which avoids the circularity failure mode of fitting a threshold and reporting performance on the same data. The paper's strength is its breadth of scenarios and systematic reporting of OOD/ID rates; its weakness is the lack of experimental controls needed for a comparative benchmark. The manuscript does not ship machine-checked proofs or code, and the main recommendation rests on comparisons in which the uncertainty mechanism and the base architecture vary together.","major_comments":[{"comment":"The comparison conflates the uncertainty mechanism with the base architecture. MC dropout is implemented on a ConvLSTM (ConvLSTM-D), BNN on a convolutional stack with Bayesian dense layers, De1 on four identical small CNNs, and De2 on a mixture of CNN and ConvLSTM learners; no deterministic single-model baseline shares the architecture of a De1 or De2 base learner. The higher OOD detection of De1/De2 (e.g., 93.9% and 97.5% vs. 71.4% and 70.4% under tau1 in Table 6) could therefore be due to architecture rather than ensembling. The very low tau1 values for De1/De2 in Table 5 (0.0001 in several scenarios) reinforce this risk, because a model that is extremely confident on ID data will flag almost any OOD point under an IQR-based threshold. Please add at least one deterministic CNN baseline with the same architecture as the De1 base learner, and ideally a single ConvLSTM baseline matching De2's LSTM-based learners, so that the ensemble advantage is measured relative to the same architecture.","section":"Section 3.2, Tables 5-7"},{"comment":"The data-preparation procedure leaks information between training and test sets. Bursts of 512 points are extracted every 200 points, so consecutive bursts overlap by 312 samples; a random 70/30 split of these bursts places overlapping windows from the same continuous bearing signal in both training and test partitions. This can inflate ID accuracy and artificially separate ID and OOD entropy distributions, and it also affects the validation-based threshold computation. Please use a disjoint split by time intervals or another non-overlapping assignment, and restate the conclusions if the numbers change.","section":"Section 3.1, Figure 7"},{"comment":"All results are point estimates from a single run; the tables report no confidence intervals, no multiple seeds, and no significance tests. Several claims rely on small margins or scenario-dependent reversals, such as the De1 vs. De2 comparison under tau1 in Tables 6 and 7, so the ranking cannot be assessed statistically as presented. Please add repeated-seed results with confidence intervals or pairwise tests, or explicitly state the single-run limitation and temper the ranking claims.","section":"Sections 4.1-4.2, Tables 6-10"},{"comment":"The BNN architecture is described inconsistently. The text and figure describe Conv+Pool blocks followed by flattening and Bayesian dense layers with no LSTM, but Table A2 lists an LSTM layer with 64 nodes before the Bayesian dense layer. This ambiguity makes the BNN implementation irreproducible and should be resolved; the same issue affects the description of De2's LSTM-based learners, which the text calls two dense layers but Table A4 lists as a single dense layer after the LSTM.","section":"Appendix Table A2 vs. Section 3.2.2/Figure 10"},{"comment":"The first sentence of Section 4.3 contradicts Table 11. The text says training for De1 and De2 is significantly faster than for ConvLSTM-D and BNN, but Table 11 shows the opposite: De1 and De2 take 688.44 and 639.69 seconds versus 239.55 and 266.18 seconds for ConvLSTM-D and BNN. The following sentence and Section 5 correctly state that prediction is faster for the ensembles while training is longer; please correct this internal contradiction.","section":"Section 4.3, Table 11"}],"minor_comments":[{"comment":"The noise-type label 'Weilbul' should be 'Weibull' in the first row of Table 8 and in the corresponding rows of Tables 9 and 10.","section":"Tables 8-10"},{"comment":"The statement that the OOD entropy distribution 'shifts rightward, toward lower entropy values' is self-contradictory: a rightward shift means higher entropy, which is also what the subsequent explanation of easier detection under stronger noise implies. Please correct the direction described.","section":"Section 4.2"},{"comment":"The noise-injection procedure should state explicitly whether the same 20% of examples were used for the validation and test sets and across all models; the fixed seed guarantees reproducibility but not clarity about this choice.","section":"Section 3.3"},{"comment":"The phrase 'first comprehensive comparative study' is stronger than the evidence supports, given that the literature review in Table 1 covers a small set of papers and the comparison uses a single dataset; consider tempering the wording.","section":"Abstract and Introduction"},{"comment":"The number of Monte Carlo samples differs across methods (K=10 for dropout and BNN, K=4 for ensembles), and the paper justifies this as the minimum for stable results; a sentence acknowledging that entropy estimates from K=4 and K=10 may not be directly comparable would help readers interpret the differences.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope, but in its current form the headline recommendation that ensembles are preferable rests on two fixable design problems: the architecture confound and the overlapping-window leakage. If the authors add the missing single-model baselines and a temporal split, the paper could be accepted; if not, the conclusions should be narrowed to the specific architectures tested rather than to ensembling as an uncertainty mechanism. I see no citation or novelty disclosure concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this paper is worth reading as a benchmark, but its central claim is not yet proven. The genuinely new pieces are the non-Gaussian noise scenarios (Impulse, Rayleigh, Weibull) and the F1-based entropy threshold tau2. The comparison of MC dropout, BNN, and deep ensembles on CWRU is clearly presented, and the tables back up the reported numbers. If you need a practical map of how these methods behave on a rotating-machinery benchmark, this is a reasonable starting point.\n\nWhat it does well: the noise injection is carefully described; the entropy-based OOD pipeline is easy to follow; and the paper is honest about being limited to one dataset and about assuming the noise type is known. The tau2 threshold is a sensible practical addition: choosing the entropy cutoff that maximizes F1 on validation is more aligned with deployment than a pure outlier rule.\n\nWhere it falls short: the core conclusion that deep ensembles dominate is confounded by architecture. Each method uses a different base network: ConvLSTM for dropout, a conv stack with Bayesian dense layers for BNN, small CNNs for De1, and a mix for De2. There is no single deterministic CNN baseline. Tables 6 and 7 show De1 and De2, which happen to be the smallest networks, also produce very low tau1 values (0.0001), meaning they are extremely confident on ID data. The observed gap might come from the architecture's inductive bias, not from ensembling as an uncertainty mechanism. A proper control would be a single CNN with the same architecture as one De1 base learner. That missing experiment is the main flaw.\n\nAlso, there are no confidence intervals, multiple seeds, or significance tests, so the margin between De1 and De2 (e.g., 93.9% vs 97.5%) may be noise. The random split of overlapping bursts from the same time series (stride 200, burst length 512) is a known leakage risk for CWRU; it may not invalidate the OOD ranking, but it should be addressed with a time-based split or burst-level grouping.\n\nMinor point: the text claims tau1 is invariably lower than tau2, but Table 8 has several exceptions (e.g., BNN Weibull 0 dB, ConvLSTM-D Gaussian 5 dB). Easy fix.\n\nWho this is for: practitioners choosing an uncertainty method for rotating-machinery diagnosis will get useful guidance, but the confound means the ensemble advantage is plausible, not established. I would send this to review because the benchmark and threshold are worth having in the literature, but I would push for the missing baseline, repeated runs, and a fix to the overlap issue before acceptance. I would not cite it in my own work until the ensemble claim is controlled.","headline":"A useful benchmark with a practical new threshold, but the ensemble advantage is confounded by base architecture and needs a deterministic CNN control.","tokens_in":22659,"tokens_out":2759,"would_cite":false,"duration_ms":26385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep ensemble architectures—especially a heterogeneous ensemble called De2—are the most reliable uncertainty-aware models for fault diagnosis in rotating machinery, detecting out-of-distribution data better and faster than Monte Carlo…","keywords":["Fault Diagnosis","Uncertainty","Trustworthy AI","Industry 4.0","Deep Learning","Out of Distribution Data","deep ensembles","epistemic and aleatoric uncertainty"],"falsifier":"Train a deterministic CNN with the same capacity as a single De2 learner on CWRU and measure its OOD detection under the same thresholds; if it reaches De2's roughly 90–97% accuracy, the ensemble benefit is an artifact of backbone choice rather than the ensembling mechanism. Alternatively, average the predictions of three independently seeded ConvLSTM-D models and compare with De2.","tokens_in":21662,"feed_emoji":"⚙️","tokens_out":6278,"duration_ms":51528,"temperature":0.7,"pith_summary":"This paper compares four uncertainty-aware deep learning architectures on a bearing fault diagnosis benchmark to see which can best flag data that falls outside the training distribution—either because it comes from an unseen fault (epistemic uncertainty) or because sensor noise corrupts the signal (aleatoric uncertainty). The authors find that deep ensembles, and particularly a heterogeneous ensemble they call De2, detect out-of-distribution data more reliably than Monte Carlo dropout or a Bayesian neural network under both kinds of uncertainty, while also making faster predictions. They also propose a new entropy threshold, based on F1 score, that reduces the number of trustworthy samples wrongly flagged as untrustworthy compared with the standard outlier-based threshold. If the finding holds, deep ensembles become the practical default for trustworthy fault diagnosis in industrial settings.","feed_headline":"Deep ensembles win on uncertainty-aware fault diagnosis","feed_subtitle":"Heterogeneous ensemble De2 catches unseen faults and noisy data most reliably, and fastest.","key_machinery":"The central machinery is the prediction-entropy test: each model produces K predictions per example (K=10 for dropout and BNN, K=4 for ensembles), the average softmax scores are converted to entropy, and an entropy threshold decides whether the example is trustworthy (in-distribution) or untrustworthy (out-of-distribution). The paper compares two thresholds: τ1, the inter-quartile-range outlier rule applied to in-distribution validation entropies, and τ2, a new threshold chosen to maximize F1 score on the validation confusion matrix. The architecture that carries the argument is De2, a four-learner deep ensemble with two simple CNN learners and two hybrid ConvLSTM learners, whose diversity is credited with the most accurate uncertainty quantification.","core_discovery":"On the CWRU rotating-bearing benchmark, across six epistemic scenarios (each holding out one fault class) and twelve aleatoric scenarios (four noise types at three signal-to-noise ratios), the deep ensemble models De1 and De2 outperform ConvLSTM with MC dropout and a Bayesian neural network at separating in-distribution from out-of-distribution data by prediction entropy. De2, combining two simple CNNs with two ConvLSTM-based learners, achieves the top average OOD detection rates—97.5% and 89.1% under the two thresholds for epistemic uncertainty, and up to 100% under strong noise—while keeping in-distribution misclassification low and delivering the shortest prediction time. The paper also introduces an F1-score-based entropy threshold (τ2) that is consistently higher than the existing IQR-based threshold (τ1), trading a small loss in OOD recall for a substantial reduction in false alarms on in-distribution data.","pith_inferences":["The paper's recommendation presupposes that the observed advantage comes from ensembling itself, since each method uses a different backbone; a natural test is to compare a single strong CNN with a deep ensemble built on the same backbone.","The F1-based threshold τ2 could be transferred to other OOD detection settings beyond fault diagnosis, wherever a validation set contains both ID and OOD samples; its behavior under class imbalance is worth probing.","Since the paper assumes the noise type is known at deployment, a practical extension would be a noise-type classifier upstream, or training with mixed noise types, to remove that assumption.","The 20% noise-injection protocol means the detector never sees pure-signal baselines for those OOD samples; real signals might mix clean and noisy segments within a single burst, which could change optimal thresholds."],"forward_implications":["If De2's advantage generalizes, practitioners deploying uncertainty-aware fault diagnosis should prefer heterogeneous deep ensembles over dropout sampling or BNNs, especially when unseen fault classes are likely.","The choice of entropy threshold matters: conservative τ1 maximizes OOD recall at the cost of more false alarms, while τ2 minimizes manual re-inspection burden because it flags fewer in-distribution samples.","Under slight noise, all methods struggle to distinguish noisy OOD from clean ID data, so noise should be controlled at data-collection time rather than only at inference.","Deep ensembles' faster prediction time means earlier fault detection and intervention, despite longer training.","Non-Gaussian noise types (Weibull, Impulse, Rayleigh) matter: detection performance differs by noise type, so evaluations restricted to Gaussian noise may misstate real-world performance."],"supporting_citations":[{"why":"Provides the IQR-based entropy threshold τ1 and the uncertainty-aware deep ensemble approach that De2 extends.","marker":"[36]"},{"why":"Defines the deep ensemble method whose identical-learner and diverse-learner configurations become De1 and De2.","marker":"[31]"},{"why":"Establishes Monte Carlo dropout as a way to sample predictive distributions, which ConvLSTM-D uses at inference.","marker":"[10]"},{"why":"Supplies the taxonomy of epistemic and aleatoric uncertainty and the dropout, BNN, and ensemble methods the study compares.","marker":"[8]"},{"why":"Provides the convolutional LSTM architecture that forms the ConvLSTM-D baseline and one learner type inside De2.","marker":"[4]"},{"why":"Documents the CWRU bearing dataset used for all experiments.","marker":"[49]"},{"why":"Frames out-of-distribution detection under dataset shift, motivating the ID/OOD test design.","marker":"[7]"},{"why":"Shows an identical-learner deep ensemble applied to fault diagnosis, which De1 replicates.","marker":"[35]"}],"fun_headline_variants":["Deep ensembles dominate uncertainty-aware fault diagnosis","Uncertainty-aware DL: deep ensembles beat the rest","Deep ensembles: fastest, most reliable OOD detection in fault diagnosis","Deep ensembles predict unseen faults best, fastest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes each method's performance reflects its uncertainty mechanism, because the dropout, Bayesian, and ensemble models are built on different neural network backbones.","fun_headline_variants_meta":{"raw":{"variants":["Deep ensembles dominate uncertainty-aware fault diagnosis","Uncertainty-aware DL: deep ensembles beat the rest","Deep ensembles: fastest, most reliable OOD detection in fault diagnosis","Deep ensembles predict unseen faults best, fastest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000553,"raw_usage":{"total_tokens":2665,"prompt_tokens":1003,"completion_tokens":1662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1600}},"tokens_in":619,"tokens_out":1662,"duration_ms":13726,"temperature":1.0,"reasoning_tokens":1600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:17.065963+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a deterministic CNN with the same capacity as a single De2 learner on CWRU and measure its OOD detection under the same thresholds; if it reaches De2's roughly 90–97% accuracy, the ensemble benefit is an artifact of backbone choice rather than the ensembling mechanism. Alternatively, average the predictions of three independently seeded ConvLSTM-D models and compare with De2.","supporting_citations":[{"cited_title":"Han, Y.-F","cited_arxiv_id":null,"evidence_quote":"Provides the IQR-based entropy threshold τ1 and the uncertainty-aware deep ensemble approach that De2 extends."},{"cited_title":"Lakshminarayanan, A","cited_arxiv_id":null,"evidence_quote":"Defines the deep ensemble method whose identical-learner and diverse-learner configurations become De1 and De2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes Monte Carlo dropout as a way to sample predictive distributions, which ConvLSTM-D uses at inference."},{"cited_title":"Abdar, F","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of epistemic and aleatoric uncertainty and the dropout, BNN, and ensemble methods the study compares."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the convolutional LSTM architecture that forms the ConvLSTM-D baseline and one learner type inside De2."},{"cited_title":"Neupane, J","cited_arxiv_id":null,"evidence_quote":"Documents the CWRU bearing dataset used for all experiments."},{"cited_title":"Ovadia, E","cited_arxiv_id":null,"evidence_quote":"Frames out-of-distribution detection under dataset shift, motivating the ID/OOD test design."},{"cited_title":"Tuyet-Doan, H.-A","cited_arxiv_id":null,"evidence_quote":"Shows an identical-learner deep ensemble applied to fault diagnosis, which De1 replicates."}],"review_version":1}