{"id":"1a459bc5-b09b-489c-b051-283954ec01e8","arxiv_id":"2505.11738","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Agreement between an independent ensemble (EMM) and a black-box AI on head CT scans predicts the AI's accuracy, enabling real-time confidence-based review of intracranial hemorrhage predictions.","lead":"This paper introduces EMM, a system that monitors a commercial medical AI by checking how often five separately trained models agree with its verdict on each patient scan. It shows that higher agreement predicts higher accuracy for intracranial hemorrhage detection, potentially letting radiologists know when to trust the AI and when to look more carefully.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EMM's role as an independent second opinion is unverified for the FDA-cleared primary: if that model was trained on RSNA data, agreement may reflect shared training-data correlation rather than case-level correctness, inflating the reported stratification.","rationale":"The reader's weakest assumption aligns with mine, and the reader's CONDITIONAL verdict is appropriate. The most load-bearing issue is not post-hoc threshold selection or the idealized radiologist-correction assumption, though those also matter; it is the independence of the second opinion. Without independence, the central claim that agreement level is a valid real-time proxy for correctness is not established for the FDA-cleared model. The paper includes a large institutional evaluation dataset, thoughtful ablations, and honest disclosure of the unknown training data, so this is a testable assumption rather than a defect in execution. The proposed same-split versus disjoint-split experiment is feasible with the open-source primary because the authors already control its training split, making it a decisive and inexpensive check. If the test shows no meaningful difference, the framework's core claim is substantially strengthened; if it shows a difference, the reported clinical benefits should be re-estimated under a genuinely independent EMM. Therefore, the verdict should remain CONDITIONAL pending this verification.","tokens_in":12360,"tokens_out":4059,"duration_ms":43484,"concrete_test":"Using the existing open-source primary model, train EMM sub-models on the same 50% RSNA split used to retrain the primary and on the disjoint 50% split; evaluate both on the same 2,919-study institutional test set with ED-SPAUC, ED-SNAUC, and agreement-stratified accuracy. If same-split EMM materially outperforms disjoint-split EMM, training-data overlap is a confound and the FDA-cleared results must be re-analyzed with an independently trained EMM or an external validation cohort; if the two are statistically equivalent, the independence assumption is supported. Report bootstrapped confidence intervals for the difference in error-detection performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires EMM to be an independent second opinion. The paper's own Methods state, 'we do not have access to information regarding whether the FDA-cleared ICH AI model utilized this dataset during its development,' while EMM was trained on the full RSNA 2019 dataset. If the black-box primary was trained on RSNA, EMM agreement is not independent consensus but partly shared training-data correlation: both models could learn the same dataset-specific shortcuts, so high agreement may reflect correlated errors rather than case-level correctness. For the open-source primary, the authors deliberately retrained it on 50% of RSNA and reserved the remaining 50% for EMM, which shows they recognize the need for disjoint training; however, this safeguard is absent for the FDA-cleared model that anchors the headline results. If overlap inflates agreement, the reported accuracy stratification, including the 'increased confidence' group, is an upper bound and the false-confidence cases (4% of studies where EMM fully agreed with an incorrect FDA prediction) could be more frequent in deployment. The concern is not that overlap necessarily destroys the method, since agreement may still carry some signal, but that the magnitude and safety of the benefit are unverified for the primary commercial use case.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Ensembled Monitoring Model (EMM), an ensemble of five independently trained 3D CNNs that monitor a black-box primary AI model for intracranial hemorrhage (ICH) detection. EMM compares each primary-model prediction against its own sub-model predictions and reports a 0-100% agreement level, which is then stratified into increased, similar, or decreased confidence groups using thresholds tuned to the primary model's accuracy at different agreement levels. The authors evaluate EMM on 2,919 institutional CT studies using two primary models (an FDA-cleared black-box model and an open-source RSNA 2019 challenge model), showing that agreement level correlates with primary-model accuracy, that error-detection performance improves with more training data and more sub-models, and that applying the suggested actions can yield relative accuracy gains while incurring false-alarm burdens. The paper includes prevalence-adjusted analyses at 30%, 15%, and 5% ICH prevalence, a Shapley feature analysis on a 281-case manually annotated subset, and an explicit discussion of limitations including the risk of simultaneous EMM and primary-model failure.","tokens_in":12557,"tokens_out":3484,"duration_ms":39497,"significance":"If the central association is robust, the EMM framework addresses a real and currently unmet need: real-time, case-by-case confidence assessment for black-box commercial radiology AI without access to weights, activations, or ground-truth labels. The study's strengths include evaluation on a reasonably large, independent institutional dataset, evaluation with two different primary models, prevalence adjustments across clinically relevant levels, and a thorough ablation of training-data size, ensemble size, and model capacity. The manuscript also explicitly acknowledges the key risks of shared training data and simultaneous failure. However, the headline results for the FDA-cleared model rest on an unverifiable independence assumption, and the confidence-stratification thresholds are fitted and evaluated on the same dataset, so the reported gains are likely optimistic. The work is a valuable contribution to the monitoring literature provided these issues are addressed or clearly bounded.","major_comments":[{"comment":"The independence of EMM from the FDA-cleared primary model is load-bearing for the central claim that EMM agreement measures case-level correctness rather than shared training-data correlation. In the Methods you state: 'we do not have access to information regarding whether the FDA-cleared ICH AI model utilized this dataset during its development,' while EMM was trained on the full RSNA 2019 dataset. If the FDA-cleared model was also trained on RSNA, high EMM agreement could reflect both models having learned the same dataset-specific shortcuts, inflating the reported accuracy stratification and the size of the 'increased confidence' group. For the open-source primary you deliberately retrained it on 50% of RSNA and reserved the remaining 50% for EMM, demonstrating awareness of this issue; the same safeguard is absent for the commercial model. Please provide a quantitative sensitivity analysis or an explicit argument bounding the effect of potential training-data overlap, or temper the headline claims for the FDA-cleared model accordingly.","section":"Methods, 'ICH Detection AI models'"},{"comment":"The confidence-stratification thresholds (increased/similar/decreased) are established from the primary model's accuracy at different EMM agreement levels on the same 2,919-study evaluation dataset, and the subsequent accuracy gains and false-alarm rates are computed on that same dataset. This creates a circularity/overfitting risk: the reported relative gains (4.7%, 11%, and 38% at 30%, 15%, and 5% prevalence) and the distribution of cases across confidence groups are not independent estimates. The thresholds should be fixed on a training or validation subset using a prespecified rule, or validated on a held-out set or via cross-validation, before the clinical benefit is quantified. As written, the magnitude of the benefit is an optimistic in-sample estimate.","section":"Results, 'EMM Enables Confidence-based Image Review Optimization' and Figure 3"},{"comment":"The relative accuracy improvement is computed under the assumption that radiologists will 'always correctly label the cases' in the decreased-confidence group. This assumption directly drives the reported gains and cannot be satisfied in practice; radiologist interpretive error on difficult ICH cases is well documented. The paper presents no sensitivity analysis for imperfect radiologist relabeling (e.g., 90% or 80% correct relabeling). Without such an analysis, the net benefit of EMM-guided review relative to false-alarm burden is an upper bound, and the threshold recommendations for clinical use are not yet supported.","section":"Methods, 'Analysis of the tradeoff between false alarm rate and the relative accuracy improvement'"}],"minor_comments":[{"comment":"The error-detection metrics ED-SPAUC and ED-SNAUC are used throughout the ablation results but are never formally defined in the main text or Methods; the caption of Figure 4 gives only the full names. Please provide explicit definitions, including what curves are being integrated and how the binary error-detection decision is derived from EMM agreement.","section":"Methods, 'EMM Evaluation Dataset' / Figure 4 caption"},{"comment":"The Shapley analysis is performed on a manually selected subset (N=281) with 210 ICH-positive and 71 ICH-negative cases, including 43 ICH-negative cases specifically chosen to contain hemorrhage mimics. This selection is not random and likely inflates the importance of mimic-related features; the generalizability of the feature-importance findings to a naturalistic population should be discussed.","section":"Results, 'EMM agreement levels are associated with different features'"},{"comment":"The sentence reporting 'relative accuracy improvements of 4.7%, 11%, and 38% versus false-alarm rates of 0.89%, 0.45%, and 0.14%' mixes percentages of different denominators (relative improvement vs. rate among all cases); clarifying the denominators in the text or a table would improve interpretability.","section":"Results, 'EMM Enables Confidence-based Image Review Optimization'"},{"comment":"The limitation paragraph correctly notes that EMM can fail together with the primary model and instill false confidence, but the main-text discussion of the 4% false-confidence cases (full EMM agreement with an incorrect FDA prediction) would be more informative if it reported the corresponding case counts for the open-source primary model as well.","section":"Discussion, 'limitations persist'"},{"comment":"The acronym 'EMM' is used for both the framework and the specific five-model ensemble; consider distinguishing the general framework from the particular instantiation to ease reading.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly within the scope of a clinical AI/radiology informatics journal and addresses an important gap. My main concern is that the central quantitative claims for the FDA-cleared model rest on an unverified training-data-independence assumption, and that the threshold evaluation is in-sample; both issues are fixable with additional analyses or with more cautious language, but they currently affect the reliability of the headline numbers. I would also encourage the editor to ask for a data/code availability statement, since the reproducibility of the exact threshold-fitting procedure is otherwise hard to assess."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper introduces EMM, an ensemble-based monitor for black-box radiology AI that flags case-by-case confidence without touching the primary model's internals. That is a real gap, and the ICH application is well executed: 2,919 studies, an FDA-cleared and an open-source primary, ablations on data volume, ensemble size, and model size, and an honest discussion of the 4% of cases where EMM fully agrees with a wrong primary prediction. The Shapley analysis of agreement drivers is a nice addition.\n\nThe central association—EMM agreement correlates with primary-model accuracy—holds up on this dataset. The soft spots are in the quantified benefits. The confidence thresholds were selected after looking at accuracy versus agreement on the same evaluation data, so the stratification gains are optimistic. The trade-off analysis assumes radiologists always correctly relabel cases flagged for decreased confidence; the paper states this in Methods but it makes the accuracy improvements upper bounds. And the independence question for the FDA-cleared primary is unresolved: EMM is trained on RSNA 2019, and the authors say they do not know whether the commercial model used that data. For the open-source primary they deliberately retrained on a disjoint 50%, which shows they know the issue, but the headline numbers come from the FDA-cleared model where that safeguard is absent. If there is training-data overlap, agreement could reflect correlated errors rather than independent consensus, inflating the stratification.\n\nNone of this kills the method. The framework is sensible, the data are real, and the paper is transparent about its limitations. But the reported gains should be treated as upper bounds, and the commercial-model independence needs external validation.\n\nMy take: this deserves a careful peer review, likely conditional with revision. It is a legitimate step toward real-time monitoring of deployed medical AI, and I would cite it in my work. I'd also bring it to our reading group.\n\nRecommendation: accept for peer review, with the expectation that the authors address threshold overfitting and validate the independence assumption on a separate cohort.","headline":"A promising black-box AI monitor with solid empirical support, but the headline gains are upper bounds due to post-hoc thresholds and an unresolved training-data-overlap question for the commercial model.","tokens_in":13141,"tokens_out":2808,"would_cite":true,"duration_ms":28166,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An independent ensemble of five models can monitor a black-box intracranial hemorrhage detection AI in real time by using agreement with the primary model's output as a per-case confidence signal.","keywords":["Ensembled Monitoring Model","intracranial hemorrhage","AI monitoring","ensemble agreement","black-box AI","confidence stratification","real-time monitoring","head CT"],"falsifier":"Use a primary ICH model whose training data are known to be completely disjoint from the dataset used to train EMM, run EMM on a large held-out set, and check whether the accuracy gradient across increased/similar/decreased agreement remains as steep as reported; if it flattens, the agreement signal is an artifact of shared training data rather than of case difficulty.","tokens_in":12125,"feed_emoji":"🧠","tokens_out":9961,"duration_ms":98409,"temperature":0.7,"pith_summary":"The paper claims that a black-box AI detector can be monitored in real time without touching its internals, by running an independently trained ensemble alongside it and watching how often they agree. Using intracranial hemorrhage detection on head CT as the test case, the authors measure agreement between five sub-models and a primary detector over 2,919 studies, and show that agreement level sorts cases into groups with markedly different primary-model accuracy. They report that full agreement marks about half the cases as safe to trust, while low agreement flags a smaller set where focused radiologist re-review improves overall accuracy. If the claim holds, it gives clinical sites a way to supervise commercial AI products case by case, without ground-truth labels or model access.","feed_headline":"Agreement among five models tells when to trust an ICH AI","feed_subtitle":"In 2,919 head CTs, ensemble agreement revealed which ICH-AI predictions were likely wrong.","key_machinery":"The core object is the EMM agreement level: the percentage of five independently trained 3D convolutional networks whose binary ICH prediction matches the primary model's prediction, computed at inference time from outputs alone. Because the ensemble is trained on an independent public dataset and never sees the primary model's weights or activations, its agreement is positioned as an external second opinion rather than a recalibrated version of the model's own confidence. The agreement level is turned into clinical action by a thresholding scheme: thresholds are chosen from curves of primary-model accuracy at each agreement level, separately for positive and negative predictions and for different ICH prevalences, producing increased, similar, and decreased confidence groups.","core_discovery":"The central discovery is that the fraction of EMM sub-models that match the primary model's binary ICH prediction is a usable per-case proxy for whether that prediction is correct. On an FDA-cleared black-box model, full EMM agreement coincided with correct predictions in 1,479 of 2,919 cases; partial agreement split into 848 correct and 454 incorrect primary predictions; and in 138 cases both the ensemble and the primary model agreed but were wrong, usually on subtle hemorrhages or hemorrhage mimics. Dividing cases into increased, similar, and decreased confidence groups and re-reviewing only the decreased-confidence group improves accuracy by 4.7% to 38% relative to the unmonitored baseline, depending on ICH prevalence, with false-alarm rates under 0.9%. The authors conclude that agreement level, not the primary model's own internal confidence, should drive the clinical action taken on each case.","pith_inferences":["The same output-only agreement monitoring could be applied to other black-box radiology AI tasks whenever a labeled dataset is available to train the ensemble, so the design is not specific to hemorrhage detection.","Tracking EMM agreement levels longitudinally would let a site detect drift in the primary model's behavior before a ground-truth audit, adding a monitoring signal to existing quality-control pipelines.","A prospective workflow trial that records actual radiologist re-review time and decision changes would quantify the net cognitive-load reduction the authors expect."],"forward_implications":["In cases of full EMM agreement, radiologists can use the primary model's prediction without extra scrutiny; these cases made up about half of the 2,919 studies (1,479 cases).","Sending only low-agreement cases for re-review raises overall ICH detection accuracy by 4.7% to 38% relative to baseline at 30%, 15%, and 5% prevalence, with false-alarm rates between 0.14% and 0.89%.","EMM's error-detection performance improves with more training studies (saturating near 10,000), more sub-models (4–5 as a practical starting point), and larger sub-models when training data are ample.","The favorable balance between accuracy gains and false alarms does not hold for ICH-negative predictions at low prevalences, so agreement thresholds should be tuned to each site's prevalence.","In a small fraction of cases (138 of 2,919), both EMM and the primary model agree but are wrong, typically on extremely subtle hemorrhages or strong hemorrhage mimics, so full agreement is not a guarantee of correctness."],"supporting_citations":[{"why":"Public head-CT ICH challenge dataset on which all EMM sub-models are trained; supplies the task-matched labels the ensemble needs.","marker":"[34]"},{"why":"Challenge dataset page that defines the ICH detection task and labels, also the training source for the open-source primary model.","marker":"[48]"},{"why":"Open-source ICH detection model used as the second primary system; provides a non-commercial test bed with known architecture.","marker":"[33]"},{"why":"Prior framework for evaluating clinical AI without ground-truth annotations that EMM adapts into per-case real-time monitoring.","marker":"[15]"}],"fun_headline_variants":["Ensemble agreement flags unreliable ICH-AI predictions","Five-model consensus predicts ICH-AI errors in real time","Monitoring AI by ensemble consensus cuts false alarms","When to trust ICH AI: follow sub-model agreement","Ensemble voting reveals confidence in black-box ICH detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the public dataset used to train the monitoring ensemble is not the same data that trained the black-box model being monitored, so agreement counts as an independent second opinion rather than shared memory.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble agreement flags unreliable ICH-AI predictions","Five-model consensus predicts ICH-AI errors in real time","Monitoring AI by ensemble consensus cuts false alarms","When to trust ICH AI: follow sub-model agreement","Ensemble voting reveals confidence in black-box ICH detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1427,"prompt_tokens":900,"completion_tokens":527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":449}},"tokens_in":516,"tokens_out":527,"duration_ms":5618,"temperature":1.0,"reasoning_tokens":449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:48:09.036545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a primary ICH model whose training data are known to be completely disjoint from the dataset used to train EMM, run EMM on a large held-out set, and check whether the accuracy gradient across increased/similar/decreased agreement remains as steep as reported; if it flattens, the agreement signal is an artifact of shared training data rather than of case difficulty.","supporting_citations":[],"review_version":1}