Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)

Akshay S. Chaudhari; Andrew Johnston; Andrew Walker Campion; Arogya Koirala; Bonnie A. Armstrong; Camila Gonzalez; David B. Larson; Derrick Laurel; Hye Sun Na; Lina Cheuy

REVIEW 3 major objections 5 minor 3 references

Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An independent ensemble of five models can monitor a black-box intracranial hemorrhage detection AI in real time by using agreement with the primary model's output as a per-case confidence signal.

desk verdict A promising black-box AI monitor with solid empirical support, but the headline gains are upper bounds due to post-hoc thresholds and an unresolved training-data-overlap question for the commercial model. read the letter →

arxiv 2505.11738 v1 pith:7PG7SGYG submitted 2025-05-16 cs.AI

Zhongnan Fang , Andrew Johnston , Lina Cheuy , Hye Sun Na , Magdalini Paschali , Camila Gonzalez , Bonnie A. Armstrong , Arogya Koirala

show 5 more authors

Derrick Laurel Andrew Walker Campion Michael Iv Akshay S. Chaudhari David B. Larson

This is my paper · ORCID

classification cs.AI

keywords EnsembledMonitoringModelintracranialhemorrhageAIensembleagreementblack-boxconfidencestratificationreal-timeheadCT

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a black-box AI detector can be monitored in real time without touching its internals, by running an independently trained ensemble alongside it and watching how often they agree. Using intracranial hemorrhage detection on head CT as the test case, the authors measure agreement between five sub-models and a primary detector over 2,919 studies, and show that agreement level sorts cases into groups with markedly different primary-model accuracy. They report that full agreement marks about half the cases as safe to trust, while low agreement flags a smaller set where focused radiologist re-review improves overall accuracy. If the claim holds, it gives clinical sites a way to supervise commercial AI products case by case, without ground-truth labels or model access.

What carries the argument

The core object is the EMM agreement level: the percentage of five independently trained 3D convolutional networks whose binary ICH prediction matches the primary model's prediction, computed at inference time from outputs alone. Because the ensemble is trained on an independent public dataset and never sees the primary model's weights or activations, its agreement is positioned as an external second opinion rather than a recalibrated version of the model's own confidence. The agreement level is turned into clinical action by a thresholding scheme: thresholds are chosen from curves of primary-model accuracy at each agreement level, separately for positive and negative predictions and for different ICH prevalences, producing increased, similar, and decreased confidence groups.

What would settle it

Use a primary ICH model whose training data are known to be completely disjoint from the dataset used to train EMM, run EMM on a large held-out set, and check whether the accuracy gradient across increased/similar/decreased agreement remains as steep as reported; if it flattens, the agreement signal is an artifact of shared training data rather than of case difficulty.

Watch

Extended reading notes

Core claim

The central discovery is that the fraction of EMM sub-models that match the primary model's binary ICH prediction is a usable per-case proxy for whether that prediction is correct. On an FDA-cleared black-box model, full EMM agreement coincided with correct predictions in 1,479 of 2,919 cases; partial agreement split into 848 correct and 454 incorrect primary predictions; and in 138 cases both the ensemble and the primary model agreed but were wrong, usually on subtle hemorrhages or hemorrhage mimics. Dividing cases into increased, similar, and decreased confidence groups and re-reviewing only the decreased-confidence group improves accuracy by 4.7% to 38% relative to the unmonitored baseline, depending on ICH prevalence, with false-alarm rates under 0.9%. The authors conclude that agreement level, not the primary model's own internal confidence, should drive the clinical action taken on each case.

Load-bearing premise

The framework assumes that the public dataset used to train the monitoring ensemble is not the same data that trained the black-box model being monitored, so agreement counts as an independent second opinion rather than shared memory.

Editorial extensions

If this is right

In cases of full EMM agreement, radiologists can use the primary model's prediction without extra scrutiny; these cases made up about half of the 2,919 studies (1,479 cases).
Sending only low-agreement cases for re-review raises overall ICH detection accuracy by 4.7% to 38% relative to baseline at 30%, 15%, and 5% prevalence, with false-alarm rates between 0.14% and 0.89%.
EMM's error-detection performance improves with more training studies (saturating near 10,000), more sub-models (4–5 as a practical starting point), and larger sub-models when training data are ample.
The favorable balance between accuracy gains and false alarms does not hold for ICH-negative predictions at low prevalences, so agreement thresholds should be tuned to each site's prevalence.
In a small fraction of cases (138 of 2,919), both EMM and the primary model agree but are wrong, typically on extremely subtle hemorrhages or strong hemorrhage mimics, so full agreement is not a guarantee of correctness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same output-only agreement monitoring could be applied to other black-box radiology AI tasks whenever a labeled dataset is available to train the ensemble, so the design is not specific to hemorrhage detection.
Tracking EMM agreement levels longitudinally would let a site detect drift in the primary model's behavior before a ground-truth audit, adding a monitoring signal to existing quality-control pipelines.
A prospective workflow trial that records actual radiologist re-review time and decision changes would quantify the net cognitive-load reduction the authors expect.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper introduces EMM, an ensemble-based monitor for black-box radiology AI that flags case-by-case confidence without touching the primary model's internals. That is a real gap, and the ICH application is well executed: 2,919 studies, an FDA-cleared and an open-source primary, ablations on data volume, ensemble size, and model size, and an honest discussion of the 4% of cases where EMM fully agrees with a wrong primary prediction. The Shapley analysis of agreement drivers is a nice addition.

The central association—EMM agreement correlates with primary-model accuracy—holds up on this dataset. The soft spots are in the quantified benefits. The confidence thresholds were selected after looking at accuracy versus agreement on the same evaluation data, so the stratification gains are optimistic. The trade-off analysis assumes radiologists always correctly relabel cases flagged for decreased confidence; the paper states this in Methods but it makes the accuracy improvements upper bounds. And the independence question for the FDA-cleared primary is unresolved: EMM is trained on RSNA 2019, and the authors say they do not know whether the commercial model used that data. For the open-source primary they deliberately retrained on a disjoint 50%, which shows they know the issue, but the headline numbers come from the FDA-cleared model where that safeguard is absent. If there is training-data overlap, agreement could reflect correlated errors rather than independent consensus, inflating the stratification.

None of this kills the method. The framework is sensible, the data are real, and the paper is transparent about its limitations. But the reported gains should be treated as upper bounds, and the commercial-model independence needs external validation.

My take: this deserves a careful peer review, likely conditional with revision. It is a legitimate step toward real-time monitoring of deployed medical AI, and I would cite it in my work. I'd also bring it to our reading group.

Recommendation: accept for peer review, with the expectation that the authors address threshold overfitting and validate the independence assumption on a separate cohort.

Referee Report

3 major / 5 minor

Summary. The paper proposes the Ensembled Monitoring Model (EMM), an ensemble of five independently trained 3D CNNs that monitor a black-box primary AI model for intracranial hemorrhage (ICH) detection. EMM compares each primary-model prediction against its own sub-model predictions and reports a 0-100% agreement level, which is then stratified into increased, similar, or decreased confidence groups using thresholds tuned to the primary model's accuracy at different agreement levels. The authors evaluate EMM on 2,919 institutional CT studies using two primary models (an FDA-cleared black-box model and an open-source RSNA 2019 challenge model), showing that agreement level correlates with primary-model accuracy, that error-detection performance improves with more training data and more sub-models, and that applying the suggested actions can yield relative accuracy gains while incurring false-alarm burdens. The paper includes prevalence-adjusted analyses at 30%, 15%, and 5% ICH prevalence, a Shapley feature analysis on a 281-case manually annotated subset, and an explicit discussion of limitations including the risk of simultaneous EMM and primary-model failure.

Significance. If the central association is robust, the EMM framework addresses a real and currently unmet need: real-time, case-by-case confidence assessment for black-box commercial radiology AI without access to weights, activations, or ground-truth labels. The study's strengths include evaluation on a reasonably large, independent institutional dataset, evaluation with two different primary models, prevalence adjustments across clinically relevant levels, and a thorough ablation of training-data size, ensemble size, and model capacity. The manuscript also explicitly acknowledges the key risks of shared training data and simultaneous failure. However, the headline results for the FDA-cleared model rest on an unverifiable independence assumption, and the confidence-stratification thresholds are fitted and evaluated on the same dataset, so the reported gains are likely optimistic. The work is a valuable contribution to the monitoring literature provided these issues are addressed or clearly bounded.

major comments (3)

[Methods, 'ICH Detection AI models'] The independence of EMM from the FDA-cleared primary model is load-bearing for the central claim that EMM agreement measures case-level correctness rather than shared training-data correlation. In the Methods you state: 'we do not have access to information regarding whether the FDA-cleared ICH AI model utilized this dataset during its development,' while EMM was trained on the full RSNA 2019 dataset. If the FDA-cleared model was also trained on RSNA, high EMM agreement could reflect both models having learned the same dataset-specific shortcuts, inflating the reported accuracy stratification and the size of the 'increased confidence' group. For the open-source primary you deliberately retrained it on 50% of RSNA and reserved the remaining 50% for EMM, demonstrating awareness of this issue; the same safeguard is absent for the commercial model. Please provide a quantitative sensitivity analysis or an explicit argument bounding the effect of potential training-data overlap, or temper the headline claims for the FDA-cleared model accordingly.
[Results, 'EMM Enables Confidence-based Image Review Optimization' and Figure 3] The confidence-stratification thresholds (increased/similar/decreased) are established from the primary model's accuracy at different EMM agreement levels on the same 2,919-study evaluation dataset, and the subsequent accuracy gains and false-alarm rates are computed on that same dataset. This creates a circularity/overfitting risk: the reported relative gains (4.7%, 11%, and 38% at 30%, 15%, and 5% prevalence) and the distribution of cases across confidence groups are not independent estimates. The thresholds should be fixed on a training or validation subset using a prespecified rule, or validated on a held-out set or via cross-validation, before the clinical benefit is quantified. As written, the magnitude of the benefit is an optimistic in-sample estimate.
[Methods, 'Analysis of the tradeoff between false alarm rate and the relative accuracy improvement'] The relative accuracy improvement is computed under the assumption that radiologists will 'always correctly label the cases' in the decreased-confidence group. This assumption directly drives the reported gains and cannot be satisfied in practice; radiologist interpretive error on difficult ICH cases is well documented. The paper presents no sensitivity analysis for imperfect radiologist relabeling (e.g., 90% or 80% correct relabeling). Without such an analysis, the net benefit of EMM-guided review relative to false-alarm burden is an upper bound, and the threshold recommendations for clinical use are not yet supported.

minor comments (5)

[Methods, 'EMM Evaluation Dataset' / Figure 4 caption] The error-detection metrics ED-SPAUC and ED-SNAUC are used throughout the ablation results but are never formally defined in the main text or Methods; the caption of Figure 4 gives only the full names. Please provide explicit definitions, including what curves are being integrated and how the binary error-detection decision is derived from EMM agreement.
[Results, 'EMM agreement levels are associated with different features'] The Shapley analysis is performed on a manually selected subset (N=281) with 210 ICH-positive and 71 ICH-negative cases, including 43 ICH-negative cases specifically chosen to contain hemorrhage mimics. This selection is not random and likely inflates the importance of mimic-related features; the generalizability of the feature-importance findings to a naturalistic population should be discussed.
[Results, 'EMM Enables Confidence-based Image Review Optimization'] The sentence reporting 'relative accuracy improvements of 4.7%, 11%, and 38% versus false-alarm rates of 0.89%, 0.45%, and 0.14%' mixes percentages of different denominators (relative improvement vs. rate among all cases); clarifying the denominators in the text or a table would improve interpretability.
[Discussion, 'limitations persist'] The limitation paragraph correctly notes that EMM can fail together with the primary model and instill false confidence, but the main-text discussion of the 4% false-confidence cases (full EMM agreement with an incorrect FDA prediction) would be more informative if it reported the corresponding case counts for the open-source primary model as well.
[General] The acronym 'EMM' is used for both the framework and the specific five-model ensemble; consider distinguishing the general framework from the particular instantiation to ease reading.

Circularity Check

1 steps flagged · score 5.0 of 10

The EMM agreement-to-accuracy association is an independent empirical finding, but the reported increased/similar/decreased confidence stratification and its relative accuracy gains are partly by construction: the thresholds are fitted on the same 2,919-study evaluation set to make the group accuracies significantly higher/lower than baseline.

fitted input called prediction [Discussion (third paragraph); Results, 'EMM Enables Confidence-based Image Review Optimization']
"The thresholds for defining the three accuracy groups in this study were established based on expert radiologist assessment and the primary model’s performance at different EMM agreement levels (sFigure3), with separate analyses for ICH-positive and ICH-negative primary model predictions. The thresholds to indicate increased and decreased confidence were specifically designated so that the overall ICH detection accuracy would be significantly higher or lower, respectively, than that with only the primary model (baseline)."

The stratification thresholds are chosen using the primary model's observed accuracy at each EMM agreement level on the same 2,919-study dataset, and the identical dataset is then used to report that the increased-confidence group has significantly higher and the decreased-confidence group significantly lower accuracy than the no-EMM baseline. Because the thresholds were explicitly 'designated so that the overall ICH detection accuracy would be significantly higher or lower,' the headline stratification and the resulting relative-accuracy-gain numbers (e.g., 4.7%, 11%, 38%) are a fitted description of the calibration set rather than an out-of-sample prediction.

full rationale

The central EMM mechanism is not circular: the five sub-models are trained on the public RSNA 2019 dataset, the open-source primary model is deliberately retrained on a disjoint 50% split, and the evaluation is performed on a separate institutional dataset of 2,919 studies. The claim that higher EMM agreement is associated with higher primary-model accuracy is an empirical correlation, not a tautology. The paper also honestly discloses that it does not know whether the FDA-cleared primary model used the RSNA dataset, and that EMM can fail together with the primary model in 4% of cases; these are independence and safety limitations, not circularity. However, one load-bearing piece of the reported result does reduce to its own inputs. The thresholds defining the increased/similar/decreased confidence groups were set using the primary model's accuracy at different EMM agreement levels on the evaluation dataset, and the same dataset was then used to measure the accuracy of those groups and the relative improvements from reviewing the decreased-confidence group. The paper states explicitly that the thresholds were 'specifically designated so that the overall ICH detection accuracy would be significantly higher or lower' than baseline, which means the headline stratification gains are partly constructed by the threshold choice. This is a fitted-parameter-called-prediction pattern rather than full definitional circularity, because the underlying agreement-accuracy ordering and the independent training of EMM retain genuine empirical content. Score 5 reflects this partial circularity; the paper would need a held-out threshold-validation split or pre-specified thresholds to turn the stratified gains into a true prediction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

EMM is a system of five standard 3D CNNs; it introduces no new physical or conceptual entities. The main free parameter is the set of agreement thresholds, which is fitted to the evaluation data. The axioms are the empirical premises of the monitoring approach and the idealized radiologist-revision assumption used in the benefit calculation.

free parameters (1)

EMM confidence stratification thresholds (increased/similar/decreased) = For FDA-cleared model, positive predictions: 100% / 60-80% / 0-40%; negative predictions: 100% / 20-80% / 0%
These thresholds were hand-selected based on the primary model's accuracy at different EMM agreement levels on the same 2,919-study evaluation set (sFigure 3), then used to report accuracy gains. This is a post-hoc fit to the evaluation data and may make the reported separation between confidence groups optimistic.

assumptions (3)

domain assumption Independently trained sub-models on the same task provide an unbiased second opinion about the primary model's prediction.
This is the core premise of EMM. The paper tests it empirically, but it is assumed when comparing agreement with correctness.
domain assumption The public RSNA 2019 dataset used to train EMM is representative of the deployment distribution for the FDA-cleared primary model, and the primary model did not train on the same data.
Explicitly acknowledged in Methods: "we do not have access to information regarding whether the FDA-cleared ICH AI model utilized this dataset during its development." If the primary model trained on RSNA, EMM agreement could be inflated by data overlap rather than independent competence.
ad hoc to paper Radiologists will always correctly relabel cases in the decreased-confidence group when the analysis computes relative accuracy gains.
Stated in Methods: "we assumed that the radiologists will always correctly label the cases, improving overall accuracy." This idealization directly determines the reported relative improvements in Figure 3c.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)." pith.science (2026). https://pith.science/paper/7PG7SGYG

@misc{pith2026250511738,
  author       = {Pith},
  title        = {Pith review of: Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7PG7SGYG}},
  note         = {Machine review of arXiv:2505.11738}
}

read the original abstract

Artificial intelligence (AI) tools for radiology are commonly unmonitored once deployed. The lack of real-time case-by-case assessments of AI prediction confidence requires users to independently distinguish between trustworthy and unreliable AI predictions, which increases cognitive burden, reduces productivity, and potentially leads to misdiagnoses. To address these challenges, we introduce Ensembled Monitoring Model (EMM), a framework inspired by clinical consensus practices using multiple expert reviews. Designed specifically for black-box commercial AI products, EMM operates independently without requiring access to internal AI components or intermediate outputs, while still providing robust confidence measurements. Using intracranial hemorrhage detection as our test case on a large, diverse dataset of 2919 studies, we demonstrate that EMM successfully categorizes confidence in the AI-generated prediction, suggesting different actions and helping improve the overall performance of AI tools to ultimately reduce cognitive burden. Importantly, we provide key technical considerations and best practices for successfully translating EMM into clinical settings.

Figures

Figures reproduced from arXiv: 2505.11738 by the authors.

**Figure 2.** EMM agreement associated with different features. a. Example cases for which EMM showed different levels of agreement with the FDA-cleared primary ICH detection model. Cases with full EMM agreement typically showed clear presence or absence of ICH, while cases with partial agreement often displayed subtle ICH or features mimicking hemorrhage. b. Quantitative analysis on the importance of features affecting EMM agree… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith

[18]

& Gales, M

Malinin, A. & Gales, M. Predictive Uncertainty Estimation via Prior Networks. in Advances in Neural Information Processing Systems vol. 31 (Curran Associates, Inc., 2018). 19. Kull, M. et al. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. in Advances in Neural Information Processing Systems vol....

arXiv 2018
[36]

Mutasa, S., Sun, S. & Ha, R. Understanding artificial intelligence based radiology studies: What is overfitting? Clinical Imaging 65, 96–99 (2020). 37. Feng, J. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. npj Digit. Med. 5, 1–9 (2022). 38. Larson, D. B. A Vision for...

arXiv doi:10.48550/arxiv.2401.12208 2020
[55]

Li, X. et al. Hematoma Expansion Context Guided Intracranial Hemorrhage Segmentation and Uncertainty Estimation. IEEE Journal of Biomedical and Health Informatics 26, 1140–1151 (2022). 56. Li, X. et al. The state-of-the-art 3D anisotropic intracranial hemorrhage segmentation on non-contrast head CT: The INSTANCE challenge. Preprint at https://doi.org/10.4...

arXiv doi:10.48550/arxiv.2301.03281 2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [18]

& Gales, M

Malinin, A. & Gales, M. Predictive Uncertainty Estimation via Prior Networks. in Advances in Neural Information Processing Systems vol. 31 (Curran Associates, Inc., 2018). 19. Kull, M. et al. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. in Advances in Neural Information Processing Systems vol....

arXiv 2018

[2] [36]

Mutasa, S., Sun, S. & Ha, R. Understanding artificial intelligence based radiology studies: What is overfitting? Clinical Imaging 65, 96–99 (2020). 37. Feng, J. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. npj Digit. Med. 5, 1–9 (2022). 38. Larson, D. B. A Vision for...

arXiv doi:10.48550/arxiv.2401.12208 2020

[3] [55]

Li, X. et al. Hematoma Expansion Context Guided Intracranial Hemorrhage Segmentation and Uncertainty Estimation. IEEE Journal of Biomedical and Health Informatics 26, 1140–1151 (2022). 56. Li, X. et al. The state-of-the-art 3D anisotropic intracranial hemorrhage segmentation on non-contrast head CT: The INSTANCE challenge. Preprint at https://doi.org/10.4...

arXiv doi:10.48550/arxiv.2301.03281 2022