{"id":"95b52858-4ff4-4870-9b07-63491b563590","arxiv_id":"2506.14400","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Across mortality, graft failure, and triage prediction, subgroup analysis shows that aggregate accuracy hides clinically relevant disparities, including lower PRC for Black and female patients.","lead":"This paper measures how well medical AI models perform on patient subgroups, not just the whole population, across three clinical prediction tasks. It finds that patients who are Black, female, older, or missing data can get markedly worse predictions, and argues such breakdowns should be reported before deployment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mortality White-Black PRC gap may be a base-rate artifact; AUPRC is prevalence-dependent and the paper's only significant test does not isolate model performance.","rationale":"The reader's weakest_assumption focuses on statistical robustness: confidence intervals are absent for two datasets and bootstrap intervals for small subgroups are wide. My concern is narrower and more specific: even where significance is reported (Mortality White vs Black), the metric used (AUPRC) is not comparable across subgroups with different positive-class prevalence. The paper itself notes the mortality-rate difference (Black 9% vs overall 13%) in Section 3.2, yet continues to interpret the PRC gap as a model performance disparity. Since AUPRC's random baseline equals the prevalence, a lower value in a lower-prevalence subgroup can occur without any degradation in ranking quality. The ROC values in Table 1 (0.89 vs 0.86) are consistent with a much smaller true gap, reinforcing the concern. This is an internal validity issue, not a disagreement with consensus: the paper's central claim that 'the model does perform significantly better for Whites compared to Blacks' is the single strongest empirical evidence offered, and it is called into question. The conceptual argument for subgroup-level evaluation does not collapse if this particular result fails, but the empirical demonstration is weakened. The reader's CONDITIONAL verdict already captures the need for additional analysis, so I do not change the verdict; I would sharpen the required condition to include prevalence-adjusted metrics or explicit control for base rates. The concrete test above would settle whether the concern lands. If the adjusted gap remains significant, the paper's strongest empirical claim would be substantially reinforced.","tokens_in":9093,"tokens_out":4596,"duration_ms":49678,"concrete_test":"Recompute the Mortality White vs Black comparison using AUROC and a prevalence-adjusted precision-recall metric (e.g., precision-recall gain, PRG) or a stratified bootstrap that equalizes subgroup prevalence; then run the one-sided bootstrap test on the adjusted difference. If the White-Black gap becomes non-significant, or if the AUROC difference of 0.03 is non-significant, the claim that the model performs significantly better for Whites is a base-rate artifact and should be retracted or reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline empirical result—that the model 'performs significantly better for Whites compared to Blacks' in mortality (Section 3.2)—is established by a bootstrap test on AUPRC (Appendix A.3, Table 8). But AUPRC is not a prevalence-invariant measure of classifier quality: for fixed discriminative ability, expected AUPRC moves with the positive-class base rate because the random baseline is the prevalence. In the Mortality data (Table 5), test-set mortality is 11% for White patients and 9% for Black patients, versus 12% overall. A lower AUPRC for Black patients is therefore expected even if the model's risk ranking is equally good, and the reported ROC values (White 0.89, Black 0.86) show a much smaller gap than the PRC values (0.62 vs 0.45). The bootstrap in A.3 resamples within each subgroup, preserving each subgroup's prevalence, so the p<0.05 result tests a composite of discrimination and base rate, not model performance disparity. Because this PRC gap is the main quantitative support for the central claim, the empirical demonstration is not yet established. The same caveat applies to graft-failure gender PRC differences, although no significance test is reported there; the paper's own Limitations section acknowledges the lack of uncertainty quantification but does not address prevalence confounding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes subgroup-level performance of machine learning models on three medical prediction tasks: ICU mortality (MIMIC-III), kidney graft failure (German transplant center), and emergency triage (German ambulance records). Using previously trained models evaluated on fixed test sets, the authors compute ROC and PRC (or precision/recall/F1 for the multi-class triage task) for demographic and clinical subgroups and compare subgroup metrics against the full reference test set. They report apparent disparities, such as lower PRC for Black patients in mortality, for women and older patients in graft failure, and for children, elderly, and patients with missing age in triage. They argue that aggregate metrics can hide such subgroup gaps and advocate routine subgroup-level performance reporting, transparency documentation analogous to a package insert, and targeted model improvement for disadvantaged groups. One one-sided bootstrap significance test is run for the mortality task; the paper's Limitations section acknowledges that the subgroup analyses are exploratory and that uncertainty is not quantified across all datasets.","tokens_in":9314,"tokens_out":4958,"duration_ms":53195,"significance":"If the empirical results are robust, the paper makes a useful and timely practical point: subgroup-disaggregated evaluation should become a routine part of clinical AI development and documentation, and aggregate ROC/PRC values alone are insufficient for deployment decisions. The paper brings together three real-world clinical datasets, including two non-US settings, and grounds its recommendations in existing governance discussions such as the EU AI Act. Its strengths include direct evaluation on held-out subgroups, explicit data-distribution tables, and an unusually candid Limitations section. However, the headline significant disparity is established through an AUPRC-based bootstrap test, and AUPRC is prevalence-dependent; as a result the central empirical demonstration needs reanalysis. In addition, most reported subgroup differences are presented without confidence intervals or significance tests, so the cross-task generalizability claim is not yet fully supported.","major_comments":[{"comment":"The claim that the model 'performs significantly better for Whites compared to Blacks' is based on a bootstrap test of AUPRC, but AUPRC is not a prevalence-invariant measure of classifier quality. In Table 5, the test-set mortality rate is 11% for White patients and 9% for Black patients, versus 12% overall; for a fixed ranking quality, a lower-prevalence subgroup will have a lower expected AUPRC. The reported ROC values (White 0.89, Black 0.86) show a much smaller gap than the PRC values (0.62 vs. 0.45), which is exactly the pattern expected from prevalence confounding. The bootstrap procedure in A.3 resamples within each subgroup, thereby preserving each subgroup's prevalence, so the p<0.05 result tests a composite of discrimination and base rate rather than model performance disparity. Please re-run the significance analysis on AUROC, or use prevalence-adjusted or standardized PRC, or otherwise separate discrimination and calibration from outcome prevalence, and reinterpret the White-Black finding accordingly.","section":"Section 3.2, Appendix A.3, Tables 5 and 8"},{"comment":"Significance tests and confidence intervals are reported only for the mortality task; the graft-failure and triage tables present point estimates without uncertainty quantification. Some subgroups are very small (e.g., the Triage 'No Age' group is about 1% of the 9,001-case dataset in Table 6), and the paper itself acknowledges roughly 30% label noise in triage. Consequently, the observed F1 and PRC gaps in Tables 2 and 3 could reflect sampling variation or label noise rather than genuine model disparities, and the paper's general conclusion that subgroup disparities are 'common' across tasks is not yet established. Please provide bootstrap confidence intervals or other uncertainty measures for all three tasks, or explicitly restrict the empirical claims to the mortality task and treat the others as exploratory illustrations.","section":"Tables 1-3 and Limitations"},{"comment":"The significance analysis performs pairwise subgroup comparisons without correcting for multiple testing, and the subgroup splits are post hoc (as the Limitations section concedes). With many pairwise comparisons and only one reaching p<0.05, the headline White-vs-Black result could be a Type I error. Please either prespecify the comparisons, apply a multiple-comparison correction, or present the White-Black result as hypothesis-generating rather than confirmatory.","section":"Appendix A.3 and Section 2.3"}],"minor_comments":[{"comment":"The High Age (>75) test row shows 834 patients with 24 deaths, a mortality rate of about 3%, which is inconsistent with the overall test mortality rate of 12% and with the training row for the same subgroup (17%). Please verify these numbers; this looks like a typographical error.","section":"Table 5"},{"comment":"The statement that PRC 'even further decreases for Black women to PRC=0.36 (not shown in the table)' reports an important intersectional result that is not included in any table or appendix. Please add this result to the reported data.","section":"Section 2.3"},{"comment":"The graft-failure model is described as a 'fast Gradient Boosting Regressor' even though the task is binary classification evaluated with ROC and PRC. Please clarify how continuous regression outputs are converted to binary predictions or scores.","section":"Section 2.1"},{"comment":"The sentence about large language models perpetuating stereotypes in clinical settings cites Alnegheimish et al. (2024), which is a time-series anomaly-detection paper. Please replace this with a directly relevant reference on clinical LLM bias.","section":"Section 1"},{"comment":"The text says the 'overall average' mortality rate is 13%, but Table 5 reports 13% for training and 12% for the reference test set. Please align the text with the table.","section":"Section 3.2"},{"comment":"The confidence intervals in Table 8 are very wide for small subgroups (e.g., Asian [0.2608, 0.9351], Hispanic [0.4290, 0.9851]), yet Table 1 presents the corresponding point estimates without any flag. Please mark unstable subgroups in the main tables or move the confidence intervals into the main text.","section":"Table 8 and Table 1"},{"comment":"The triage model's integration of 'expert rules' is not described in enough detail to be reproduced. Please specify which rules are used and how they modify the transformer outputs.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's practical message—that subgroup-level performance reporting should be standard in medical AI documentation—is valuable and within the journal's scope. The main obstacle is empirical: the headline significant disparity rests on an AUPRC comparison that is confounded by subgroup prevalence, and most other results lack uncertainty quantification. I do not see grounds for rejection; the analyses can be re-run or re-interpreted, and the transparency recommendations remain useful regardless. I would also gently flag the citation mismatch and the apparent typo in Table 5 as signs that the manuscript would benefit from a careful revision pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you want a quick sense of where medical-AI subgroup evaluation actually stands in practice. The new thing here is not the idea—aggregate metrics hide subgroup disparities is well established—but the concrete measurements: graft-failure and triage datasets from German hospitals, plus MIMIC-III mortality, all sliced by demographic and clinical subgroups with previously published models. The missing-age effect in triage is a genuinely nice observation, and the transparency argument (a package insert for models) is sensible and well tied to the EU AI Act.\n\nThe paper is also refreshingly honest. The limitations section admits the analyses are exploratory, the triage labels are about 30% noisy, and statistical significance is not quantified across all datasets. No circularity: models come from prior work, and the subgroup analysis is post hoc.\n\nBut the soft spots are real, and the stress-test note is right. The only significance test is on AUPRC in the mortality task, and AUPRC is prevalence-dependent. In the test set, White patients have 11% mortality, Black patients 9%. So even with identical discrimination, you'd expect a lower AUPRC for the Black subgroup. The reported ROC values (0.89 vs 0.86) show a much smaller gap than the PRC values (0.62 vs 0.45), which is exactly what you'd see from a base-rate shift plus a little model difference. The bootstrap resamples within subgroups, preserving each subgroup's prevalence, so the p<0.05 result does not isolate model quality. That's a load-bearing flaw for the paper's single quantitative headline. The same concern applies to the graft-failure gender PRC difference, where no significance test is reported at all.\n\nEven with that, the central qualitative argument holds up: overall performance can look fine while specific subgroups fare worse, and reporting subgroup-level metrics is a reasonable minimal standard. The paper overreaches when it says it has 'demonstrated' the link between fairness and transparency, but the evidence is directionally fine.\n\nMy bottom line: this deserves a serious referee, but only if the referee pushes hard on the prevalence issue. The authors should either report a prevalence-adjusted metric (e.g., ROC, or stratified versions of AUPRC) or explicitly frame the PRC gaps as including base-rate differences, and they should add confidence intervals to the main tables. As is, I wouldn't cite the specific numbers, but I'd consider citing the triage missing-age result.\n\nFor peer review: yes, send it out. It's not a desk reject—it has real data and an honest question—but it needs revision before it is publishable.\n\nBest,\n[You]","headline":"A useful, honest call for subgroup-level reporting in medical AI, but the headline White-Black PRC gap is likely a base-rate artifact, so the quantitative claim needs fixing, not just polishing.","tokens_in":9895,"tokens_out":1537,"would_cite":false,"duration_ms":18382,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aggregate performance metrics can hide major patient-subgroup disparities in medical AI, so subgroup-level evaluation should be a routine part of clinical deployment.","keywords":["medical AI fairness","subgroup-level evaluation","performance disparity","precision-recall curve","mortality prediction","graft failure prediction","triage prediction","model transparency"],"falsifier":"Take the same fixed mortality model and evaluate it on a much larger independent test cohort with hundreds of Black patients, computing a tight confidence interval for the Black–White difference in precision-recall score; if the interval straddles zero or the gap disappears, the paper's headline disparity is sampling noise rather than a stable property of the model.","tokens_in":8882,"feed_emoji":"🩺","tokens_out":7977,"duration_ms":70536,"temperature":0.7,"pith_summary":"This paper argues that medical machine-learning models can post strong aggregate scores while performing markedly worse for specific patient groups, and that subgroup-level evaluation is therefore a necessary step before clinical deployment. Across three tasks—ICU mortality, kidney graft failure, and emergency triage—overall metrics look healthy, but precision-recall scores drop for Black patients, women, older patients, deceased-donor recipients, and children. In the mortality task, a nonparametric bootstrap test finds the White–Black precision-recall difference significant at $p<0.05$: 0.62 versus 0.45. The paper concludes that transparency through routine subgroup reporting, modeled on a 'package insert,' should accompany medical AI, and that seeing these gaps should trigger targeted model improvement rather than blind deployment.","feed_headline":"Aggregate AI metrics hide subgroup failures in medicine","feed_subtitle":"A mortality model looks strong overall, then scores 0.45 for Black patients—subgroup reporting should be routine.","key_machinery":"The mechanism is subgroup-level evaluation applied to a fixed trained model: filter the reference test set by patient characteristics (age, sex, ethnicity, donor status, missing data), recompute ROC-AUC, AUPRC, or precision/recall/F1 within each slice, and compare each slice with the full test set. AUPRC is the load-bearing metric because it is sensitive to class imbalance and rare positive events, which is where aggregate accuracy hides the most; in the mortality task it is paired with a one-sided nonparametric bootstrap test (1,000 resamples, pairwise PRC differences, $p<0.05$) that turns a raw gap into a significance claim. The same machinery is applied across three tasks, three model architectures, and two countries, so the finding is not tied to one dataset.","core_discovery":"The central discovery is that fairness in medical AI cannot be read from aggregate performance: a model can meet conventional standards on the full test set yet be substantially less reliable for identifiable patient groups. On MIMIC-III in-hospital mortality, the reference AUPRC is 0.61 while Black patients receive 0.45 and Black women 0.36; the bootstrap hypothesis test shows the model performs significantly better for White than Black patients, the pairwise comparison the test flags as significant. In graft failure prediction, PRC falls from 0.72 for younger patients to 0.51 for older patients, from 0.61 for men to 0.49 for women, and from 0.70 for living donors to 0.53 for deceased donors. In triage, the most urgent classes are missed more often for children, very old patients, and patients whose age is missing. The paper treats these disparities not as a reason to ban models but as information that must be documented and acted on, since a biased model with clear warnings may still help where no decision support exists.","pith_inferences":["If subgroup reporting becomes standard, hospital procurement could shift toward models whose worst-slice performance matches the hospital's own patient mix, creating market pressure to collect diverse training data.","Only the mortality task received a significance test; applying the same bootstrap procedure to graft failure and triage would likely show that some headline gaps, such as male–female differences in mortality, are within sampling noise, while the triage 'no age' collapse may be the most robust and actionable finding.","Missingness itself—patients with no recorded age—behaves like a protected subgroup in the triage data; treating data-quality slices as first-class fairness axes would be a natural extension the paper gestures at but does not develop.","The package-insert proposal implies a regulatory test: a model card that omits subgroup numbers should be considered incomplete documentation, an idea that goes beyond what current practice requires."],"forward_implications":["Aggregate metrics like overall AUC or F1 should not be the acceptance criterion for clinical deployment; subgroup-specific numbers need to be reported alongside them.","Documentation akin to a package insert—subgroup performance, training-data distribution, and disclaimers for underrepresented groups—becomes a practical expectation for medical AI, consistent with high-risk AI documentation duties.","Once gaps are visible, the responsible path is targeted calibration, retraining, or data collection for disadvantaged subgroups instead of either silent deployment or outright refusal to use the model.","Clinicians can use subgroup reports to decide when to trust, override, or question a model output, for example for Black patients in mortality prediction or elderly patients in triage.","Surprising disparities can flag underlying medical or data problems, such as creatinine-based eGFR bias in graft failure, that are not purely algorithmic."],"supporting_citations":[{"why":"Supplies the MIMIC-III critical care database used for the mortality prediction task.","marker":"Johnson et al., 2016"},{"why":"Defines the in-hospital mortality prediction setting after the first 48 hours of admission.","marker":"Yang and Wu, 2021"},{"why":"Provides the multimodal time-series-and-text architecture used for the mortality model.","marker":"Zhang et al., 2022"},{"why":"Extends the multimodal fusion approach for medical text exploitation used in the mortality experiments.","marker":"Ravichandran et al., 2024"},{"why":"Describes the gradient boosting model applied to graft failure prediction.","marker":"Roller et al., 2022"},{"why":"Introduces the transformer-based triage prediction model extended with structured features.","marker":"Maschhur et al., 2024"},{"why":"Grounds the transparency and package-insert argument for AI-based health technologies.","marker":"Ott and Dabrock, 2022"},{"why":"Supplies the governance framing of clinical decision-making with AI and the information-leaflet idea.","marker":"Samhammer et al., 2023"},{"why":"Provides the regulatory hook: the EU AI Act requires instructions for use to include performance information for specific groups in high-risk systems.","marker":"European Union, 2024"}],"fun_headline_variants":["Aggregate scores conceal medical AI's subgroup failures","Fairness in medical AI demands subgroup-level audits","One model: 0.61 overall AUPRC, 0.36 for Black women","Medical AI's overall accuracy hides disparate care","Subgroup performance: the missing metric in medical AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the measured subgroup differences, especially the lower precision-recall score for Black patients in the mortality model, are true model behavior rather than artifacts of small samples—yet significance testing was run on only one task, and the bootstrap intervals for small subgroups are wide (Black patients: 0.232 to 0.634).","fun_headline_variants_meta":{"raw":{"variants":["Aggregate scores conceal medical AI's subgroup failures","Fairness in medical AI demands subgroup-level audits","One model: 0.61 overall AUPRC, 0.36 for Black women","Medical AI's overall accuracy hides disparate care","Subgroup performance: the missing metric in medical AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":2051,"prompt_tokens":911,"completion_tokens":1140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1059}},"tokens_in":527,"tokens_out":1140,"duration_ms":11196,"temperature":1.0,"reasoning_tokens":1059,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:52:18.416522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same fixed mortality model and evaluate it on a much larger independent test cohort with hundreds of Black patients, computing a tight confidence interval for the Black–White difference in precision-recall score; if the interval straddles zero or the gap disappears, the paper's headline disparity is sampling noise rather than a stable property of the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the in-hospital mortality prediction setting after the first 48 hours of admission."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multimodal time-series-and-text architecture used for the mortality model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Extends the multimodal fusion approach for medical text exploitation used in the mortality experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the gradient boosting model applied to graft failure prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the transparency and package-insert argument for AI-based health technologies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the governance framing of clinical decision-making with AI and the information-leaflet idea."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the regulatory hook: the EU AI Act requires instructions for use to include performance information for specific groups in high-risk systems."}],"review_version":1}