{"id":"ecc96a45-fd7b-462a-9450-e1940d9da28b","arxiv_id":"2505.04720","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Bayesian analysis of MICCAI 2023 papers estimates that most outperformance claims in medical imaging AI are not statistically substantiated at a 5% false-claim threshold.","lead":"This paper tests whether medical imaging AI papers that claim to beat the state of the art have statistical support, using a Bayesian model on 347 MICCAI 2023 papers. It estimates that over 86% of classification and 53% of segmentation papers have a greater than 5% chance that their top-ranked method does not truly outperform the second-ranked method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Congruence clamping turns the headline 86%/53% into worst-case estimates; the sensitivity analysis never explores feasible high-congruence values, so the central percentages are not robustly identified.","rationale":"The paper makes a credible and important qualitative point: the large majority of MICCAI 2023 classification and segmentation papers claim outperformance, very few use statistical significance testing, typical deltas are about 0.01, and test sets are often small. The Bayesian derivation is internally consistent, and the code is released. However, the quantitative headline depends on an unobserved quantity, model congruence, whose feasible range is only partially explored. For classification, the external median is infeasible for high-accuracy papers, and the clamping to the lower bound selects the endpoint that maximizes false-claim probability. The sensitivity analysis with Q1/Q3 still does not cover the upper part of the feasible interval, so it cannot establish that the 86% and 53% figures are typical rather than worst-case. The reader's weakest assumption correctly identified this issue, and the recommended conditionality is appropriate: the authors should release the extracted data, extend sensitivity across the full feasible congruence range, and temper the wording of the headline percentages. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":19427,"tokens_out":7864,"duration_ms":87590,"concrete_test":"Run the published code on the extracted per-paper dataset, after making a de-identified version available, and recompute the classification false-claim probabilities with p11 set to the upper feasible bound min(pA_hat, pB_hat) and the segmentation probabilities with rAB=0.95 or rAB=1. Compare the percentages of papers exceeding the 5% threshold with the reported 86% and 53%. Also report the distribution of feasible lower bounds for classification papers to document how many are affected by clamping. If the recomputed percentages fall substantially below the headline values, the central claim should be reframed as a worst-case analysis rather than an estimate of typical false-claim prevalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline percentages are valid only if the assumed model-congruence values are plausible for the analyzed MICCAI papers. For classification, the formula in Supplementary C.2 requires p11 to lie in [pA+pB-1, min(pA,pB)]. The external median congruence of 0.67 is below the lower bound for most MICCAI papers because their reported accuracies are around 0.9, so the procedure clamps p11 to pA+pB-1 for the majority of papers. This is the maximal-disagreement endpoint and it maximizes P(theta_A<=theta_B | reported results), so 86% is effectively a worst-case count under this clamping. The sensitivity analysis with Q3=0.83 does not resolve the issue because 0.83 is still below the lower bound for many high-accuracy papers; the upper half of the feasible interval is never explored. For the median classification paper (delta=0.01, n=500, accuracies about 0.91 versus 0.90), the upper endpoint gives P(false claim) of about 1%, whereas the lower endpoint gives about 30%, so the classification of that paper as a false claim depends entirely on the clamping choice. The same structure appears for segmentation, where the correlation rAB is varied only up to Q3=0.82 and the endpoint r=1 is never considered. Because the extracted dataset is not publicly available, the exact prevalence cannot currently be checked externally.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the validity of outperformance claims in medical imaging AI by analyzing a cohort of 347 MICCAI 2023 classification and segmentation papers. The authors document reporting practices (e.g., >80% of papers bold the best mean metric, yet <15% perform statistical testing), then introduce a Bayesian framework to compute the probability that the top-ranked method is not truly better than the second-ranked method given reported test set sizes, reported metric differences, and externally estimated 'model congruence' parameters. Using median congruence assumptions, they report that >5% false-claim probability occurs in 86% of classification papers and 53% of segmentation papers, and that demonstrating a 0.01 accuracy or DSC improvement requires test sets roughly 8-10 times larger than current medians. The paper provides detailed derivations in the Supplementary Information, as well as code and practical lookup tables for reviewers.","tokens_in":19704,"tokens_out":3000,"duration_ms":31717,"significance":"If the central prevalence numbers are robust, this is an important and field-relevant contribution: it quantifies a suspected weakness in benchmarking practices and supplies concrete, reusable formulas and sample-size guidance. Strengths include the carefully documented manual data extraction with double annotation and conflict resolution, the transparent Bayesian derivations in the Supplementary Information, the availability of code, and the practical framing for reviewers. The finding that typical reported deltas of 0.01 are often within sampling noise given common test set sizes is compelling and likely robust. However, the headline percentages (86% and 53%) depend critically on model-congruence values that are not directly observable in the analyzed papers; the current sensitivity analysis does not explore the full feasible range of these parameters, and in the classification case the procedure clamps the congruence to the endpoint that maximizes false-claim probabilities.","major_comments":[{"comment":"The classification false-claim probability depends on the model congruence p11, which is constrained to the interval [pA+pB-1, min(pA,pB)] as derived in Supplementary C.2. For the analyzed MICCAI papers, reported accuracies are often around 0.9, so the lower bound pA+pB-1 is typically 0.79-0.89. The externally estimated median congruence (0.67) and Q1 (0.47) fall below this lower bound, and the procedure replaces them with the lower bound itself, which is the maximal-disagreement endpoint and therefore maximizes the resulting false-claim probability. The sensitivity analysis in Figure 5 uses Q3=0.83, which is still below the upper bound min(pA,pB) for many high-accuracy papers; the upper half of the feasible interval (e.g., congruence values 0.9-0.99) is never explored. As a concrete illustration, for the median classification paper (delta=0.01, n=500, accuracies about 0.91 vs. 0.90), the upper endpoint yields a false-claim probability of roughly 1%, while the lower endpoint yields roughly 30%, so whether such a paper is counted as a false claim depends entirely on the clamping choice. The paper should either report results across the full feasible congruence interval for each paper, or provide evidence that near-maximal disagreement is realistic for the top-two methods in MICCAI 2023 papers.","section":"Methods, Data analysis subset; Supplementary C.2"},{"comment":"For segmentation, the congruence parameter rAB is varied only up to the 75th percentile of the MSD-derived correlation (Q3=0.82), while the endpoint rAB=1 (or values close to 1) is never considered. Since the segmentation false-claim formula in Supplementary C.3 uses (sA^2+sB^2-2 sA sB rAB)^(-1/2) in the Student t-statistic, higher positive correlation can dramatically reduce the estimated probability of a false claim, particularly at the small test set sizes (median 62) common in these papers. The current sensitivity analysis therefore does not bound the full range of possible prevalence estimates. At minimum, the paper should present results for rAB over the entire feasible range and clarify whether the external MSD correlation estimates are representative of the correlation between the top-two methods in MICCAI 2023 papers, which may be more aligned than the median of all pairwise method correlations.","section":"Figure 5 and Methods, segmentation congruence"},{"comment":"The extracted dataset from 347 papers is not publicly released; the authors state that it 'can be made available from the corresponding author upon reasonable request' due to concerns about individual author reputation. Because the headline percentages (86% and 53%) are computed from this specific extraction and depend on the congruence handling, the current manuscript does not allow independent verification of the central claims. A de-identified version with only test set sizes, reported metric values, and method rankings for the included papers would preserve privacy while enabling full reproducibility of the prevalence analysis.","section":"Data availability"}],"minor_comments":[{"comment":"In the sentence listing exclusion criteria, 'we had to exclude papers which only only cross-validation' contains a duplicated 'only'; please correct.","section":"Methods, Data analysis subset"},{"comment":"The phrase 'The integral 1 is not tractable analytically' should refer to 'Equation (1)' or 'the integral in Equation (1)' for clarity.","section":"Supplementary C.2"},{"comment":"Some Q3 values in Figure 6 appear non-monotonic (for example, the cell for test set size 500-1,000 and delta 0.005-0.01 lists Q3=0.38 while smaller-delta cells show higher Q3 values); please proofread the table for internal consistency.","section":"Figure 6"},{"comment":"The paper states that 'our sensitivity analyses confirm that the methodology remains robust across varying parameter assumptions,' but the sensitivity analysis only varies congruence up to Q3 (0.83 for classification, 0.82 for segmentation). Please temper this claim or extend the sensitivity analysis to the full feasible range.","section":"Main text, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The central statistical machinery is sound and the topic is highly relevant to the journal's readership. The main risk is that the headline percentages are presented as point estimates when, under the authors' own parameterization, they are effectively worst-case values under the congruence clamping. I would encourage the editor to ask for a substantial revision that either reframes the claims as upper bounds or performs a systematic sensitivity analysis over the full feasible congruence range, and to request that the extracted dataset (even in anonymized form) be released to allow external checking."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper is worth reading and worth refereeing, but the two headline numbers—86% of classification papers and 53% of segmentation papers with a >5% probability of false outperformance—are worst-case estimates, not robust point estimates. The stress-test note is correct. For classification, the external median congruence p11=0.67 falls below the feasible lower bound pA+pB−1 for most of the high-accuracy MICCAI papers, so the procedure clamps p11 to the lower bound, which maximizes the false-claim probability. The sensitivity analysis goes up to Q3=0.83, but for many papers that is still below the lower bound, so the upper half of the feasible interval is never explored. For the median classification paper, the false-claim probability swings from roughly 30% at the lower endpoint to about 1% at the upper endpoint. That means the classification of that paper as a false claim is entirely determined by the clamping choice.\n\nWhat is genuinely new: the systematic MICCAI 2023 audit (over 80% bold-face claims, only 10–13% with any significance testing, median delta 0.01) and the Bayesian false-claim framework with practical lookup tables for reviewers. The derivations in Supplementary C are mathematically correct; the Dirichlet and Student-t formulas are standard but cleanly adapted. The code is on GitHub. The reporting-practice findings alone are solid and independently useful.\n\nThe soft spots are real but bounded. The lack of public extracted data is a genuine checkability problem. The classification congruence estimate comes from the authors' private dataset, so the key parameter cannot be independently assessed. SD imputation for segmentation adds another layer of uncertainty, though the reported sensitivity there is more reassuring.\n\nI would not dismiss this paper. The qualitative conclusion—that typical MICCAI test sizes and deltas cannot substantiate claims of superiority—is robust and important. But the abstract's 86%/53% phrasing is too strong. For peer review, I would ask for (1) release of the extracted data (anonymized paper IDs, metrics, test sizes) to allow external recomputation; (2) sensitivity analysis over the full feasible congruence range, not just Q1/Q3; and (3) presentation of the false-claim rates as a range across congruence assumptions rather than a single number.\n\nThis paper deserves a serious referee. It will be cited regardless, and the field needs this conversation; the authors just need to make the central claim match the evidence.","headline":"A useful, important audit of MICCAI reporting practices, but the headline 86%/53% false-claim rates are worst-case estimates that depend on an unverified congruence assumption.","tokens_in":20350,"tokens_out":2370,"would_cite":true,"duration_ms":23736,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bayesian re-analysis finds most medical-imaging AI outperformance claims unsubstantiated: >5% chance of a false claim in 86% of classification and 53% of segmentation papers.","keywords":["medical imaging AI","outperformance claims","Bayesian analysis","model congruence","test set size","statistical significance testing","medical image classification","medical image segmentation"],"falsifier":"Take any set of medical-imaging papers that report per-test-sample predictions or confidence intervals, compute the actual accuracy congruence or Dice correlation between the top two methods, and recompute the false-claim probabilities; if the true congruence values approach the upper feasible bound, the share of papers exceeding the 5% threshold would drop well below the reported 86% and 53%.","tokens_in":19249,"feed_emoji":"🎲","tokens_out":12591,"duration_ms":116565,"temperature":0.7,"pith_summary":"This paper asks whether newly proposed medical-imaging AI methods genuinely beat the state of the art, and its answer is that for most published comparisons the reported numbers do not support the claim. It uses a Bayesian calculation that combines reported mean accuracy or Dice scores, test-set sizes, and an externally estimated 'model congruence' between competing methods to estimate the probability that the top-ranked method is actually no better than the second-ranked one. Applied to 347 MICCAI 2023 classification and segmentation papers, the calculation finds a high probability (>5%) of a false outperformance claim in 86% of classification papers and 53% of segmentation papers, with median reported improvements around 0.01. The point that matters is that gaps of this size are within the noise of typical test sets, so benchmarking practice as it stands can systematically misdirect research effort.","feed_headline":"Most medical-imaging AI outperformance claims likely arise by chance","feed_subtitle":"A >5% chance the runner-up actually wins was found for 86% of classification and 53% of segmentation papers.","key_machinery":"The central object is model congruence, the unmeasured degree to which two competing methods' per-sample errors align: for classification, the proportion of test items both methods classify correctly, $\\hat{p}_{1,1}$; for segmentation, the correlation $r_{AB}$ between the two methods' per-image Dice scores. Congruence enters two Bayesian identities that carry the argument: the classification posterior is $\\mathrm{Dirichlet}(x_1+1,\\,x_2+1,\\,n-x_1-x_2+2)$ with $x_1=n(\\hat{p}_A-\\hat{p}_{1,1})$ and $x_2=n(\\hat{p}_B-\\hat{p}_{1,1})$, and the segmentation posterior for the winner-minus-runner-up mean Dice is a Student-$t$ with $n-1$ degrees of freedom and spread $\\sqrt{s_A^2+s_B^2-2s_A s_B r_{AB}}$. Since no analyzed paper reports congruence, the authors take its empirical distribution from external tasks, replace each paper's value with the median (or Q1/Q3 in sensitivity analysis), and clamp into the feasible bounds implied by the reported accuracies; the clamped classification values are often at the lower bound $p_A+p_B-1$, the point of maximal disagreement between the top two methods.","core_discovery":"The paper's central claim is that most outperformance claims in medical imaging AI are unsubstantiated: given only the means, test-set sizes, and a reasonable assumption about how much the two leading methods agree, the probability that the runner-up is actually as good as or better than the winner exceeds conventional thresholds in the majority of papers. To get this probability it derives posterior distributions for the two true performances under non-informative priors — a Dirichlet posterior over accuracies for classification, a Student-$t$ posterior for the difference in mean Dice for segmentation — and evaluates them with Monte Carlo sampling. Because the agreement quantity (model congruence) is never reported, it is imported from external experiments at median values of 0.67 for both tasks, with each paper's value clamped to the feasible range. The headline numbers are 86% of classification and 53% of segmentation papers exceeding a 5% false-claim probability, and 58% and 25% exceeding 30%; the paper reads this as evidence that typical benchmarking claims do not survive a chance test.","pith_inferences":["The same calculation could be run on other venues and on metrics such as AUC or F1; because an image-level classification yields only one bit of evidence per sample, false-claim rates on such metrics are likely at least as high as those reported for accuracy.","A direct policy test of the paper's thesis: if the conference required authors to report per-sample predictions or confidence intervals, one could measure whether the distribution of claimed gains and test sizes shifted and whether replication rates improved.","The formulas also suggest a cheap reviewer-side check — treating any reported improvement below about 0.01–0.02 at test sizes of a few hundred samples as unproven without paired statistics — which the paper supports implicitly but does not itself state."],"forward_implications":["A reported accuracy gain of 0.01 needs a test set roughly eight times the current median of 500 images to get the false-claim probability below 5%.","A reported Dice gain of 0.01 needs roughly ten times the median segmentation test set of 62 images.","With fewer than 15% of papers doing any statistical significance testing, most published superiority claims currently contain no internal check that the winning method is distinguishable from noise.","Reviewers can use the paper's lookup tables or formulas to flag a candidate claim as likely false from the reported test size and performance gap alone.","If larger test sets and uncertainty reporting became standard, many 'state-of-the-art' results would need to be reframed, redirecting attention to methods whose gains actually exceed sampling noise."],"supporting_citations":[{"why":"Establishes that small sample sizes yield large error bars, the statistical premise for why common test sets cannot support 0.01-level claims.","marker":"[2]"},{"why":"Supplies the external multi-organ segmentation data used to estimate the correlation (congruence) between methods' Dice scores and to impute missing standard deviations.","marker":"[4]"},{"why":"Supply the private multi-task medical imaging data and software from which the accuracy congruence statistics for classification are derived.","marker":"[5, 6]"},{"why":"The five architectures applied across the classification tasks to generate the congruence values.","marker":"[16–20]"},{"why":"Provides the method used to impute missing standard deviations for segmentation papers.","marker":"[21]"},{"why":"Provides prediction intervals for that standard-deviation imputation, used in the sensitivity analysis.","marker":"[22]"},{"why":"Documents leakage and reproducibility failures cited as causes of inflated performance in the literature.","marker":"[1]"},{"why":"Supplies the publication-bias argument explaining why unsubstantiated claims still get published.","marker":"[3]"}],"fun_headline_variants":["Most medical imaging AI outperformance claims fail a chance test","Chance likely flips the winner in most medical AI papers","86% of classification studies risk false AI outperformance claims","Medical imaging AI: 53% of segmentation claims may be false wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall on the assumed model congruence — the unobserved agreement between the two methods' predictions — which is imported from external datasets and, for most classification papers, clamped to its lowest feasible value, a choice that maximizes the estimated probability of false claims.","fun_headline_variants_meta":{"raw":{"variants":["Most medical imaging AI outperformance claims fail a chance test","Chance likely flips the winner in most medical AI papers","86% of classification studies risk false AI outperformance claims","Medical imaging AI: 53% of segmentation claims may be false wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3375,"prompt_tokens":925,"completion_tokens":2450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2379}},"tokens_in":541,"tokens_out":2450,"duration_ms":15446,"temperature":1.0,"reasoning_tokens":2379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:24:10.653466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any set of medical-imaging papers that report per-test-sample predictions or confidence intervals, compute the actual accuracy congruence or Dice correlation between the top two methods, and recompute the false-claim probabilities; if the true congruence values approach the upper feasible bound, the share of papers exceeding the 5% threshold would drop well below the reported 86% and 53%.","supporting_citations":[{"cited_title":"Cross-validation failure: Small sample sizes lead to large error bars","cited_arxiv_id":null,"evidence_quote":"Establishes that small sample sizes yield large error bars, the statistical premise for why common test sets cannot support 0.01-level claims."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the external multi-organ segmentation data used to estimate the correlation (congruence) between methods' Dice scores and to impute missing standard deviations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the method used to impute missing standard deviations for segmentation papers."},{"cited_title":"France 2030","cited_arxiv_id":null,"evidence_quote":"Provides prediction intervals for that standard-deviation imputation, used in the sensitivity analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the publication-bias argument explaining why unsubstantiated claims still get published."}],"review_version":1}