REVIEW 3 major objections 5 minor 46 references
CRS-Triage: Confidence- and Reliability-Aware Selective Triage under Incomplete Clinical Evidence
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Emergency triage gets safer when the model scores the reliability of each data source and defers low-confidence cases: under-triage drops from 13.1% to 9.6% on MIMIC-IV-ED.
desk verdict Solid applied clinical ML paper with a real gap in the reliability definition: it treats missingness as unreliability, and never tests the opposite direction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Structured reliability ρ^s (Eq. 7) is the weighted fraction of observed features; text reliability ρ^t (Eq. 8) combines text length, prediction certainty, and evidential certainty (Dirichlet-based). Disagreement δ (Eq. 9) is the normalized Jensen–Shannon divergence between the modalities' probability vectors; it drives independent, non-normalized fusion weights (Eq. 10), so unreliable or conflicting evidence is downweighted and both modalities can contribute little at once. The same signals, with fused uncertainty and margin, feed the learned confidence score s_i (Eq. 12) that decides predict-vs-defer. The selective loss (Eq. 13) minimizes accepted-case triage penalty under a coverage constr
What would settle it
Run CRS-Triage on a version of the held-out MIMIC-IV-ED set where missingness is informative: mask 30–50% of triage vitals preferentially from high-acuity (ESI 1–2) encounters while leaving low-acuity records intact. If reliability-as-completeness is sound, accepted high-acuity cases keep near the random-missingness under-triage rate; if deferral concentrates on the sickest and their penalty rises sharply, the scheme distrusts its most acute evidence. A simpler correlational check: if ESI 1–2 encounters have fewer observed structured features than ESI 4–5, the model is systematically discounti
Extended reading notes
Core claim
The claim: confidence and prediction must be learned together, not tuned afterward. CRS-Triage estimates, per encounter, the completeness of structured vitals, the reliability of the clinical text, and cross-modal disagreement (normalized Jensen–Shannon divergence). These signals weight each modality's evidence in evidential fusion (a Dirichlet-based mechanism yielding probabilities plus an uncertainty) and feed a learned confidence score that decides predict-vs-defer. Training minimizes expected triage penalty — under-triage weighted above over-triage — over accepted cases under a coverage constraint. On MIMIC-IV-ED this yields Macro-F1 0.742, under-triage down from 13.1% to 9.6%, and bette
Load-bearing premise
The load-bearing premise is that 'reliable structured data' equals 'complete structured data': Eq. 7 defines reliability as the weighted fraction of observed features, so a missing value always lowers it. If informative missingness runs the other way — the sickest patients have fewer vitals recorded at triage because care starts immediately — the model systematically downweights its most informative evidence exactly for the patients it exists to protect.
Editorial extensions
If this is right
- The CRS confidence score improves selective deferral at fixed coverage: at 80% coverage the expected triage penalty drops from 0.267 (evidential certainty) to 0.208, and under-triage among accepted cases from 5.9% to 4.7%.
- The under-triage penalty ratio is a policy dial: ratio 3 gives 9.6% under-triage and 15.1% over-triage; ratio 5 gives 8.0% and 18.5%, so a hospital can choose its position on the over-resourcing curve.
- Reliability weighting degrades gracefully under test-time perturbation: at 50% additional structured missingness, Macro-F1 stays at 0.681 and triage penalty at 0.520, versus 0.612 and 0.690 for base evidential fusion, and the margin over baselines is largest under cross-modal conflict.
- Deferred encounters carry substantially higher error and triage penalty than accepted ones, so the predict-or-defer decision concentrates human review on the cases the model is least suited to judge.
Reading between the lines
- The structured-reliability score treats 'reliable' as 'complete': a missing value always lowers it. If informative missingness runs the opposite way — the sickest patients have fewer vitals recorded because resuscitation starts immediately — the model would systematically distrust its most informative records; perturbing the test set by removing vitals preferentially from high-acuity encounters wo
- The coverage constraint is an average over training, and the confidence threshold is applied only at inference; letting the threshold respond to clinician workload or queue congestion is a natural extension the paper does not explore.
- Text length enters the text-reliability score, so a dense one-line note like 'STEMI' is scored less reliable than a padded note; weighting informativeness rather than length, and testing conflicts that co-occur with missingness (rather than in isolation), are testable refinements.
- The deferral decision is binary; a graded escalation — predict with a conservative acuity label rather than defer entirely when confidence is mid-range — is a natural follow-on that would soften the accuracy–coverage trade-off.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. CRS-Triage proposes a multimodal emergency triage model for incomplete and conflicting EHR data. Structured data and clinical text are encoded separately into evidential (Dirichlet) predictions; per-modality reliability scores are computed from feature completeness (Eq. 7) and text length/certainty (Eq. 8), then combined with cross-modal disagreement (Eq. 9) to weight modality evidence before fusion (Eqs. 10-11). A learned confidence score (Eq. 12) is trained with a selective loss (Eq. 13) and an asymmetric triage penalty (Eq. 2) to favor over-triage over under-triage, and defer low-confidence cases. On MIMIC-IV-ED, the model reports Macro-F1 0.742, QWK 0.846, under-triage rate 9.6% (vs. 13.1% for base evidential fusion), improved ECE, better risk-coverage trade-offs at 80% and 90% coverage, and favorable robustness under random masking, text degradation, and cross-modal conflict.
Significance. If the empirical claims hold, the paper's contribution is a useful end-to-end selective triage framework that jointly models modality reliability, cross-modal consistency, and asymmetric error costs. The evaluation has genuine strengths: a held-out patient-level split, results over five seeds, ablation studies, and perturbation tests that are not circular. The comparison to SelectiveNet and evidential baselines is appropriate. However, the significance is conditional on the reliability definition: structured reliability is operationalized as completeness (Eq. 7), and no direct evidence shows that the learned reliability scores track per-modality error. The informative-missingness failure mode is especially relevant for emergency triage and is not tested. As a result, the broad claim that CRS-Triage 'remains reliable under incomplete clinical evidence' is not yet fully established.
major comments (3)
- [Eq. (7) and Table 5] The structured-data reliability rho^s_i is a weighted fraction of observed features, i.e., reliability is equated with completeness. The robustness experiments mask observed entries randomly (MCAR), so they cannot detect a failure mode in which missingness is informative in the opposite direction. If high-acuity patients have fewer recorded vitals because they are taken to resuscitation immediately, Eq. 7 assigns low rho^s to exactly the patients the model should protect. Since rho^s enters the fusion weight q^s_i (Eq. 10) and the confidence score s_i (Eq. 12), the model may downweight or defer the most informative structured evidence for those patients. The missingness mask is provided to the encoder (Eq. 4), which is a partial mitigation, but no experiment verifies that this compensation is sufficient. Please add an MNAR-style evaluation or otherwise demonstrate that rho^s tracks actua
- [Eqs. (7)-(12), Table 1] The reliability scores are never directly validated. No table or figure reports, for example, the correlation or calibration between rho^s_i / rho^t_i and the actual per-modality classification error, either on clean or perturbed inputs. The paper's central claim is that fusion is 'reliability-aware,' but the only support is indirect through ablations. A direct analysis showing that low reliability scores correspond to higher per-modality error rates would substantially strengthen the paper and would address the concern that Eq. 7 is simply a completeness measure in disguise.
- [Eq. (17) and Experimental Setup] The training objective contains many hyperparameters that are never specified: lambda_pen, lambda_sel, lambda_dis, lambda_KL, lambda_aux, gamma, beta, kappa, epsilon, and the exact values of lambda_under/lambda_over beyond the ratio of 3 shown in Table 4. Without these values the multi-term loss is not reproducible, and the claimed advantage of joint training cannot be independently checked. Please provide the full hyperparameter configuration, including any scheduling or annealing (e.g., for lambda_KL).
minor comments (5)
- [Tables 1-3] No significance tests are reported over the five seeds. Many key differences appear separated by more than the reported standard deviations, but some metrics overlap (e.g., balanced accuracy for CRS-Triage is 0.734 +/- 0.009 vs. TMC's 0.738 +/- 0.009). Please add paired significance tests or confidence intervals.
- [Selective Prediction, Table 2] The confidence threshold tau is described as predefined, but the experiments at fixed coverage appear to use an empirical score quantile. Please state explicitly how tau is chosen in practice and whether the reported results use a fixed threshold or coverage-based selection.
- [Selective Prediction, Section 4.2] The text states that deferred encounters have 'substantially higher error rates and expected triage penalties,' but the deferred-case metrics are not shown in a table. Please include these numbers.
- [Overall text] Fix formatting issues such as 'to0.742from' in Section 4.1 and missing spaces around numbers/operators in equations and text. Also, the sentence beginning '418,100 adult encounters' should be rephrased for clarity.
- [Eq. (14)] If the KL regularization coefficient lambda_KL is annealed during training, as is common in evidential learning, please state the schedule. If it is fixed, that should also be stated explicitly.
Circularity Check
No significant circularity: CRS-Triage's reported improvements are empirical, out-of-sample results; the reliability and confidence components are designed mechanisms rather than definitions of the evaluation metrics.
full rationale
The derivation chain in CRS-Triage consists of designed equations (7)-(13), not a theorem whose conclusion is an input. The structured reliability ρ^s_i is defined as a weighted observed-feature fraction; the text reliability and fusion weights are learned functions; the confidence score is a learned combination of reliability, certainty, disagreement, and margin. None of these quantities is defined in terms of the evaluation outcomes (Macro-F1, triage penalty, under-triage rate) on the test set. The training objective L_sel in Eq. 13 includes the same expected triage penalty C_i that is later reported as an evaluation metric; this is an intentional alignment of the objective with the metric, not circular, because the parameters are fitted on the training split and the reported numbers are computed on a held-out test split. The ablation and sensitivity analyses change the objective and observe changed test-set metrics, which is an empirical comparison rather than a reduction. The robustness perturbations (random masking, token deletion, text replacement) alter inputs and the reliability/consistency scores respond mechanically, but whether the final predictions and deferrals remain accurate is an empirical question answered by Table 5; the score does not force the reported accuracy values. The paper's citations to evidential learning and conflictive multi-view learning are standard method attributions and are not used to import a uniqueness theorem or to forbid alternative designs. The informative-missingness concern raised by a reader is a potential limitation of the reliability operationalization, not a circularity: it questions whether ρ^s_i tracks true reliability, but it does not show that the reported result is equivalent to its inputs by construction. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (7)
- feature reliability weights η_j (j=1..d) =
unknown (learned)
- text reliability weights w_t_ρ and bias b_t_ρ =
learned
- fusion weights w_s_q, w_t_q and biases b_s_q, b_t_q =
learned
- confidence weights w_conf and bias b_conf =
learned
- under/over-triage penalty ratio =
3 (sensitivity tested at 1, 2, 3, 5)
- coverage target κ and coverage penalty β =
not reported
- loss coefficients λ_pen, λ_sel, λ_dis, λ_KL, λ_aux and disagreement gain γ =
not reported
assumptions (5)
- domain assumption The ESI label in MIMIC-IV-ED is the true triage class for each encounter.
- domain assumption The absence of a structured feature is monotonically associated with lower data reliability, so Eq. 7's availability ratio is a valid reliability proxy.
- domain assumption Text length is a useful reliability signal for chief-complaint text.
- domain assumption Cross-modal agreement implies higher prediction quality.
- domain assumption The training, validation, and test encounters are i.i.d. from the deployment distribution except for the controlled perturbations.
Cite this review
Pith. "Pith review of CRS-Triage: Confidence- and Reliability-Aware Selective Triage under Incomplete Clinical Evidence." pith.science (2026). https://pith.science/paper/3JOYHKST
@misc{pith2026260803862,
author = {Pith},
title = {Pith review of: CRS-Triage: Confidence- and Reliability-Aware Selective Triage under Incomplete Clinical Evidence},
year = {2026},
howpublished = {\url{https://pith.science/paper/3JOYHKST}},
note = {Machine review of arXiv:2608.03862}
}
read the original abstract
Emergency triage requires reliable decisions within a short time period. However, the available electronic health record (EHR) data, including structured data and clinical text, are often incomplete, unreliable, and inconsistent. This makes machine learning (ML)-based triage prediction more challenging, as existing ML models typically rely on complete and reliable EHR data to accurately predict patients' acuity levels. To address this, we propose confidence- and reliability-aware selective triage (CRS-Triage) to predict patients' acuity levels with a confidence score. By comparing the confidence score with a predefined threshold, CRS-Triage can selectively determine whether the model should make the decision or defer the case. Specifically, CRS-Triage separately evaluates the reliability of structured data and clinical text and then jointly considers the consistency between the two modalities to estimate the confidence of each prediction. Moreover, to reduce the risk of missing high-acuity patients, namely under-triage, CRS-Triage prefers to assign patients slightly higher acuity levels, namely over-triage, by penalizing under-triage errors. Experiments on the MIMIC-IV-ED dataset show that CRS-Triage achieves strong predictive performance. It also provides a better risk-coverage trade-off and remains reliable when the available EHR data are incomplete, degraded, or inconsistent across modalities.
Figures
Reference graph
Works this paper leans on
-
[1]
AMIA Annual Symposium Proceedings , year=
Benchmarking Emergency Department Triage Prediction Models with Machine Learning and Large Public Electronic Health Records , author=. AMIA Annual Symposium Proceedings , year=
-
[2]
Nature Machine Intelligence , volume=
Learning from data with structured missingness , author=. Nature Machine Intelligence , volume=. 2023 , publisher=
work page 2023
-
[3]
Proceedings of the AAAI conference on artificial intelligence , volume=
Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[4]
International conference on machine learning , pages=
Selectivenet: A deep neural network with an integrated reject option , author=. International conference on machine learning , pages=. 2019 , organization=
2019
-
[5]
Medfuse: Multimodal ehr data fusion with masked lab-test modeling and large language models , author=. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages=
-
[6]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Role of human-AI interaction in selective prediction , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[7]
Advances in Neural Information Processing Systems , volume=
Ehrshot: An ehr benchmark for few-shot evaluation of foundation models , author=. Advances in Neural Information Processing Systems , volume=
-
[8]
Multimodal fusion of ehr in structures and semantics: Integrating clinical records and notes with hypergraph and llm , author=. Medinfo 2025-Healthcare Smart x Medicine Deep: Proceedings of the 20th World Congress on Medical and Health Informatics , pages=. 2025 , organization=
work page 2025
Show all 46 references
-
[9]
Machine Learning for Healthcare Conference , pages=
Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention , author=. Machine Learning for Healthcare Conference , pages=. 2023 , organization=
2023
-
[10]
Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages=
M3care: Learning with missing modalities in multimodal healthcare data , author=. Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages=
-
[11]
arXiv preprint arXiv:2508.07617 , year=
On the limits of selective AI prediction: a case study in clinical decision making , author=. arXiv preprint arXiv:2508.07617 , year=
-
[12]
npj digital medicine , volume=
The shaky foundations of large language models and foundation models for electronic health records , author=. npj digital medicine , volume=. 2023 , publisher=
2023
-
[13]
International conference on machine learning , pages=
On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=
2017
-
[14]
arXiv preprint arXiv:2308.01222 , year=
Calibration in deep learning: A survey of the state-of-the-art , author=. arXiv preprint arXiv:2308.01222 , year=
-
[15]
npj Artificial Intelligence , year=
Uncertainty-aware abstention in medical diagnosis based on medical texts , author=. npj Artificial Intelligence , year=
-
[16]
PhysioNet , year=
Johnson, Alistair and Bulgarelli, Lucas and Pollard, Tom and Celi, Leo Anthony and Mark, Roger and Horng, Steven , title=. PhysioNet , year=. doi:10.13026/5ntk-km72 , url=
-
[17]
Archives of academic emergency medicine , volume=
Associated factors of under and over-triage based on the emergency severity index; a retrospective cross-sectional study , author=. Archives of academic emergency medicine , volume=
-
[18]
Journal of medical Internet research , volume=
Leveraging machine learning to identify subgroups of misclassified patients in the emergency department: Multicenter proof-of-concept study , author=. Journal of medical Internet research , volume=. 2024 , publisher=
2024
-
[19]
JMIR Medical Informatics , volume=
Interpretable deep learning system for identifying critical patients through the prediction of triage level, hospitalization, and length of stay: Prospective study , author=. JMIR Medical Informatics , volume=. 2024 , publisher=
2024
-
[20]
International conference on machine learning , pages=
Improving medical predictions by irregular multimodal electronic health records modeling , author=. International conference on machine learning , pages=. 2023 , organization=
2023
-
[21]
IEEE Journal of Biomedical and Health Informatics , volume=
Multimodal data matters: Language model pre-training over structured and unstructured electronic health records , author=. IEEE Journal of Biomedical and Health Informatics , volume=. 2022 , publisher=
2022
-
[22]
European conference on computer vision , pages=
Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration , author=. European conference on computer vision , pages=. 2022 , organization=
2022
-
[23]
Sentence-
Reimers, Nils and Gurevych, Iryna , booktitle =. Sentence-. 2019 , publisher =
2019
-
[24]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Think twice before selection: Federated evidential active learning for medical image analysis with domain shifts , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[25]
Advances in neural information processing systems , volume=
Evidential deep learning to quantify classification uncertainty , author=. Advances in neural information processing systems , volume=
-
[26]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Trusted multi-view deep learning with opinion aggregation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[27]
Scientific data , volume=
MIMIC-IV, a freely accessible electronic health record dataset , author=. Scientific data , volume=. 2023 , publisher=
2023
-
[28]
Proceedings of the 2nd clinical natural language processing workshop , pages=
Publicly available clinical BERT embeddings , author=. Proceedings of the 2nd clinical natural language processing workshop , pages=
-
[29]
Annals of emergency medicine , volume=
Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index , author=. Annals of emergency medicine , volume=. 2018 , publisher=
2018
-
[30]
JAMA Network Open , volume=
Evaluation of Version 4 of the emergency severity index in us emergency departments for the rate of mistriage , author=. JAMA Network Open , volume=
-
[31]
Scientific reports , volume=
Recurrent neural networks for multivariate time series with missing values , author=. Scientific reports , volume=. 2018 , publisher=
2018
-
[32]
Diagnostic and prognostic research , volume=
Informative missingness in electronic health record systems: the curse of knowing , author=. Diagnostic and prognostic research , volume=. 2020 , publisher=
2020
-
[33]
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingness , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
2025
-
[34]
Proceedings of the 2021 conference on empirical methods in natural language processing , pages=
How to leverage the multimodal EHR data for better medical prediction? , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=
2021
-
[35]
AMIA Annual Symposium Proceedings , volume=
A multimodal transformer: Fusing clinical notes with structured ehr data for interpretable in-hospital mortality prediction , author=. AMIA Annual Symposium Proceedings , volume=
-
[36]
NPJ digital medicine , volume=
Integrated multimodal artificial intelligence framework for healthcare applications , author=. NPJ digital medicine , volume=. 2022 , publisher=
2022
-
[37]
International Conference on Learning Representations , year=
Multimodal patient representation learning with missing modalities and labels , author=. International Conference on Learning Representations , year=
-
[38]
Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=
Flexcare: Leveraging cross-task synergy for flexible multimodal healthcare prediction , author=. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=
-
[39]
Advances in Neural Information Processing Systems , volume=
Are uncertainty quantification capabilities of evidential deep learning a mirage? , author=. Advances in Neural Information Processing Systems , volume=
-
[40]
International conference on machine learning , pages=
Consistent estimators for learning to defer to an expert , author=. International conference on machine learning , pages=. 2020 , organization=
2020
-
[41]
Nature Medicine , volume=
Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians , author=. Nature Medicine , volume=. 2023 , publisher=
2023
-
[42]
Scientific Reports , volume=
Conformal selective prediction with cost aware deferral for safe clinical triage under distribution shift , author=. Scientific Reports , volume=. 2026 , publisher=
2026
-
[43]
Trusted Multi-View Classification With Dynamic Evidential Fusion , year=
Han, Zongbo and Zhang, Changqing and Fu, Huazhu and Zhou, Joey Tianyi , journal=. Trusted Multi-View Classification With Dynamic Evidential Fusion , year=
-
[44]
JAMA network open , volume=
Machine learning--based prediction of clinical outcomes for children during emergency department triage , author=. JAMA network open , volume=
-
[45]
International Conference on Learning Representations , year =
Trusted Multi-View Classification , author =. International Conference on Learning Representations , year =
-
[46]
Proceedings of the AAAI conference on artificial intelligence , volume=
Reliable conflictive multi-view learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.