Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 21:06 UTCgrok-4.3pith:UDKYDXFTrecord.jsonopen to challenge →
The pith
ClaMPAPP uses an LLM only to extract features from clinical narratives and an XGBoost classifier for prediction, outperforming end-to-end LLMs in pediatric appendicitis while minimizing missed cases.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
ClaMPAPP extracts schema-constrained clinical features from note-like narratives, applies deterministic plausibility checks, and passes validated features to an XGBoost classifier trained on clinical, laboratory, and ultrasound variables. On two independent pediatric appendicitis cohorts it achieved the strongest overall diagnostic performance in both internal and external validation while minimizing missed appendicitis cases. End-to-end LLM baselines showed unstable sensitivity-specificity trade-offs and greater degradation under narrative reordering.
What carries the argument
The ClaMPAPP pipeline, which uses an LLM strictly as a constrained feature extractor and interface while delegating prediction to an XGBoost classifier after deterministic checks.
If this is right
- The hybrid design minimizes missed appendicitis cases, the key safety concern in acute triage.
- End-to-end LLMs exhibit unstable sensitivity-specificity trade-offs and degrade more under narrative reordering.
- Separating natural-language usability from predictive inference yields a more auditable pathway for clinical decision support.
- The approach integrates narrative clinical workflows with tabular machine-learning inputs without requiring direct tabular entry.
Where Pith is reading between the lines
- The same interface-plus-predictor split could be tested on other acute conditions that rely on free-text notes.
- Real-world deployment would require measuring how often the deterministic plausibility checks reject LLM-extracted features.
- The method may reduce the need for prompt engineering by moving all inference stability into the downstream classifier.
Load-bearing premise
The template-rendered and constrained-LLM-rewritten narratives, including sentence-order permutations, serve as a valid proxy for real free-text clinical documentation without introducing artifacts that bias feature extraction or model performance.
What would settle it
ClaMPAPP performance falling below that of end-to-end LLMs when both are tested on actual unstructured physician-written notes rather than generated narratives would falsify the claimed advantage of the hybrid design.
Figures
read the original abstract
Large language models (LLMs) can make clinical decision support more accessible by interpreting free-text documentation, but their direct use as diagnostic engines is limited by sensitivity to prompts, information order, and plausible but incorrect outputs. Structured machine-learning models offer more stable risk prediction, yet they require tabular inputs that are difficult to integrate with narrative clinical workflows. We present ClaMPAPP (Clinical Language-assisted Machine-learning Pipeline for Appendicitis), a hybrid system that uses an LLM as an interface rather than as the final decision-maker. ClaMPAPP extracts schema-constrained clinical features from note-like narratives, applies deterministic plausibility checks, and passes validated features to an XGBoost classifier trained on clinical, laboratory, and ultrasound variables. We evaluated ClaMPAPP on two independent pediatric appendicitis cohorts from German hospitals and compared it with end-to-end LLM baselines, including open-source and proprietary models. To preserve ground truth while testing free-text input, narratives were generated from structured electronic health records through template rendering and constrained LLM rewriting, with additional sentence-order permutation to assess positional robustness. ClaMPAPP achieved the strongest overall diagnostic performance in both internal and external validation while minimizing missed appendicitis cases, the key safety concern in acute triage. End-to-end LLMs showed unstable sensitivity-specificity trade-offs and greater degradation under narrative reordering. These results support an LLM-as-interface, ML-as-predictor design that separates natural-language usability from predictive inference and provides a more auditable pathway for clinical decision support.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents ClaMPAPP, a hybrid system for pediatric appendicitis diagnosis that employs an LLM solely as an interface to extract schema-constrained features from note-like narratives, applies deterministic plausibility checks, and routes validated features to a trained XGBoost classifier. Narratives are constructed from structured EHR data via template rendering, constrained LLM rewriting, and sentence-order permutation to enable free-text testing while preserving ground truth. The system is evaluated on two independent German pediatric cohorts (internal and external validation) and compared against end-to-end LLM baselines; the central claim is that ClaMPAPP achieves the strongest overall diagnostic performance while minimizing missed appendicitis cases, with greater robustness to narrative reordering than direct LLM approaches.
Significance. If the generated narratives serve as a valid proxy, the work provides concrete evidence for an LLM-as-interface design that separates natural-language handling from stable predictive inference, offering improved auditability and safety over end-to-end LLM use in clinical triage. The explicit comparison on independent cohorts and focus on missed-case minimization address a key practical concern in acute care.
major comments (1)
- [Section 3] Section 3 (Methods) and abstract: The central performance claims (superiority on internal/external validation, minimized missed cases) rest entirely on test inputs that are template-rendered structured data further rewritten by constrained LLM and permuted. If real physician notes exhibit different lexical distributions, abbreviations, missingness patterns, or narrative flow, both the schema-constrained extraction and downstream XGBoost inputs could be unrealistically clean, inflating metrics relative to true free-text baselines. This assumption is load-bearing for the claim that the hybrid system outperforms end-to-end LLMs on free-text clinical documentation.
minor comments (1)
- [Abstract] Abstract: No quantitative metrics, confidence intervals, or baseline details are provided despite the strong performance claims; these should be summarized with effect sizes to allow readers to assess the magnitude of improvement.
Simulated Author's Rebuttal
We thank the referee for highlighting this important methodological consideration. We address the concern point by point below.
read point-by-point responses
-
Referee: [Section 3] Section 3 (Methods) and abstract: The central performance claims (superiority on internal/external validation, minimized missed cases) rest entirely on test inputs that are template-rendered structured data further rewritten by constrained LLM and permuted. If real physician notes exhibit different lexical distributions, abbreviations, missingness patterns, or narrative flow, both the schema-constrained extraction and downstream XGBoost inputs could be unrealistically clean, inflating metrics relative to true free-text baselines. This assumption is load-bearing for the claim that the hybrid system outperforms end-to-end LLMs on free-text clinical documentation.
Authors: We agree that the generated narratives constitute a controlled proxy rather than authentic physician notes and that this limits direct claims about performance on arbitrary real-world free-text. The design was chosen to retain exact ground-truth labels from the original structured EHR while enabling systematic testing of LLM extraction robustness (via constrained rewriting) and positional sensitivity (via permutation). Real free-text notes would require separate manual annotation to establish equivalent ground truth, which was outside the scope of the available datasets. We will revise the Methods, abstract, and Discussion to state this limitation more explicitly, qualify the generalizability claims, and note that future work should include prospective evaluation on raw physician documentation. revision: yes
- Direct empirical comparison on a corpus of authentic, physician-authored free-text notes with independently verified ground truth labels.
Circularity Check
No circularity in derivation chain
full rationale
The paper presents a hybrid system evaluated on two independent pediatric appendicitis cohorts with explicit comparisons to end-to-end LLM baselines. Narrative inputs are constructed via described template rendering and constrained LLM rewriting to simulate free-text while retaining structured ground truth; this is a methodological design choice, not a self-referential definition or fitted quantity renamed as prediction. No equations, self-citation load-bearing uniqueness theorems, ansatzes smuggled via citation, or renamings of known results appear that would reduce the reported performance metrics to the inputs by construction. The central claims rest on external validation sets and independent baselines, rendering the evaluation self-contained.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Synthetic narratives generated via template rendering and constrained LLM rewriting preserve ground truth labels and are representative of real clinical free-text notes.
Reference graph
Works this paper leans on
-
[1]
Reust and Amy Williams
Carin E. Reust and Amy Williams. Acute abdominal pain in children.American Family Physician, 93(10):830– 837, 2016
2016
-
[2]
Muhammad Waseem and Cecily F. Wang. Pediatric appendicitis. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing, 2025. Last updated June 17, 2025; accessed June 14, 2026
2025
-
[3]
Acute appendicitis in young children less than 5 years: Review article.Italian Journal of Pediatrics, 43(1):15, 2017
Hamdi Hameed Almaramhy. Acute appendicitis in young children less than 5 years: Review article.Italian Journal of Pediatrics, 43(1):15, 2017
2017
-
[4]
Quigley and Samuel Stafrace
Alan J. Quigley and Samuel Stafrace. Ultrasound assessment of acute appendicitis in paediatric patients: method- ology and pictorial overview of findings seen.Insights into Imaging, 4(6):741–751, 2013
2013
-
[5]
Mangona, R
Kate Louise M. Mangona, R. Paul Guillerman, Victor S. Mangona, Jennifer Carpenter, Wei Zhang, Monica Lopez, and Robert C. Orth. Diagnostic performance of ultrasonography for pediatric appendicitis: A night and day difference?Academic Radiology, 24(12):1616–1620, 2017
2017
-
[6]
Shah, Kelly A
Sohail R. Shah, Kelly A. Sinclair, Stephanie B. Theut, Kathy M. Johnson, George W. Holcomb, III, and Shawn D. St Peter. Computed tomography utilization for the diagnosis of acute appendicitis in children decreases with a diagnostic algorithm.Annals of Surgery, 264(3):474–481, 2016
2016
-
[7]
Ri ˇcards Marcinkevi ˇcs, Patricia Reis Wolfertstetter, Sven Wellmann, Christian Knorr, and Julia E. V ogt. Us- ing machine learning to predict the diagnosis, management and severity of pediatric appendicitis.Frontiers in Pediatrics, 9:662183, 2021
2021
-
[8]
Machine-learning based prediction of ap- pendicitis for patients presenting with acute abdominal pain at the emergency department.World Journal of Emergency Surgery, 19:40, 2024
Anoeska Schipper, Peter Belgers, Rory O’Connor, Kim Ellis Jie, Robin Dooijes, Joeran Sander Bosma, Steef Kurstjens, Ron Kusters, Bram van Ginneken, and Matthieu Rutten. Machine-learning based prediction of ap- pendicitis for patients presenting with acute abdominal pain at the emergency department.World Journal of Emergency Surgery, 19:40, 2024
2024
-
[9]
Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S
Samuel G. Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S. Kohane, and Suchi Saria. The clinician and dataset shift in artificial intelligence.New England Journal of Medicine, 385(3):283–286, 2021. 18 APREPRINT-
2021
-
[10]
M. Liu, T. Okuhara, X. Chang, R. Shirabe, Y . Nishiie, H. Okada, and T. Kiuchi. Performance of ChatGPT across different versions in medical licensing examinations worldwide: Systematic review and meta-analysis.Journal of Medical Internet Research, 26:e60807, 2024
2024
-
[11]
Glicksberg, and Eyal Klang
Dana Brin, Vera Sorin, Eli Konen, Girish Nadkarni, Benjamin S. Glicksberg, and Eyal Klang. How GPT models perform on the united states medical licensing examination: A systematic review.Discover Applied Sciences, 6(10):500, 2024
2024
-
[12]
Large language models in medicine.Nature Medicine, 29(8):1930–1940, 2023
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature Medicine, 29(8):1930–1940, 2023
1930
-
[13]
Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature Medicine, 30(9):2613–2622, September 2024
Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Viel- hauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature Medicine, 30(9):2613–2622, September 2024
2024
-
[14]
Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine
Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. New England Journal of Medicine, 388(13):1233–1239, 2023
2023
-
[15]
Xgboost: A scalable tree boosting system
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016
2016
-
[16]
Food and Drug Administration, Health Canada, and Medicines and Healthcare products Regulatory Agency
U.S. Food and Drug Administration, Health Canada, and Medicines and Healthcare products Regulatory Agency. Good machine learning practice for medical device development: Guiding principles, 2021. Accessed June 14, 2026
2021
-
[17]
Katikapalli Subramanyam Kalyan and Sivanesan Sangeetha. Secnlp: A survey of embeddings in clinical natural language processing.Journal of Biomedical Informatics, 101:103323, 2020. Also available as arXiv:1903.01039
-
[18]
O’Donnell, Rainu Kaushal, Yolanda Barrón, Mark A
Heather C. O’Donnell, Rainu Kaushal, Yolanda Barrón, Mark A. Callahan, Ronald D. Adelman, and Eugenia L. Siegler. Physicians’ attitudes towards copy and pasting in electronic note writing.Journal of General Internal Medicine, 24:63–68, 2009. Published online November 8, 2008
2009
-
[19]
Hirschtick
Robert E. Hirschtick. Copy-and-paste.JAMA, 295(20):2335–2336, 2006
2006
-
[20]
Radiomics in medical imaging: Methods, applications, and challenges
Fnu Neha and Deepak Kumar Shukla. Radiomics in medical imaging: Methods, applications, and challenges. arXiv preprint arXiv:2602.00102, 2026
-
[21]
A practical score for the early diagnosis of acute appendicitis.Annals of Emergency Medicine, 15(5):557–564, 1986
Alfredo Alvarado. A practical score for the early diagnosis of acute appendicitis.Annals of Emergency Medicine, 15(5):557–564, 1986
1986
-
[22]
Pediatric appendicitis score.Journal of Pediatric Surgery, 37(6):877–881, June 2002
Madan Samuel. Pediatric appendicitis score.Journal of Pediatric Surgery, 37(6):877–881, June 2002
2002
-
[23]
Weber, Matti Tolonen, Arianna Birindelli, Walter Biffl, Ernest E
Salomone Di Saverio, Mauro Podda, Belinda De Simone, Marco Ceresoli, Goran Augustin, Alice Gori, Marja Boermeester, Massimo Sartelli, Federico Coccolini, Antonio Tarasconi, Nicola de’ Angelis, Dieter G. Weber, Matti Tolonen, Arianna Birindelli, Walter Biffl, Ernest E. Moore, Michael Kelly, Kjetil Soreide, Jeffry Kashuk, Richard Ten Broek, Carlos Augusto G...
2020
-
[24]
Artificial intelligence for predicting acute appendicitis: A systematic review.ANZ Journal of Surgery, 93(9):2070–2078, 2023
Antoinette Lam, Emily Squires, Sheryn Tan, Ng Jeng Swen, Adriano Barilla, Joshua Kovoor, Aashray Gupta, Stephen Bacchi, and Sanjeev Khurana. Artificial intelligence for predicting acute appendicitis: A systematic review.ANZ Journal of Surgery, 93(9):2070–2078, 2023
2070
-
[25]
Hilbert, Vivien Rimili, Sven Wellmann, Chris- tian Knorr, Bertram Reingruber, Julia E
Ri ˇcards Marcinkeviˇcs, Kacper Sokol, Akhil Paulraj, Melinda A. Hilbert, Vivien Rimili, Sven Wellmann, Chris- tian Knorr, Bertram Reingruber, Julia E. V ogt, and Patricia Reis Wolfertstetter. External Validation of Predictive Models for Diagnosis, Management and Severity of Pediatric Appendicitis.Frontiers in Pediatrics, 13:1587488, 2025
2025
-
[26]
Capabilities of GPT-4 on Medical Challenge Problems
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems.arXiv preprint arXiv:2303.13375, 2023. 19 APREPRINT-
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[27]
Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H. Chen, Nigam H. Shah, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas,...
2025
-
[28]
Cool, Zahir Kanjee, Andrew S
Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A. Cool, Zahir Kanjee, Andrew S. Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P. J. Olson, Adam Rodman, and Jonathan H. Chen. Large language model influence on diagnostic reasoning: A randomized clinical trial.JAMA Network Open, 7(10):e2440969, 2024
2024
-
[29]
Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S
Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S. Chaudhari. Adapted large language models can...
2024
-
[30]
Retrieval-augmented generation (RAG) in healthcare: A comprehensive review.AI, 6(9):226, 2025
Fnu Neha, Deepshikha Bhati, and Deepak Kumar Shukla. Retrieval-augmented generation (RAG) in healthcare: A comprehensive review.AI, 6(9):226, 2025
2025
-
[31]
Benchmarking large language models in retrieval- augmented generation
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval- augmented generation. InProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence, pages 17754–17762, 2024
2024
-
[32]
Prompt perturbation in retrieval- augmented generation based large language models
Zhibo Hu, Chen Wang, Yanfeng Shu, Hye-Young Paik, and Liming Zhu. Prompt perturbation in retrieval- augmented generation based large language models. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1119–1130, 2024
2024
-
[33]
Ri ˇcards Marcinkeviˇcs, Patricia Reis Wolfertstetter, Ugne Klimiene, Kieran Chin-Cheong, Alyssia Paschke, Ju- lia Zerres, Markus Denzinger, David Niederberger, Sven Wellmann, Ece Ozkan, Christian Knorr, and Julia E. V ogt. Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitis. Medical Image Analysis, 91:1...
2024
-
[34]
Data-to-text summarisation of patient records: Using computer-generated summaries to access patient histories.Patient Education and Counseling, 92(2):153–159, 2013
Donia Scott, Catalina Hallett, and Rachel Fettiplace. Data-to-text summarisation of patient records: Using computer-generated summaries to access patient histories.Patient Education and Counseling, 92(2):153–159, 2013
2013
-
[35]
Scott H. Lee. Natural language generation for electronic health records.npj Digital Medicine, 1:63, 2018
2018
-
[36]
Lee, Sujay Jain, Alex Chen, Kyoka Ono, Arabdha Biswas, Ákos Rudas, Jennifer Fang, and Jeffrey N
Simon A. Lee, Sujay Jain, Alex Chen, Kyoka Ono, Arabdha Biswas, Ákos Rudas, Jennifer Fang, and Jeffrey N. Chiang. Clinical decision support using pseudo-notes from multiple streams of EHR data.npj Digital Medicine, 8:394, 2025
2025
-
[37]
SimSUM – simulated benchmark with structured and unstructured medical records.Journal of Biomedical Semantics, 16(1):20, 2025
Paloma Rabaey, Stefan Heytens, and Thomas Demeester. SimSUM – simulated benchmark with structured and unstructured medical records.Journal of Biomedical Semantics, 16(1):20, 2025
2025
-
[38]
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, Lu Yang, Kejia Chen, Per Bjornsson, Shashir Reddy, Ryan Br...
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[39]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 20 APREPRINT-
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[40]
GPT-5.5 system card
OpenAI. GPT-5.5 system card. Technical report, OpenAI, April 2026. Accessed June 12, 2026
2026
-
[41]
OpenAI o3 and o4-mini system card
OpenAI. OpenAI o3 and o4-mini system card. Technical report, OpenAI, 2025. Accessed June 12, 2026
2025
-
[42]
Claude Opus 4.7 system card
Anthropic. Claude Opus 4.7 system card. Technical report, Anthropic, April 2026. Published April 16, 2026; accessed June 12, 2026
2026
-
[43]
Openassistant conversations-democratizing large language model alignment.Advances in neural information processing systems, 36:47669–47681, 2023
Andreas Köpf, Yannic Kilcher, Dimitri V on Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment.Advances in neural information processing systems, 36:47669–47681, 2023
2023
-
[44]
Fleming, Michael A
Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott L. Fleming, Michael A. Pfeffer, Jason Fries, and Nigam H. Shah. The shaky foundations of large language models and foundation models for electronic health records.npj Digital Medicine, 6(1):135, 2023
2023
-
[45]
Large language models in healthcare and medical applications: A review.Bioengineering, 12(6):631, 2025
Subhankar Maity and Manob Jyoti Saikia. Large language models in healthcare and medical applications: A review.Bioengineering, 12(6):631, 2025. 21
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.