REVIEW 4 major objections 6 minor 1 cited by
MedOrchestra: A Hybrid Cloud-Local LLM Approach for Clinical Data Interpretation
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read MedOrchestra shows that a cloud-generated decomposition lets a local language model stage pancreatic cancer above specialist level without seeing patient data.
desk verdict Sensible hybrid architecture and a rare clinician benchmark, but the superiority claim needs a stronger evaluation before it lands. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is an orchestrator-executor split with a validation loop. The cloud orchestrator receives only the task description, the clinical guideline, and user preferences; it outputs subtasks, self-contained prompts, synthetic test cases, and rule-based outcome logic. The local executor runs those prompts on real reports and returns subtask outputs, which the rule-based logic combines into a final stage, with majority voting over repeated runs protecting against stochastic errors.
What would settle it
Recompute accuracy on the subset of reports whose bodies contain enough staging detail, using ground truth from multidisciplinary consensus rather than the withheld conclusions; if the 20-30% of free-text reports lacking body detail are where MedOrchestra beat the clinicians, the claimed superiority is an artifact of the evaluation design.
Extended reading notes
Core claim
The paper's central claim is that the limiting factor for local language models in clinical interpretation is not raw capability but task structure. Once a cloud LLM decomposes staging into concrete subtasks, drafts self-contained prompts, validates them on synthetic cases, and fixes rule-based aggregation logic, the local model can execute the reasoning chain and outperform both unaugmented local models and clinicians. On this 100-report pancreatic cancer dataset, that division of labor yields the highest accuracy and the highest agreement with ground-truth staging in both free-text and structured formats.
Load-bearing premise
The evaluation assumes the correct stage for each report, which was reconstructed from the report's conclusion, can actually be determined from the report body alone; the paper admits 20-30% of free-text reports do not contain enough detail outside the conclusion, so if a label is not supported by the body, every system and expert is being measured against information they never saw.
Editorial extensions
If this is right
- Cloud-generated subtask prompts can be reused for other guideline-based clinical tasks because they are derived from guidelines and synthetic cases rather than patient data.
- Institutions with strict data-export rules can still obtain cloud-level reasoning by transferring prompts once, without moving any clinical text off-site.
- The large gain from free-text to structured input suggests that standardizing report format may improve machine staging as much as choosing a stronger model.
- Handling speculative language such as 'likely' or 'suspicious for' remains the main barrier, since the paper finds it causes overstaging in free-text reports.
Reading between the lines
- The ground-truth labels are reconstructed from report conclusions, so a sturdier test would use imaging-based or multidisciplinary consensus staging; no such test appears in the paper, and the measured gap over clinicians could shrink.
- The framework implicitly assumes that a cloud model's decomposition and synthesis logic transfer across institutions, so a multi-institution replication would determine whether prompts need site-specific regeneration.
- A natural extension is to ablate the validation loop and measure how much of the gain comes from task decomposition rather than from the choice of the cloud model; the paper does not isolate this.
- The manual prompt-transfer step, listed as a limitation, could be turned into a versioned and auditable prompt-as-logic artifact, making the system's reasoning traceable in clinical audits.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. MedOrchestra is a hybrid cloud-local LLM framework for clinical text interpretation. A cloud LLM (Gemini 2.5 Pro) receives a task definition, clinical guidelines, and user preferences; it decomposes the task into subtasks, generates self-contained prompts, creates synthetic test cases, and refines prompts until they pass an 80% validation threshold, all without access to patient data. A local LLM (Gemma 3 27B) then executes the prompts on-site on real reports, and the system applies cloud-generated rule-based logic, with five repeated inferences and majority voting, to produce a final outcome. The authors evaluate the system on NCCN-based pancreatic cancer staging using 100 radiology reports (50 free-text, 50 structured) from a single Korean institution. They report that MedOrchestra achieves 70.21% and 85.42% accuracy on free-text and structured reports, respectively, outperforming local LLM baselines and three board-certified clinicians, with higher Cohen's kappa. The paper also includes a candid Limitations section.
Significance. The architecture is a sensible and potentially valuable contribution: it decouples high-level reasoning from PHI-bearing inference, so no protected data leave the hospital, and it uses synthetic validation to avoid fitting prompts to the test labels. If the evaluation were sound, the result would be practically relevant for privacy-preserving clinical NLP. The authors deserve credit for stating limitations plainly, including the ground-truth ambiguity and the 20–30% of free-text bodies lacking sufficient detail. However, the empirical evidence for the headline 'clear superiority' claim is currently insufficient: the ground-truth labels may encode information from withheld Conclusion sections, the sample sizes are small, and no uncertainty or significance analysis is provided.
major comments (4)
- [Section 4.1 and Limitations] Ground-truth labels were inferred by retrospectively mapping the original radiologist Conclusion sections into NCCN stages, while all evaluated systems received only the report body. The Limitations section explicitly admits that in 20–30% of free-text reports the body lacks sufficient supporting detail and the conclusion was necessary to determine the GT. This means the GT is not guaranteed to be recoverable from the input given to any model or clinician; accuracy then partly measures agreement with hidden conclusion content rather than staging ability from the body alone. This is load-bearing for the comparison against clinicians. Please report the number and proportion of the 47 evaluated free-text cases in which the body alone determines the stage, present accuracy on the body-sufficient subset for all systems, and either re-derive GT from bodies alone or justify that conclusions add no information beyond the body.
- [Section 5, Tables 1–2] All comparisons are based on 47 free-text and 48 structured reports after excluding indeterminate cases, yet no confidence intervals, significance tests, or effect sizes are reported. The free-text margin between MedOrchestra (70.21%, 33/47) and the surgeon group (65.96%, 31/47) is two cases; a paired analysis could easily change the ranking. Cohen's kappa values are also reported without uncertainty. Please provide exact counts, per-system confidence intervals, and paired significance tests (e.g., McNemar) on the same cases.
- [Section 4.3 and Section 5.3] The evaluation uses majority voting over five runs with the tie broken by selecting the higher clinical stage, described as conservative. Since the manuscript itself reports that MedOrchestra's main error mode is overstaging ambiguous language, this tie-breaking rule may systematically bias accuracy in an unknown direction. Please report the sensitivity of the headline accuracy to the tie-breaking rule and to the number of repeated inferences T, especially since the Limitations section states that the choice of repetitions was not systematically validated.
- [Section 4.1] The GT mapping procedure is under-specified: no number of annotators, inter-annotator reliability, or adjudication rule is given, and the treatment of indeterminate GT labels is unclear (whether the indeterminate flag was applied to GT labels or only to clinician annotations). Without this, label noise in the GT itself is unquantified and could affect all comparisons.
minor comments (6)
- [Section 1] There is a typo: 'polices' should be 'policies', and the affiliation line contains 'Univeristy' instead of 'University'.
- [Table 1] Please report the denominators (N=47 and N=48) alongside the percentages so readers can reconstruct exact counts.
- [Figure 3] The label 'MedEx' is inconsistent with the name 'MedOrchestra' used in the text; please unify, and provide counts for all cells of the 2x2 grid in both panels.
- [Algorithm 1] Algorithm 1 refers to 'TestSet_i' but the text defines synthetic examples as X_syn; please define this notation explicitly.
- [References] Several reference entries contain 'and 1 others' instead of complete author lists or consistent 'et al.' usage; please fix.
- [Abstract] The phrase 'showing clear superiority across all settings' is stronger than the current statistical evidence supports; please soften it or qualify it with the study's limitations.
Circularity Check
No circularity: MedOrchestra's prompts and outcome logic are generated without clinical data or ground-truth labels, and the evaluation is an external benchmark against independently derived NCCN stage labels.
full rationale
The claimed derivation—cloud LLM decomposes the task and generates subtask prompts plus rule-based outcome logic from the NCCN guideline, while the local LLM executes on report bodies—does not fit any parameter to the test labels. Equations (1)–(7) show that the only inputs to prompt and logic generation are the task description, guideline, and user preferences; real clinical data enter only at inference (Eq. 5). Prompt refinement (Algorithm 1) is validated on synthetic test cases generated from the guideline, not on the 100 real reports or on ground-truth labels. The GT labels were constructed independently by retrospectively mapping original Conclusion sections to NCCN stages, and the model and experts saw only report bodies; this creates a possible information asymmetry, and the manuscript's own Limitations section concedes that 20–30% of free-text bodies lack sufficient detail outside the conclusion. That is a serious dataset-validity limitation, but it is not circularity: the model is not trained or tuned on GT, and the comparison against clinicians could in principle have gone either way. No load-bearing self-citation, uniqueness import, ansatz-smuggling, or renaming of a known result is present. The central claim therefore has independent empirical content despite the noted evaluation weaknesses.
Assumptions & free parameters
free parameters (5)
- Prompt validation accuracy threshold =
80%
- Number of repeated inferences (T) =
5
- Tie-breaking rule =
Higher clinical stage
- Local inference temperature =
0.2
- Cloud inference temperature =
0.8
assumptions (5)
- domain assumption The NCCN guideline is accurately encoded in the cloud-generated outcome logic L.
- domain assumption Synthetic test cases generated by the cloud LLM are representative of real report ambiguity.
- domain assumption Ground truth staging labels inferred from original Conclusion sections are valid and learnable from report bodies.
- domain assumption The local LLM's JSON structured output mode faithfully captures requested clinical features.
- domain assumption Manual transfer of prompts into an air-gapped environment preserves the intended privacy separation.
Cite this review
Pith. "Pith review of MedOrchestra: A Hybrid Cloud-Local LLM Approach for Clinical Data Interpretation." pith.science (2026). https://pith.science/paper/VPKNXVQ4
@misc{pith2026250523806,
author = {Pith},
title = {Pith review of: MedOrchestra: A Hybrid Cloud-Local LLM Approach for Clinical Data Interpretation},
year = {2026},
howpublished = {\url{https://pith.science/paper/VPKNXVQ4}},
note = {Machine review of arXiv:2505.23806}
}
read the original abstract
Deploying large language models (LLMs) in clinical settings faces critical trade-offs: cloud LLMs, with their extensive parameters and superior performance, pose risks to sensitive clinical data privacy, while local LLMs preserve privacy but often fail at complex clinical interpretation tasks. We propose MedOrchestra, a hybrid framework where a cloud LLM decomposes complex clinical tasks into manageable subtasks and prompt generation, while a local LLM executes these subtasks in a privacy-preserving manner. Without accessing clinical data, the cloud LLM generates and validates subtask prompts using clinical guidelines and synthetic test cases. The local LLM executes subtasks locally and synthesizes outputs generated by the cloud LLM. We evaluate MedOrchestra on pancreatic cancer staging using 100 radiology reports under NCCN guidelines. On free-text reports, MedOrchestra achieves 70.21% accuracy, outperforming local model baselines (without guideline: 48.94%, with guideline: 56.59%) and board-certified clinicians (gastroenterologists: 59.57%, surgeons: 65.96%, radiologists: 55.32%). On structured reports, MedOrchestra reaches 85.42% accuracy, showing clear superiority across all settings.
Figures
Forward citations
Cited by 1 Pith paper
-
Changing the Paradigm from Dynamic Queries to LLM-generated SQL Queries with Human Intervention
An LLM-generated SQL query interface with editing and explanations is proposed to replace dynamic-query sliders in medical visualization.
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Soroosh Tayebi Arasteh, Mahshad Lotfinia, Keno Bressem, Robert Siepmann, Lisa Adams, Dyke Ferber, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, and Daniel Truhn. 2024. https://arxiv.org/abs/2407.15621 Radiorag: Factual large language models for enhanced diagnostics in radiology using online retrieval augmented generation . Preprint, arXiv:2407.15621
arXiv 2024
-
[4]
Luoyao Chen, Revant Teotia, Antonio Verdone, Aidan Cardall, Lakshay Tyagi, Yiqiu Shen, and Sumit Chopra. 2024. https://arxiv.org/abs/2410.09234 Fine-tuning in-house large language models to infer differential diagnosis from radiology reports . Preprint, arXiv:2410.09234
work page Pith review arXiv 2024
-
[5]
Andrea Cozzi, Katja Pinker, Andri Hidber, Tianyu Zhang, Luca Bonomo, Roberto Lo Gullo, Blake Christianson, Marco Curti, Stefania Rizzo, Filippo Del Grande, Ritse M. Mann, and Simone Schiaffino. 2024. https://doi.org/10.1148/radiol.232133 Bi-rads category assignments by gpt-3.5, gpt-4, and google bard: A multilanguage study . Radiology, 311(1):e232133. PMI...
-
[6]
Google DeepMind. 2025. Gemini 2.5 pro. https://deepmind.google/technologies/gemini/pro/
work page 2025
-
[7]
Putora, Marlene Schmerder, and Nikola Cihoric
Felix Dennstädt, Jack Hastings, Paul M. Putora, Marlene Schmerder, and Nikola Cihoric. 2025. https://doi.org/10.1038/s41746-025-01476-7 Implementing large language models in healthcare while balancing control, collaboration, costs and security . NPJ Digital Medicine, 8(1):143
-
[8]
Google. 2025. Gemma 3 model card. https://ai.google.dev/gemma/docs/core/model_card_3?hl=en
work page 2025
Show all 34 references
-
[9]
Kyuyoung Gu, Jae Ho Lee, Jaemin Shin, Ji Ae Hwang, Ji Hoon Min, Woo Kyoung Jeong, Mi Woo Lee, Ki Duk Song, and Sung Hoon Bae. 2024. https://doi.org/10.1111/liv.15891 Using GPT-4 for LI-RADS feature extraction and categorization with multilingual free-text reports . Liver Inter...
2024 doi
-
[10]
Isaac Hartsock, Carlos Araujo, Les Folio, and Ghulam Rasool. 2025. https://doi.org/10.1007/s10278-025-01510-w Improving radiology report conciseness and structure via local large language models . Journal of Imaging Informatics in Medicine
2025 doi
-
[11]
Yihao Hou, Christoph Bert, Ahmed Gomaa, Godehard Lahmer, Daniel Höfler, Thomas Weissmann, Raphaela Voigt, Philipp Schubert, Charlotte Schmitter, Alina Depardon, Sabine Semrau, Andreas Maier, Rainer Fietkau, Yixing Huang, and Florian Putz. 2025. https://doi.org/10.3389/frai.202...
2025
-
[12]
Y. H. Ke, L. Jin, K. Elangovan, and 1 others. 2025. https://doi.org/10.1038/s41746-025-01519-z Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness . NPJ Digital Medicine, 8:187
2025 doi
-
[13]
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. https://arxiv.org/abs/2210.02406 Decomposed prompting: A modular approach for solving complex tasks . Preprint, arXiv:2210.02406
2023 arXiv
-
[14]
Hyein Kim, Bora Kim, Min Hee Choi, Jin-Young Choi, Seung Nam Oh, and Sun Young Rha. 2025. https://doi.org/10.3348/kjr.2024.1228 Conversion of mixed-language free-text ct reports of pancreatic cancer to national comprehensive cancer network structured reporting templates by usi...
2025
-
[15]
Kumbhakarna, Sonali B
Vaishali M. Kumbhakarna, Sonali B. Kulkarni, and Apurva D. Dhawale. 2020. https://doi.org/10.35940/ijitee.L8009.1091220 Nlp algorithms endowed for automatic extraction of information from unstructured free-text reports of radiology monarchy . International Journal of Innovativ...
2020
-
[16]
Jaya Maharjan, Akhila Garikipati, N. P. Singh, and 1 others. 2024. https://doi.org/10.1038/s41598-024-64827-6 Openmedlm: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models . Scientific Reports, 14:14156
2024 doi
-
[17]
Supun Manathunga and Isuru Hettigoda. 2023. https://arxiv.org/abs/2309.02884 Aligning large language models for clinical tasks . Preprint, arXiv:2309.02884
2023 arXiv
-
[18]
Michael Marks and Christina E. Haupt. 2023. https://doi.org/10.1001/jama.2023.9458 Ai chatbots, health privacy, and challenges to hipaa compliance . JAMA, 330(4):309--310
2023
-
[19]
MetaAI. 2024. Llama 4. https://www.llama.com/models/llama-4/
2024
-
[20]
Jiin Nam, Seunghyun Yoon, and Kyomin Jung. 2019. https://doi.org/10.18653/v1/W19-5043 Surf at MEDIQA 2019: Improving performance of natural language inference in the clinical domain by adopting pre-trained language model . In Proceedings of the 18th BioNLP Workshop and Shared ...
2019 doi
-
[21]
including pet-ct and validation towards clinical use
J. M. Nobel, S. Puts, J. Krdzalic, K. M. L. Zegers, M. B. I. Lobbes, S. G. F. Robben, and A. L. A. J. Dekker. 2024. https://doi.org/10.1007/s10278-023-00913-x Natural language processing algorithm used for staging pulmonary oncology from free-text radiological reports: "includ...
2024 doi
-
[22]
OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 11 othe...
2024 arXiv
-
[23]
Chen, Eric Fosler-Lussier, and Albert M
Preethi Raghavan, Jason L. Chen, Eric Fosler-Lussier, and Albert M. Lai. 2014. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4333685/ How essential are unstructured clinical narratives and information fusion to clinical trial recruitment? AMIA Joint Summits on Translational Sci...
2014
-
[24]
Doris Reichenpfader, Hans Müller, and Kerstin Denecke. 2023. https://doi.org/10.1136/bmjopen-2023-076865 Large language model-based information extraction from free-text radiology reports: a scoping review protocol . BMJ Open, 13:e076865
2023 doi
-
[25]
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://arxiv.org/abs/2302.04761 Toolformer: Language models can teach themselves to use tools . Preprint, arXiv:2302.04761
2023 arXiv
-
[26]
Jana Sedlakova, Pascal Daniore, Annina Horn Wintsch, Michèle Wolf, Milica Stanikic, Carla Haag, Christian Sieber, Guido Schneider, Kaspar Staub, Daniel Alois Ettlin, Oliver Grübner, Fabio Rinaldi, and Viktor von Wyl. 2023. https://doi.org/10.1371/journal.pdig.0000347 Challenge...
2023 doi
-
[27]
Simon S uster, St \'e phan Tulkens, and Walter Daelemans. 2017. https://doi.org/10.18653/v1/W17-1610 A short review of ethical challenges in clinical natural language processing . In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing , pages 80--87,...
2017 doi
-
[28]
Duong, Jeremy Lampert, Paul Kovatch, Robert Freeman, Edgar Argulian, Lincoln Croft, Sotirios Lerakis, Matthew Goldman, Rohan Khera, and Girish N
Abhinav Vaid, Sophia Q. Duong, Jeremy Lampert, Paul Kovatch, Robert Freeman, Edgar Argulian, Lincoln Croft, Sotirios Lerakis, Matthew Goldman, Rohan Khera, and Girish N. Nadkarni. 2024. https://doi.org/10.1093/jamia/ocae085 Local large language models for privacy-preserving ac...
2024 doi
-
[29]
Josip Vrdoljak, Zvonimir Boban, Marino Vilović, Marko Kumrić, and Joško Božić. 2025. https://doi.org/10.3390/healthcare13060603 A review of large language models in medical education, clinical decision support, and healthcare administration . Healthcare, 13(6):603
2025 doi
-
[30]
Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024. https://arxiv.org/abs/2411.03350 A comprehensive survey of small language models in the era of large langu...
2024 arXiv
-
[31]
I. C. Wiest, D. Ferber, J. Zhu, and 1 others. 2024. https://doi.org/10.1038/s41746-024-01233-2 Privacy-preserving large language models for structured medical information retrieval . NPJ Digital Medicine, 7:257
2024 doi
-
[32]
Jiageng Wu, Xian Wu, and Jie Yang. 2024. https://doi.org/10.24963/ijcai.2024/829 Guiding clinical reasoning with large language models via knowledge seeds . In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI '24), pages 7491--74...
2024 doi
-
[33]
Shuqi Yang, Mingrui Jing, Shuai Wang, Jiaxin Kou, Manfei Shi, Weijie Xing, Yan Hu, and Zheng Zhu. 2025. https://arxiv.org/abs/2502.11861 Exploring large language models in healthcare: Insights into corpora sources, customization strategies, and evaluation metrics . Preprint, a...
2025 arXiv
-
[34]
Zijian Zhou, Miaojing Shi, Meng Wei, Oluwatosin Alabi, Zijie Yue, and Tom Vercauteren. 2024. https://arxiv.org/abs/2403.06728 Large model driven radiology report generation with clinical quality reinforcement learning . Preprint, arXiv:2403.06728
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.