Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports

Ali Yilmaz; Michael Bietenbeck; Sina Amirrajab; Volker Vehof

REVIEW 3 major objections 6 minor 1 cited by

Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Locally run open-source LLMs extracted cardiac MRI diagnoses from 109 German reports with F1 up to 0.98, and the top four beat a board-certified cardiologist.

desk verdict A plausible feasibility benchmark of nine open-source LLMs on German CMR report classification, undercut mainly by an unvalidated translation step and a small sample. read the letter →

arxiv 2506.00060 v1 pith:QROXZGT4 submitted 2025-05-29 cs.CY cs.AIcs.CL

Sina Amirrajab , Volker Vehof , Michael Bietenbeck , Ali Yilmaz This is my paper · ORCID

classification cs.CYcs.AIcs.CL

keywords largelanguagemodelsprivacy-preservingAICMRimagingreportsdiagnosticclassificationGermanclinicaltextopen-sourceLLMsdecisionsupportF1scoreevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether privacy-preserving, locally deployed open-source LLMs can extract diagnostic categories from free-text German cardiovascular magnetic resonance reports. It reports that most of nine models scored average F1 values above 0.93 on 109 reports spanning eight diagnostic categories and normal findings, with the best model reaching 0.98. The top four models also outperformed a board-certified cardiologist, whose F1 score was 0.94. The authors argue this demonstrates the feasibility of automated, resource-efficient imaging-report analysis without sending patient data to external cloud services.

What carries the argument

The machinery is a three-stage text pipeline: split each clinical report at the boundary between quantitative measurements and narrative findings, translate the German narrative into English with a multilingual LLM, and then feed the English text to a locally hosted open-source LLM through a standardized prompt that requests one of eight diagnostic categories in structured JSON. The classification task and the confusion-matrix analysis of misclassification pairs carry the comparison, with execution time per patient as the secondary performance measure.

What would settle it

Run the same 109 reports through a second independent translation and repeat the classification: if any model's $F_1$ drops below 0.93 or Gemma2-27B no longer leads, the pipeline's reported accuracy and ranking are artifacts of the LLaMA3.3 translation step.

Watch

Extended reading notes

Core claim

The paper's central claim is that a straightforward pipeline of splitting each report, translating its narrative text from German to English, and asking a locally hosted open-source LLM to choose one diagnostic category can classify CMR reports as accurately as or better than an expert reader. On the 109-report dataset, Gemma2-27B achieved the highest average F1 of 0.98, followed by Qwen2.5-32B at 0.96 and DeepSeek-R1-32B at 0.95, while all but two models stayed above 0.93. The top four LLMs exceeded the cardiologist's 0.94 F1 on every metric, and the fastest models processed a patient report in under half a second. The authors conclude that privacy-preserving, locally run LLMs are a viable mechanism for automated diagnostic categorization of imaging reports.

Load-bearing premise

The automatic German-to-English translation preserves all diagnostic content; if it introduces errors, every reported F1 score and the model ranking could change.

Editorial extensions

If this is right

Hospitals can run the full analysis on local hardware, so CMR reports never leave the institution; the reported accuracy suggests this does not require sacrificing classification quality.
Several mid-size open models match or exceed the performance of the expert reader on this task, making automated triage or decision support for CMR reporting a realistic near-term application.
The fastest models, at about 0.4 to 1.4 seconds per patient, are compatible with high-throughput clinical use, while the slowest reasoning model had both the longest runtime and worse accuracy.
The main remaining error pattern is myocarditis being confused with ischemic or dilated cardiomyopathy, which matches the human evaluator's difficulty and points to where further work is most needed.
The authors plan to scale the evaluation to more than 4,000 cases using a consensus of the three best models, which would test whether the accuracy holds beyond this small single-center set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to replace the automatic German-to-English translation with a German-native LLM or a professionally translated version of the same reports; if scores shift, part of the reported performance is an artifact of the translation step rather than the classification models themselves.
The evaluation forces a single diagnostic label per report, yet real CMR reports often describe multiple concurrent conditions; moving to multi-label classification could lower per-label precision and reveal a ceiling not visible in this eight-way forced-choice design.
Because the reports come from one center and were selected by one cardiologist for well-established diagnoses, the absolute F1 values should be re-estimated on unselected multi-center data before relying on them in production.
The human benchmark is a single cardiologist with one year of experience; comparing against a more experienced reader or a panel could change how the 'beats the cardiologist' result is interpreted.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Quick take: this is a useful feasibility benchmark, not a clinical study. The authors compare nine open-source LLMs on a new dataset of 109 German CMR reports, all run locally via Ollama. Most models do well, with average F1 above 0.93 and Gemma2-27B at 0.98. That result is plausible and the benchmark is a legitimate contribution to the LLM-in-medicine literature.

What's genuinely new: this specific task — classifying German CMR reports into eight diagnostic categories with open-source models — appears not to have been benchmarked before. The local deployment angle matters for GDPR-constrained hospitals. The paper is clearly written, reports confusion matrices and execution times, and uses external ground truth from an expert cardiologist.

Now the soft spots. The biggest is the unvalidated German-to-English translation step. All 109 reports are translated with LLaMA3.3, which is itself one of the nine evaluated models. No back-translation or human rating is reported. Since the F1 gaps are small (0.93 to 0.98) and the cardiologist comparison is made against the English text while the human reads German, the ranking could shift with a different translation. The Discussion acknowledges the small dataset but not this step. That is a moderate but real limitation, not a fatal one. The relative ordering of the models is likely still informative, but the absolute scores and the "outperforms cardiologist" claim are not well-supported.

Other issues: no confidence intervals or significance tests; the human comparator is a board-certified cardiologist with only one year of experience; the method section has inconsistent model names (LLaMA 3.2/3.3 vs LLaMA 3 2.3B/3B); and no mention of temperature or sampling settings, which matters for reproducibility. The dataset is not released, so the result is not independently reproducible.

Who benefits: groups thinking about local LLM deployment for German-language medical text, and anyone benchmarking open-source LLMs on non-English clinical data. It deserves serious peer review, though the revision should add translation validation, confidence intervals, and a consistent model list.

Best, [You]

Referee Report

3 major / 6 minor

Summary. This feasibility study evaluates nine locally deployed open-source LLMs on a classification task: assigning 109 German clinical CMR reports (translated to English by LLaMA3.3) to one of several cardiac diagnostic categories. The authors report accuracy, precision, recall, F1, confusion matrices, and per-patient execution times, and compare the models with a board-certified cardiologist. The headline result is that most models achieve average F1 scores above 0.93, with Gemma2-27B reaching 0.98, and that the top four models outperform the cardiologist (F1 0.94).

Significance. If the results are robust, the paper provides a useful and timely feasibility demonstration that open-source LLMs can be run locally for privacy-preserving classification of German CMR narrative reports. The study has concrete strengths: a clearly described preprocessing pipeline (report splitting, translation, local inference with Ollama), a uniform evaluation of nine models on the same clinical task, structured JSON output for machine-readable classification, confusion matrices per model, and transparent reporting of execution times and hardware. The claim that locally run open-source models can achieve high classification performance with modest compute is clinically relevant and supports the broader trend toward privacy-compliant use of LLMs in hospitals. The central contribution is empirical comparison rather than methodological novelty, and the significance is currently bounded by missing validation of the translation step and by the absence of statistical inference, which limits the strength of the comparative claims.

major comments (3)

[Methods, Preprocessing; Figure 4] The German-to-English translation step using LLaMA3.3 is unvalidated, yet every reported F1 score and the comparison with the cardiologist depend on this translation. The manuscript provides no back-translation, no human rating of translation fidelity, and no comparison with an independent translator or a second translation method. Because diagnostic categories hinge on specific modifiers (e.g., subepicardial, contrast enhancement, ECV), a single mistranslation can change the predicted class. Additionally, LLaMA3.3 is itself one of the nine evaluated classifiers, so its F1 may partly reflect self-consistency with its own translation rather than independent CMR understanding. I ask the authors to add a translation validation (at minimum on a sample of reports) and, ideally, a sensitivity analysis in which a subset of reports is translated by an independent method or by a human and the downstream classification is re-run.
[Methods, Statistics; Results, Figures 1 and 2] No confidence intervals, significance tests, or measures of agreement are reported for the classification metrics. With only 109 reports, the difference between the best model (F1 0.98) and the cardiologist (F1 0.94) could be within sampling variability; the same applies to the small gaps among the top models (0.95 to 0.98). Because the same 109 reports are evaluated by all models, paired methods such as bootstrap confidence intervals for paired data or a McNemar test on the confusion between model and human classifications would be appropriate. Without such error bars, the ranking and the 'outperformed the cardiologist' claim are not statistically supported.
[Results; Discussion; Abstract] The claim that 'the top four LLMs outperformed our board-certified cardiologist across all evaluation metrics' is not a controlled comparison as presented. The human evaluator is described as a board-certified cardiologist with one year of experience, and it is unclear whether this human read the original German reports or the translated English text, whether the same category list and output format were used, and whether the human was given the same standardized instruction. If the human assessed the original German reports while the LLMs assessed translated English text, the comparison is confounded by input language and task format. I ask the authors to either report complete details of the human evaluation protocol or soften the claim to a descriptive observation that the top models had higher point estimates than this one human reader under different reading conditions.

minor comments (6)

[Methods, Framework and Models; Results] The model name 'DeepSeek-V1' appears in the Results (approximate F1 0.76), but the Methods list only DeepSeek's distilled R1 variants (70B, 32B, 7B). Please clarify whether this is a typo for one of the R1 models or an additional model, and ensure the naming is consistent across the abstract, methods, results, and figures.
[Abstract; Results] The abstract states that 'all other evaluated models attained average scores above 0.93' and immediately adds that Mistral and DeepseekR1-7B were the only exceptions; this is internally contradictory. Please rephrase to state precisely which models exceeded 0.93 and which did not.
[Methods, Cardiovascular Magnetic Resonance Imaging Reports] The listed cohort counts (HCM n=15, CA n=14, CS n=14, MYO n=14, ICM n=17, DCM n=16, normal n=19) sum to 109 but identify only seven categories, while the text and Figure 2 refer to eight diagnostic categories. Please clarify what the eighth category is or correct the count, and make the category list consistent throughout the manuscript.
[Methods, Statistics and Framework and Models] The exact prompt template and decoding settings (temperature, max tokens, sampling strategy) are not provided. Since LLM output is sensitive to prompt phrasing, please include the full prompt and generation parameters as supplementary material to support reproducibility.
[Methods, Cardiovascular Magnetic Resonance Imaging Reports; Statistics] The manuscript mentions a cardiologist with extensive experience who selected the reports and defined the ground truth, and a board-certified cardiologist with one year of experience who served as the human evaluator. Please clarify whether these are different individuals and describe the ground-truth validation process (e.g., consensus, independent review).
[Results; Figure 3] The text and figure caption refer to 'LLaMA 3 (2.3B)' and 'LLaMA 3 (3B)', but the Methods list LLaMA 3.2 (3B) and LLaMA 3.3 (70B). Please check the model names and parameter counts for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is an external benchmark with ground-truth labels; the LLaMA3.3 translation step is a performance confound, not a definitional reduction.

full rationale

No circularity found. The study is an empirical benchmark: nine open-source LLMs classify 109 German CMR reports, translated to English by LLaMA3.3, into eight diagnostic categories, and the outputs are compared against externally defined ground-truth labels selected by an experienced cardiologist. No model parameter is fitted to the evaluation set, no diagnostic category is defined in terms of a model's output, and no load-bearing claim depends on a self-citation; the cited references are model documentation and general medical LLM literature. The use of LLaMA3.3 as both translator and evaluated classifier is a potential performance confound, because that model sees text it generated itself, but it is not circular: the target labels and the scoring metric are independent of LLaMA3.3's translation, and translation fidelity is an empirical preprocessing question rather than a definitional equivalence. Thus the central claim—most models exceed 0.93 F1 and top models exceed the cardiologist—rests on an external benchmark, not on the models' own outputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical measurement, not a derivation. It rests on the correctness of the reference labels, the fidelity of German-to-English translation by LLaMA3.3, the fairness of the single prompt across models, and the representativeness of the 109-report single-center sample. None of these are quantified or validated in the paper, so the classification scores inherit uncertainty from all four.

assumptions (4)

domain assumption The reference diagnoses used as ground truth are correct.
A cardiologist selected reports based on diagnostic criteria, but no second reader or adjudication is reported; errors in ground truth would directly bias all metrics.
domain assumption The LLaMA3.3 translation preserves diagnostic meaning.
The pipeline translates German to English with LLaMA3.3; any loss of nuance in translation affects classification performance and is not evaluated.
ad hoc to paper The single standardized prompt is equally fair to all models.
Models vary in instruction-following; a single prompt may favor certain models, and prompt sensitivity is not tested.
domain assumption The class distribution in the 109 reports is representative of the clinical population.
The sample is small and single-center; class sizes are 14-19, so macro-F1 has high variance and may not generalize.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports." pith.science (2026). https://pith.science/paper/QROXZGT4

@misc{pith2026250600060,
  author       = {Pith},
  title        = {Pith review of: Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QROXZGT4}},
  note         = {Machine review of arXiv:2506.00060}
}

read the original abstract

Purpose: We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. Materials and Methods: We evaluated nine open-source LLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models. Results: Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories. Google's Gemma2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32B with F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistral and DeepseekR1-7B being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports. Conclusion: Our findings demonstrate the feasibility of implementing open-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fast and resource-efficient diagnostic categorization.

Figures

Figures reproduced from arXiv: 2506.00060 by the authors.

**Figure 1.** LLM comparison in terms of different evaluation metrics. The best model is determined based on the highest mean score across metrics ordered from left to right. The performance of human evaluator is highlighted. The Mistral model performed slightly lower, with scores around 0.89, while DeepSeek-V1 had the weakest performance at approximately 0.76. Most models exhibited consistent performance across all four metrics,… view at source ↗

**Figure 2.** depicts confusion matrices illustrating model performance and human performance in classifying eight cardiac diagnostic categories from unstructured imaging reports. Strong diagonal dominance in Gemma2-27b, DeepSeek-R1-32b, and Qwen2-5-32b, indicates high accuracy, with occasional misclassifications observed for myocarditis (MYO). challenges in differentiating pathologically similar diseases. Myocarditis, in particu… view at source ↗

**Figure 3.** shows the execution time (seconds per patient) for each LLM and human evaluator, highlighting significant efficiency differences. Smaller models like LLaMA 3 (2.3B) and Mistral were the fastest, while larger models, such as DeepSeek-R1 (70B), had the highest execution time without a clear accuracy advantage. performances above 0.93 across most evaluation metrics, highlighting their effectiveness and usefulness in au… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The proposed pipeline for analyzing imaging reports to automatically extract diagnostic information from the text, including three steps to 1) split clinical reports into a first quantitative and a second narrative information, 2) translate the narrative text from Germ…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Development of an automated, reliable, and clinically meaningful artificial intelligence (AI) tool for diagnosing cardiac disease from conventional cardiovascular magnetic resonance (CMR) images
cs.CV 2026-07 conditional novelty 6.0 of 10

A five-class CMR disease classifier built from LLM-derived report labels and three fine-tuned vision foundation models reached ensemble AUCs of 0.84–0.97 on a single-center test set.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

McDonagh, T. A. et al. 2021 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur. Hear. J. 42 (2021)

work page 2021
[2]

Ponikowski, P. et al. 2016 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur. Hear. J. 37 (2016)

work page 2016
[3]

Hundley, W. G. et al. Society for cardiovascular magnetic resonance (SCMR) guidelines for reporting cardiovascular magnetic resonance examinations. J. Cardiovasc. Magn. Reson. 24 (2022). 5/6

work page 2022
[4]

Bunck, A. C. et al. Structured reporting in cross-sectional imaging of the heart: Reporting templates for cmr imaging of cardiomyopathies. RoF o F ortschritte auf dem Gebiet der Rontgenstrahlen und der Bildgebenden V erfahren192 (2020)

work page 2020
[5]

Zhang, K. et al. Revolutionizing health care: The transformative impact of large language models in medicine. J Med Internet Res 27 (2025)

work page 2025
[6]

Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Medicine 29, 1930–1940 (2023)

work page 2023
[7]

Meng, X. et al. The application of large language models in medicine: A scoping review. iScience 27, 109713 (2024)

work page 2024
[8]

Clusmann, J. et al. The future landscape of large language models in medicine. Commun. Medicine 3 (2023)

work page 2023

Show all 16 references

[9]

Park, Y . J.et al. Assessing the research landscape and clinical utility of large language models: a scoping review. BMC Med Inf. Decis Mak 24 (2024)

2024
[10]

Gilbert, S. et al. Large language model ai chatbots require approval as medical devices. Nat. Medicine 29, 2396–2398 (2023)

2023
[11]

Pauws, S. et al. Bridging the gap: a practical step-by-step approach to warrant safe implementation of large language models in healthcare. Front Artif Intell 8 (2025)

2025
[12]

Grattafiori, A. et al. The llama 3 herd of models (2024). ArXiv:2407.21783v3

2024 arXiv
[13]

G., T. et al. Gemma 2: Improving open language models at a practical size (2024). ArXiv:2408.00118v3

2024 arXiv
[14]

Jiang, A. Q. et al. Mistral 7b (2023). ArXiv:2310.06825v1

2023 arXiv
[15]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025)

DeepSeek-AI et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025). ArXiv:2501.12948v1

2025 arXiv
[16]

Peter-Lancier-Stiftung

Qwen et al. Qwen2.5 technical report (2024). ArXiv:2412.15115v2. Acknowledgements This study was supported by a research grant from the "Peter-Lancier-Stiftung", Hamburg, Germany (project title "OPT-AI- CMR", ID BD6505565). Author contributions statement S.A., first author, co...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

McDonagh, T. A. et al. 2021 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur. Hear. J. 42 (2021)

work page 2021

[2] [2]

Ponikowski, P. et al. 2016 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur. Hear. J. 37 (2016)

work page 2016

[3] [3]

Hundley, W. G. et al. Society for cardiovascular magnetic resonance (SCMR) guidelines for reporting cardiovascular magnetic resonance examinations. J. Cardiovasc. Magn. Reson. 24 (2022). 5/6

work page 2022

[4] [4]

Bunck, A. C. et al. Structured reporting in cross-sectional imaging of the heart: Reporting templates for cmr imaging of cardiomyopathies. RoF o F ortschritte auf dem Gebiet der Rontgenstrahlen und der Bildgebenden V erfahren192 (2020)

work page 2020

[5] [5]

Zhang, K. et al. Revolutionizing health care: The transformative impact of large language models in medicine. J Med Internet Res 27 (2025)

work page 2025

[6] [6]

Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Medicine 29, 1930–1940 (2023)

work page 2023

[7] [7]

Meng, X. et al. The application of large language models in medicine: A scoping review. iScience 27, 109713 (2024)

work page 2024

[8] [8]

Clusmann, J. et al. The future landscape of large language models in medicine. Commun. Medicine 3 (2023)

work page 2023