REVIEW 5 major objections 5 minor 26 references
Performance and Practical Considerations of Large and Small Language Models in Clinical Decision Support in Rheumatology
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that a smaller mixture-of-experts language model, Mixtral-8x7b-32768, paired with retrieval-augmented generation, reaches higher diagnostic and therapeutic accuracy than larger models on a ten-case rheumatology…
desk verdict A useful pilot benchmark for RAG in rheumatology, but the headline claim that SLMs beat LLMs is not supported by n=10 guideline-aligned cases with no statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is retrieval-augmented generation: a pipeline that turns clinical guideline documents into a vector index, retrieves the most relevant passages for each case, re-ranks them, and inserts them as context before the model generates a diagnosis or treatment plan. The paper also credits Mixtral-8x7b-32768's mixture-of-experts architecture — many specialized subnetworks activated per token — with letting a relatively small model use that external knowledge effectively. Two metrics carry the comparison: F1 for agreement between model output and guideline-based ground truth rated by experts, and a retrieval-augmented generation assessment score for faithfulness and relevance of the generated text.
What would settle it
Run the identical protocol on a larger, systematically sampled set of rheumatology cases with a blinded expert panel; if Mixtral-8x7b-32768 with retrieval no longer matches or beats the larger models, the central comparison collapses. A simpler check is to recompute the ranking using only cases where both raters agree on the correct diagnosis and see whether the leading model changes.
Extended reading notes
Core claim
The central claim is that on a rheumatology decision-support benchmark built from ten anonymized clinic cases, the 8x7-billion-parameter mixture-of-experts model Mixtral-8x7b-32768, when given guideline passages retrieved on the fly, outperforms all larger models tested in both diagnosis and treatment planning when no diagnosis is supplied. The paper reports F1 scores of 72% for diagnosis and 73% for treatment in that configuration, and its highest retrieval-augmented generation assessment score (81%). It also finds that retrieval's benefit is not uniform: Nemotron performs well without retrieval, while Claude-3.5-Sonnet has a high baseline that retrieval does not improve. The conclusion is that smaller models with retrieval can match or exceed frontier models on a narrow clinical task while being cheaper, using less energy, and running locally.
Load-bearing premise
The entire ranking depends on the ten standardized patient cases and their guideline-derived ground truth being representative of real rheumatology decisions, and on the two expert raters' scoring being consistent enough that the F1 and retrieval-quality metrics reflect true clinical correctness.
Editorial extensions
If this is right
- If the result generalizes, smaller models with retrieval can deliver useful clinical decision support in hospitals that cannot afford frontier-model APIs or their energy demands.
- Retrieval should be evaluated per model rather than assumed beneficial, since the paper shows it helps some models and not others.
- Because no model reached specialist-level accuracy, deployment will keep a human expert in the loop and require validation that catches clinically dangerous errors that small F1 gaps can hide.
- The same retrieval pipeline could support earlier triage and prioritization, potentially shortening the long diagnostic delay characteristic of rheumatoid arthritis.
Reading between the lines
- A likely extension of the paper's logic is that general-purpose LLM leaderboards are weak predictors of narrow medical performance, and that retrieval quality may matter more than parametric knowledge for such tasks.
- Re-running the protocol on a larger, independently ratified case set could show whether the Mixtral-with-retrieval advantage is stable or an artifact of the ten cases.
- The cost and energy results imply a concrete adoption path: keep a small local model plus a guideline index for routine cases, and escalate only low-confidence outputs to a cloud model or a specialist.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports an experimental comparison of five language models (GPT-4o, Mixtral-8x7b-32768, Nemotron-70b-instruct, Qwen-Turbo 2.5, Claude-3.5-Sonnet) on ten standardized rheumatology patient cases derived from anonymized records. Each model is tested under four conditions (with/without RAG and with/without a pre-specified diagnosis). Outcomes are F1 scores for diagnosis and treatment recommendations and RAGAS scores. The headline claim is that the SLM Mixtral-8x7b-32768 with RAG achieved the highest diagnostic and therapeutic F1 when no pre-diagnosis was provided, and the paper concludes that SLMs with RAG can be cost-effective clinical decision support tools with expert oversight.
Significance. The question addressed is timely and practically important: small, locally deployable models with retrieval could democratize clinical decision support if their performance rivals frontier models. The paper also quantifies factual alignment with RAGAS and positions the results in a resource-limited care context. However, the significance of the current results is heavily conditional. The reported comparison is confounded by the overlap between the retrieval corpus and the guideline-based reference standard, the top scores in Table 1 are selected across four conditions, and the n=10 sample without confidence intervals or significance tests cannot support a general ranking. These issues are load-bearing, not cosmetic.
major comments (5)
- [Methods: RAG Implementation and Evaluation Procedure; Results] The RAG system indexes 'relevant medical guidelines (e.g., EULAR, ACR, SIGN)' while the ground truth is 'guideline-based ground truths.' This is an answer-key leakage risk: the documents that define the correct answer are the same documents injected into the context for RAG conditions, so a model that excerpts retrieved guideline text is rewarded on treatment and, to a lesser extent, diagnosis scores regardless of its clinical reasoning. The paper's attribution of Mixtral's advantage to 'the effectiveness of its Mixture-of-Experts architecture in leveraging external knowledge' is therefore untestable in this design. The Limitations section acknowledges the small case count but does not acknowledge this overlap between the retrieval corpus and the reference standard.
- [Table 1; Results] Table 1 reports the highest score 'across test conditions' in each category, not a single controlled comparison. For example, Mixtral's diagnostic F1 of 72% comes from the RAG/no-pre-diagnosis condition while Nemotron's 71% comes from the no-RAG/no-pre-diagnosis condition. With only ten cases and no confidence intervals, significance tests, or effect sizes, 1-2 percentage point differences cannot support the claim that SLMs with RAG outperform larger models. The abstract's general statement that SLMs with RAG 'achieve higher diagnostic and therapeutic performance than larger models' is not supported by the reported evidence.
- [Methods: Evaluation Procedure; Author contributions] The F1 computation over 'guideline-based ground truths' is not described in sufficient detail. The two expert raters' judgments are mentioned only in the author contributions, and there is no inter-rater reliability statistic, no pooling or adjudication rule, and no example of scored outputs. Since the conclusions hinge on small F1 differences, the scoring procedure must be documented and its reliability estimated; otherwise the numerical rankings may reflect scoring noise rather than model performance.
- [Data availability; Code availability] The case texts, exact prompts, model versions, decoding hyperparameters, and scoring code are not released; the paper states only that data and code are available 'upon reasonable request.' A benchmarking claim with n=10 is not auditable without these materials. The absence of released case texts and prompts makes it impossible to reproduce the reported F1 and RAGAS scores or to assess whether the ten cases are representative.
- [Abstract; Conclusions] The paper's own results show strong no-RAG performance from Nemotron (diagnostic F1 71%) and Qwen-Turbo (treatment F1 72%), substantial model-dependent variability, and Claude underperforming in both categories. The conclusion should therefore be limited to the narrow observation that, on this ten-case set and under one selected configuration, Mixtral with RAG reached the top F1 scores. The broader comparative claim that SLMs with RAG outperform larger models is not established by the current design and should be withdrawn or substantially reworked.
minor comments (5)
- [Figure 1] The caption 'Types of sequence alignments' does not match the bar chart of RAGAS scores; it appears to be a leftover from another manuscript and should be corrected.
- [References 14, 19, 25, 26] Several citations are mismatched: the F1 definition cites a rheumatology review (reference 26) rather than a statistics or evaluation source, the RAGAS definition cites a radiology article (reference 19) rather than the RAGAS paper (reference 25), and reference 14 appears to be a CRC Press book with an ISBN that does not match the cited NeurIPS paper on Mixture-of-Experts with Expert Choice Routing.
- [Table 1] The 'Model Size' column labels Nemotron and Qwen-Turbo as SLMs but does not state their parameter counts, and the <100B parameter threshold is applied inconsistently; Qwen-Turbo 2.5 is not obviously below 100B parameters, so the distinction between SLM and LLM should be clarified or removed.
- [Methods: Models Tested] The paper does not report the exact model versions, API sampling parameters, temperature settings, or date of runs, all of which can substantially affect clinical text generation and should be documented for reproducibility.
- [Conclusions] The phrase 'no model consistently reached specialist-level accuracy' is ambiguous because no specialist-level threshold was defined and no direct comparison with clinicians was performed; the authors should either define a quantitative threshold or rephrase the claim.
Circularity Check
RAG advantage is partially built into the benchmark: the retrieved guideline corpus is the same source that defines the guideline-based ground truth.
-
other
[Methods: Retrieval-Augmented Generation (RAG) Implementation; Methods: Evaluation Procedure; Limitations]
"The RAG system involved indexing relevant medical guidelines (e.g., EULAR, ACR, SIGN) using a FAISS (IndexFlatL2) vector store. ... Retrieved passages were re-ranked using FlashRank before being provided as context to the LLMs. Model outputs (diagnosis and treatment recommendations) were evaluated against guideline-based ground truths. ... All results were benchmarked against evidence-based guideline recommendations."
The documents fed into the RAG index (EULAR, ACR, SIGN guidelines) are the same body of recommendations from which the evaluation's 'guideline-based ground truths' are derived. Under the RAG condition, models receive retrieved passages from the scoring rubric in their context; under the no-RAG condition, they do not. Any model that extracts and echoes guideline text is therefore rewarded with higher F1 for diagnosis and treatment, independent of its clinical reasoning. The paper's attribution that Mixtral-8x7b-32768's top scores reflect 'the effectiveness of its Mixture-of-Experts architecture in leveraging external knowledge' is untestable while the external knowledge contains the answer source.
full rationale
The paper's central empirical claim is that SLMs with RAG outperform larger models on ten rheumatology cases. The specific reduction is visible in the Methods: the same EULAR/ACR/SIGN guidelines indexed for retrieval are the basis of the 'guideline-based ground truths' used for F1 scoring. Thus part of the measured RAG benefit is constructed: the correct answer text is placed in the model's context. This is not a case of self-citation or imported uniqueness; the references to MoE routing, RAGAS, and FAISS are external and not load-bearing. The presence of no-RAG conditions and external evaluation metrics limits but does not remove the circularity. Because the benchmark's reference standard overlaps with the RAG input corpus, the headline ranking of Mixtral-8x7b-32768+RAG is at least partially an artifact of answer-key leakage rather than independent clinical competence.
Assumptions & free parameters
free parameters (1)
- Prompt and decoding hyperparameters
assumptions (4)
- domain assumption Guideline-based ground truth diagnoses and treatment plans are correct and complete for all ten cases.
- domain assumption F1 score computed from expert judgments is a valid measure of diagnostic and therapeutic decision support quality.
- domain assumption RAGAS score faithfully reflects factual alignment and relevance of model responses.
- domain assumption The ten anonymized cases are representative of the rheumatology caseload at a tertiary German clinic.
Cite this review
Pith. "Pith review of Performance and Practical Considerations of Large and Small Language Models in Clinical Decision Support in Rheumatology." pith.science (2026). https://pith.science/paper/FJFVDQX6
@misc{pith2026250707983,
author = {Pith},
title = {Pith review of: Performance and Practical Considerations of Large and Small Language Models in Clinical Decision Support in Rheumatology},
year = {2026},
howpublished = {\url{https://pith.science/paper/FJFVDQX6}},
note = {Machine review of arXiv:2507.07983}
}
read the original abstract
Large language models (LLMs) show promise for supporting clinical decision-making in complex fields such as rheumatology. Our evaluation shows that smaller language models (SLMs), combined with retrieval-augmented generation (RAG), achieve higher diagnostic and therapeutic performance than larger models, while requiring substantially less energy and enabling cost-efficient, local deployment. These features are attractive for resource-limited healthcare. However, expert oversight remains essential, as no model consistently reached specialist-level accuracy in rheumatology.
Reference graph
Works this paper leans on
-
[1]
James Thirunavukarasu, A. et al. Large language models in medicine: current potential and opportunities for development. Nat. Med. 8, 1930–1940 (2023)
work page 2023
-
[2]
Lee, P., Bubeck, S. & Petro, J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N. Engl. J. Med. 388, 1233–1239 (2023). DOI: 10.1056/NEJMsr2214184
-
[4]
Krusche, M., Callhoff, J., Knitza, J. & Ruffer, N. Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol. Int. 44, 303–306 (2024)
work page 2024
-
[5]
Holzer, M. T. et al. Artificial intelligence in rheumatology: status quo and quo vadis—results of a national survey among German rheumatologists. Ther. Adv. Musculoskelet. Dis. 16, 1759720X241275818 (2024). DOI: 10.1177/1759720X241275818
- [6]
-
[7]
Benavent, D. & Madrid-García, A. Large language models and rheumatology: are we there yet? Rheumatol. Adv. Pract. 9, rkae119 (2024). DOI: 10.1093/rap/rkae119
-
[8]
Yang, R. et al. Retrieval-augmented generation for generative artificial intelligence in medicine. arXiv preprint arXiv:2406.12449 (2024)
arXiv 2024
-
[9]
Griewing, S., Lechner, F., Gremke, N. et al. Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach. J. Cancer Res. Clin. Oncol. 150, 451 (2024). DOI: 10.1007/s00432-024-05964-3
Show all 26 references
-
[10]
Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Adv. Neural Inf. Process. Syst. 33, 9459–9474 (2020)
2020
-
[11]
& Ranjan, R
Gupta, S. & Ranjan, R. A comprehensive survey of retrieval-augmented generation (RAG): Evolution, current landscape and future directions. arXiv preprint arXiv:2410.12837 (2024)
2024 arXiv
-
[12]
Masanneck, L., Meuth, S. G. & Pawlitzki, M. Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurology. npj Digit. Med. 8, 137 (2025). DOI: 10.1038/s41746-025-01536-y
2025 doi
-
[13]
& Angelopoulos, A
Chiang, W.-L. & Angelopoulos, A. LMSYS Chatbot Arena: An open platform for large language model benchmarking. LMSYS Chatbot Arena https://chat.lmsys.org (2023)
2023
-
[14]
Zhou, Y . et al. Mixture-of-Experts with Expert Choice Routing. CRC Press (2022). ISBN: 978-1-4398-3005-5
2022
-
[15]
& Petzold, L
Wang, Y ., Zhao, Y . & Petzold, L. Are large language models ready for healthcare? A comparative study on clinical language understanding. arXiv preprint arXiv:2304.05368 (2023)
2023 arXiv
-
[16]
Gilson, A. et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9, e45312 (2023). DOI: 10.2196/45312
2023 doi
-
[17]
Goh, E. et al. Large Language Model Influence on Diagnostic Reasoning. JAMA Netw. Open 7, e2440969 (2024)
2024
-
[18]
& Bastardot, F
Roustan, D. & Bastardot, F. The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on Hallucinations. Interact. J. Med. Res. 14, e59823 (2025)
2025
-
[19]
Weinert, D. A. & Rauschecker, A. M. Enhancing large language models with retrieval-augmented generation: a radiology-specific approach. Radiol. Artif. Intell. 7, e240313 (2025). DOI: 10.1148/ryai.240313
2025 doi
-
[20]
& Knitza, J
Lechner, F., Kuhn, S. & Knitza, J. Harnessing large language models for rheumatic disease diagnosis: advancing hybrid care and task shifting. Int. J. Rheum. Dis. 28, e70124 (2025). DOI: 10.1111/1756-185X.70124
2025 doi
-
[21]
E., Mankowitz, T
Weissman, G. E., Mankowitz, T. & Kanter, G. P. Unregulated large language models produce medical device-like output. npj Digit. Med. 8, 148 (2025). DOI: 10.1038/s41746-025-01544-y
2025 doi
-
[22]
Knitza, J. et al. Machine learning-based improvement of an online rheumatology referral and triage system. Front. Med. (Lausanne) 9, 963348 (2022). DOI: 10.3389/fmed.2022.963348
2022
-
[23]
Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
2022 arXiv
-
[24]
Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)
2022 arXiv
-
[25]
& Schockaert, S
Es, S., James, J., Espinosa-Anke, L. & Schockaert, S. RAGAS: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023)
2023 arXiv
-
[26]
& Rezaei, N
Momtazmanesh, S., Nowroozi, A. & Rezaei, N. Artificial intelligence in rheumatoid arthritis: current status and future perspectives: a state-of-the-art review. Rheumatol. Ther. 9, 1249–1304 (2022). Acknowledgments N/A Data Availability Statement Data is available upon reasonab...
2022
-
[2024]
Rheumatol
Z. Rheumatol. 83, 249–284 (2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.