REVIEW 4 major objections 6 minor 1 cited by
Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Compact quantized language models can already produce clinically meaningful answers to open-ended medical cases on iPhones and iPads, and the main limit to wider deployment is device memory, not processing power.
desk verdict A genuinely useful first map of what runs on Apple devices for clinical QA, but the accuracy rankings rest on unvalidated protocol modifications and should be read as suggestive, not load-bearing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a mobile benchmarking harness rather than a single mathematical object: a custom iOS app built around the MLX framework, an array-based inference stack for Apple silicon, runs 4-bit-quantized models entirely on-device, clears the GPU cache after every question, and logs tokens per second, time to first token, memory use, thermal state, and battery level. The scored output comes from the AMEGA benchmark, whose 20 open-ended clinical cases are rated by a GPT-4o judge under a modified protocol: temperature 0.1 instead of 0, the follow-up reask step removed, and reasoning traces stripped before evaluation. The quantized MLX conversion is what lets 8-billion-parameter models fit into the 6 to 8 GB of memory on recent iPhones, while each device's iOS memory budget determines which models can run at all.
What would settle it
Run the same 13 quantized models through the original, unmodified benchmark procedure—temperature 0.0, follow-up reask enabled, reasoning text retained—and check whether the accuracy ranking changes materially; if Med42 or Aloe no longer lead, or Phi-3 Mini drops well below Llama 3.1 8B, the headline accuracy results are an artifact of the modified judging setup. A blind human clinician rating of the same responses would independently settle whether the GPT-4o scores reflect clinical quality.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a measured feasibility result: after 4-bit quantization and conversion to the MLX format, open-weight language models from 1 to 8 billion parameters can run on unmodified iPhones and iPads released between 2020 and 2024 and can answer AMEGA clinical cases at 8 to 120 tokens per second. Accuracy is led by the medical fine-tunes Aloe 8B (490.9) and Med42 8B (490.0), with the compact general-purpose Phi-3 Mini 3.8B close behind at 464.6 and generally faster than larger models. Medical fine-tuning is not a guarantee of quality: the Bio-Medical family scored well below the untuned Llama 3.1 8B model, and the reasoning-heavy DeepSeek R1 models were both the slowest and less accurate than their non-reasoning counterparts. On older devices, throughput remains usable for interactive conversation, and where a model fails to run the cause is iOS's per-app memory ceiling rather than chip speed—for example, a model needing 2.24 GB was terminated on a 4 GB iPhone with a 2.25 GB app limit.
Load-bearing premise
The accuracy ranking rests on the assumption that GPT-4o's scores under the modified testing conditions—temperature 0.1, no follow-up reask, stripped reasoning text, and compressed 4-bit models—measure clinical reasoning fairly, without hurting some models more than others.
Editorial extensions
If this is right
- Privacy-preserving clinical question answering is already possible on current iPhones and iPads, without sending patient data to a cloud service.
- A model's memory footprint, not its parameter count or the device's processor, decides whether it can run, so future devices with more RAM will widen the set of usable medical models more than faster chips will.
- Medical fine-tuning labels do not predict benchmark performance; the untuned Llama 3.1 8B beat several medically fine-tuned models, so real deployments need per-model validation on realistic clinical cases.
- Throughput of 8 to 30 tokens per second on everyday devices exceeds average reading speed, making interactive on-device use practical in clinical settings.
- Reasoning-style models such as DeepSeek R1 are slower and not more accurate on this benchmark, suggesting that long chain-of-thought outputs are a poor use of limited edge compute.
Reading between the lines
- If the modified GPT-4o scoring is fair, the best on-device 8B models score about 500 AMEGA points, roughly 70 percent of the score reported for a much larger cloud model under the original protocol—enough to suggest triage-level support, though not equivalence.
- Removing the reask step may systematically favor smaller models, since reasking prolongs outputs and can trigger hallucinations; re-running with reask enabled could change the ranking even if the feasibility result stands.
- The thermal measurements—throughput drops of 18 to 55 percent in the serious thermal state—imply that real deployments need thermal-aware scheduling or duty cycling; testing whether short bursts preserve both speed and accuracy would be a natural extension.
- Because all devices are from one vendor's ecosystem, the memory-limit conclusion may not transfer to other operating systems with different memory management; repeating the same 13-model benchmark on another mobile runtime would test that boundary.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript benchmarks 13 open-weight LLMs running on-device via Apple's MLX framework across six iOS devices, scoring clinical-reasoning performance with the AMEGA benchmark. The authors report that medical models Med42 and Aloe achieve the highest AMEGA scores, that Phi-3 Mini offers a strong accuracy/speed trade-off, and that memory, rather than raw compute, is the main deployment constraint. The paper also logs throughput, thermal state, and memory behavior across devices.
Significance. If the accuracy ranking were validated, this paper would provide a practically useful map for deploying clinical LLMs on consumer Apple hardware, with genuine value in the open-source HealthBench toolchain and the realistic real-device measurements of throughput, thermal state, and memory limits. The feasibility findings—older devices can run smaller models, memory is the binding constraint—rest on direct measurements and are reasonably solid. The accuracy ranking, however, rests on an unvalidated modification of the AMEGA protocol and on single-run scores without variance reporting, so the headline model-comparison claims are not yet trustworthy. For a clinical AI venue, the benchmark rigor must be strengthened before the accuracy claims can be accepted.
major comments (4)
- [§2.2, §2.5, Table 3] The central accuracy ranking is based on AMEGA scores collected under a protocol that differs from the published benchmark in four consequential ways: generation temperature is 0.1 instead of 0.0 (§2.2), the reask process is removed (§2.1), <think> blocks are stripped from reasoning models (§2.5), and all converted models are quantized to 4-bit (§2.3). The authors themselves state that temperature 'can significantly influence output quality' (§4.3) and that the reask exclusion 'require[s] further validation' (§2.1). No control sub-study is reported that quantifies the effect of these deviations on the scores. Consequently, the reported ordering—for instance, the 25.9-point gap between Med42/Aloe and Phi-3 Mini—cannot be attributed to clinical reasoning ability rather than to interactions between the modified protocol and model family. This is the load-bearing assumption for the abstract and §4.1 claims.
- [§3.1, Table 3] The accuracy scores are reported only as means across devices, with no standard deviation, confidence interval, or per-device breakdown. Since each model-device combination appears to have been run once and the generation temperature is 0.1, sampling variability is entirely uncharacterized; the ordering of models with scores within roughly 25 points may be within noise. Additionally, models do not run on the same set of devices (e.g., Med42 runs only on devices with at least 8 GB, while Phi-3 Mini also runs on smaller devices), so the means are not directly comparable. The authors should report per-device scores and variability, ideally with repeated runs or a statistical comparison on a matched device subset.
- [§2.3, Table 1] The quantization is applied unevenly: DeepSeek R1 1.5B is kept at 8-bit while all other converted models use 4-bit. This breaks comparability of that model's score (179.8) with Llama 3.2 1B (256.5), and no ablation is provided to measure the accuracy impact of 4-bit quantization for any model. Section 4.3 acknowledges that the conversion 'likely led to a reduction in model performance,' but the direction and magnitude of that effect are not quantified. Without such an ablation, the cross-model accuracy comparisons in Table 3 cannot be separated from quantization-induced artifacts.
- [§2.5, §4.3] The GPT-4o evaluator is used without a human-verification substudy. Although the original AMEGA benchmark relies on the same judge, the responses judged here come from quantized, on-device models with modified generation parameters, and the judge's behavior on this distribution is uncalibrated. The paper's limitation section concedes that a human evaluation 'could have identified additional issues or resulted in different answer ratings' (§4.3). A small human-rated sample of responses would provide a direct check on whether the judge biases particular model families, which is especially important given the removal of the reask process from the original evaluation pipeline.
minor comments (6)
- [§1 vs. §2.1] The Introduction states that AMEGA contains '20 clinical cases across 13 medical specialties,' while §2.1 says the cases cover '20 medical specialties'; these numbers should be reconciled.
- [§3.1 vs. Table 3] The model is referred to as 'MedLlama3-v20' in the text but as 'MedLlama 8B' in Table 3; a single consistent identifier should be used throughout.
- [§2.2] The sentence beginning 'A cool-down phase is initiated before inference for each questin begins' contains a typo ('questin' for 'question') and should be rewritten for clarity.
- [Table 1 vs. Table 2] Table 1 reports 'Memory Usage (GB)' while Table 2 uses 'Memory (GiB)' and 'Memory Limit (GiB)'; the units should be made consistent and explicitly defined.
- [Figure 1b] The caption for Figure 1b states that the minimum value 'reflects the average T/s on the slowest device'; it should clarify that this is the mean over questions for that device, not the minimum instantaneous throughput.
- [References] Reference [19] spells 'Microsoft' as 'Mircosoft'; other minor citation formatting issues (e.g., inconsistent URL line breaks in [1]) could be cleaned up.
Circularity Check
No circularity: accuracy claims are measured against an external benchmark (AMEGA) with an external judge (GPT-4o), and no fitted parameter or self-citation chain forces the reported ranking.
full rationale
The paper's central claims are empirical benchmarking results rather than derived predictions. Model scores in Table 3 are obtained by running publicly available, pre-trained models on mobile devices and having GPT-4o evaluate the generated responses with the published AMEGA rubric. No parameter is fit to the reported AMEGA scores, and no model is trained or tuned in this paper. The protocol modifications (temperature raised to 0.1, reask removed, think blocks stripped, 4-bit MLX quantization) are genuine validity threats, and the authors candidly acknowledge this in Section 4.3: 'These parameter modifications can significantly influence output quality, potentially affecting both accuracy and inference speed.' However, a validity threat is not circularity: the reported measurements are not equivalent to the evaluation inputs by construction. There is also no load-bearing self-citation. The HealthBench application and Stanford Spezi LLM module are described as open-source implementation artifacts used to collect the measurements, not as authority for the accuracy findings, and the benchmark itself is attributed to Fast et al. (an external group). The memorization concern in Section 4.1 is a contamination hypothesis, not a circular derivation, and does not make the benchmark scores an input to themselves. Overall, the accuracy, speed, and feasibility conclusions are self-contained empirical observations; the paper's main risks are validity and generalization, not circularity.
Assumptions & free parameters
free parameters (3)
- Inference temperature =
0.1
- Output token limit =
2048 standard, 4096 reasoning
- MLX quantization bits =
4-bit for all models except DeepSeek R1 1.5B at 8-bit
assumptions (4)
- domain assumption AMEGA benchmark scores measure clinical reasoning quality
- domain assumption GPT-4o evaluates model answers without bias
- domain assumption MLX framework and unified memory fairly represent on-device LLM performance
- domain assumption Single run per model-device pair is representative
Cite this review
Pith. "Pith review of Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning." pith.science (2026). https://pith.science/paper/XVOKXQKO
@misc{pith2026250208954,
author = {Pith},
title = {Pith review of: Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/XVOKXQKO}},
note = {Machine review of arXiv:2502.08954}
}
read the original abstract
The deployment of Large Language Models (LLM) on mobile devices offers significant potential for medical applications, enhancing privacy, security, and cost-efficiency by eliminating reliance on cloud-based services and keeping sensitive health data local. However, the performance and accuracy of on-device LLMs in real-world medical contexts remain underexplored. In this study, we benchmark publicly available on-device LLMs using the AMEGA dataset, evaluating accuracy, computational efficiency, and thermal limitation across various mobile devices. Our results indicate that compact general-purpose models like Phi-3 Mini achieve a strong balance between speed and accuracy, while medically fine-tuned models such as Med42 and Aloe attain the highest accuracy. Notably, deploying LLMs on older devices remains feasible, with memory constraints posing a greater challenge than raw processing power. Our study underscores the potential of on-device LLMs for healthcare while emphasizing the need for more efficient inference and models tailored to real-world clinical reasoning.
Figures
Forward citations
Cited by 1 Pith paper
-
Preserving Privacy, Increasing Accessibility, and Reducing Cost: An On-Device Artificial Intelligence Model for Medical Transcription and Note Generation
Fine-tuning a 1B Llama model on synthetic endocrinology data improves structured medical note generation and substantially reduces LLM-judged hallucinations and omissions in a browser-based, on-device deployment.
Reference graph
Works this paper leans on
-
[1]
2023, year of open LLMs, December 2023
Clémentine Fourrier. 2023, year of open LLMs, December 2023. URL https:// huggingface.co/blog/2023-in-llms
work page 2023
-
[2]
Nobel Dhar, Bobin Deng, Dan Lo, Xiaofeng Wu, Liang Zhao, and Kun Suo. An Empirical Analysis and Resource Footprint Study of Deploying Large Language Models on Edge Devices. In Proceedings of the 2024 ACM Southeast Conference, ACMSE ’24, pages 69–76, New York, NY, USA, April 2024. Association for Com- puting Machinery. ISBN 9798400702372. doi: 10.1145/3603...
-
[3]
PrivateLoRA For Effi- cient Privacy Preserving LLM, November 2023
Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. PrivateLoRA For Effi- cient Privacy Preserving LLM, November 2023. URLhttp://arxiv.org/abs/2311. 14030. arXiv:2311.14030 [cs]
arXiv 2023
-
[4]
Lawrence O. Gostin, Sam F. Halabi, and Kumanan Wilson. Health Data and Privacy in the Digital Era.JAMA, 320(3):233–234, July 2018. ISSN 0098-7484. doi: 10.1001/ jama.2018.8374. URL https://doi.org/10.1001/jama.2018.8374
-
[5]
How Much Energy Do LLMs Consume? Unveil- ing the Power Behind AI, July 2024
Sourabh Mehta. How Much Energy Do LLMs Consume? Unveil- ing the Power Behind AI, July 2024. URL https://adasci.org/ how-much-energy-do-llms-consume-unveiling-the-power-behind-ai/
work page 2024
-
[6]
It’s time to bench the medical exam benchmark
Inioluwa Deborah Raji, Roxana Daneshjou, and Emily Alsentzer. It’s time to bench the medical exam benchmark. NEJM AI, 2(2):AIe2401235, January 2025. doi: 10.1056/AIe2401235. URL https://ai.nejm.org/doi/full/10.1056/AIe2401235. Publisher: Massachusetts Medical Society
-
[7]
Desai, Kueiyu Joshua Lin, and Jie Yang
Bowen Gu, Rishi J. Desai, Kueiyu Joshua Lin, and Jie Yang. Probabilistic medical predictions of large language models.npj Digital Medicine, 7(1):1–9, December 2024. ISSN 2398-6352. doi: 10.1038/s41746-024-01366-4. URLhttps://www.nature.com/ articles/s41746-024-01366-4. Publisher: Nature Publishing Group
-
[8]
Dennis Fast, Lisa C. Adams, Felix Busch, Conor Fallon, Marc Huppertz, Robert Siepmann, Philipp Prucker, Nadine Bayerl, Daniel Truhn, Marcus Makowski, 16 Alexander Löser, and Keno K. Bressem. Autonomous medical evaluation for guideline adherence of large language models. npj Digital Medicine, 7, 2024. doi: 10.1038/s41746-024-01356-6. URL https://www.nature...
Show all 30 references
-
[9]
Smoothquant: Accurate and efficient post-training quantization for large language models, 2024
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https://arxiv.org/abs/2211.10438
2024 arXiv
-
[10]
MLX: Efficient and flexible machine learning on apple silicon, 2023
Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on apple silicon, 2023. URLhttps://github. com/ml-explore
2023
-
[11]
ImportingPhan- toms: Measuring LLM Package Hallucination Vulnerabilities, January 2025
ArjunKrishna, ErickGalinkin, LeonDerczynski, andJeffreyMartin. ImportingPhan- toms: Measuring LLM Package Hallucination Vulnerabilities, January 2025. URL http://arxiv.org/abs/2501.19012. arXiv:2501.19012 [cs] version: 1
2025 arXiv
-
[12]
Your battery status is being used to track you online.The Guardian, August 2016
Alex Hern. Your battery status is being used to track you online.The Guardian, August 2016. ISSN 0261-3077. URL https://www.theguardian.com/technology/ 2016/aug/02/battery-status-indicators-tracking-online
2016
-
[13]
Domain-specialized llm: Financial fine-tuning and utilization method using mistral 7b
Cheonsu Jeong. Domain-specialized llm: Financial fine-tuning and utilization method using mistral 7b. Journal of Intelligence and Information Systems, 30(1):93–120, March 2024. ISSN 2288-4882. doi: 10.13088/jiis.2024.30.1.093. URL http://dx. doi.org/10.13088/jiis.2024.30.1.093
2024 doi
-
[14]
Bio-medical-llama-3-2-1b-cot-012025: A reasoning-enhanced biomed- ical language model, 2025
ContactDoctor. Bio-medical-llama-3-2-1b-cot-012025: A reasoning-enhanced biomed- ical language model, 2025. URL https://huggingface.co/ContactDoctor/ Bio-Medical-Llama-3-2-1B-CoT-012025
2025
-
[15]
Llama-3.2-1b-instruct, 2024
Meta Llama. Llama-3.2-1b-instruct, 2024. URL https://huggingface.co/ meta-llama/Llama-3.2-1B-Instruct
2024
-
[16]
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...
2025 arXiv
-
[17]
Bio-medical-3b-cot-012025: A high-performance biomedical lan- guage model with reasoning capabilities, 2025
ContactDoctor. Bio-medical-3b-cot-012025: A high-performance biomedical lan- guage model with reasoning capabilities, 2025. URL https://huggingface.co/ ContactDoctor/Bio-Medical-3B-CoT-012025
2025
-
[18]
Llama-3.2-3b-instruct, 2024
Meta Llama. Llama-3.2-3b-instruct, 2024. URL https://huggingface.co/ meta-llama/Llama-3.2-3B-Instruct
2024
-
[19]
Phi-3-mini-4k-instruct, 2024
Mircosoft. Phi-3-mini-4k-instruct, 2024. URL https://huggingface.co/ microsoft/Phi-3-mini-4k-instruct
2024
-
[20]
Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...
2024 arXiv
-
[21]
Bio-medical-llama-3-8b: A high-performance biomedical language model, 2024
ContactDoctor. Bio-medical-llama-3-8b: A high-performance biomedical language model, 2024. URL https://huggingface.co/ContactDoctor/ Bio-Medical-Llama-3-8B
2024
-
[22]
Aloe: A family of fine-tuned open healthcare llms, 2024
Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Jordi Bayarri-Planas, Adrian Tor- mos, Daniel Hinjos, Pablo Bernabeu-Perez, Anna Arias-Duart, Pablo Agustin Martin- Torres, Lucia Urcelay-Ganzabal, Marta Gonzalez-Mallo, Sergio Alvarez-Napagao, Ed- uard Ayguadé-Parra, and Ulises Cor...
2024
-
[23]
medllama3-v20, 2024
YonseiMAILAB. medllama3-v20, 2024. URL https://huggingface.co/ ProbeMedicalYonseiMAILab/medllama3-v20
2024
-
[24]
Med42-v2: A suite of clinical llms, 2024
Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. Med42-v2: A suite of clinical llms, 2024. URL https: //arxiv.org/abs/2408.06142
2024 arXiv
-
[25]
Llama-3.1-8b-instruct, 2024
Meta Llama. Llama-3.1-8b-instruct, 2024. URL https://huggingface.co/ meta-llama/Llama-3.1-8B-Instruct
2024
-
[26]
Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar
Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient Large Language Model Inference with Limited Memory, July 2024. URL http://arxiv.org/abs/2312.11514. arXiv:23...
2024 arXiv
-
[27]
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023. URLhttp://arxiv.org/abs/2201.11903. arXiv:2201.11903 [cs]
2023 arXiv
-
[28]
Bouw, and Stephen Cobb
Tolga Çöplü, Marc Loedi, Arto Bendiken, Mykhailo Makohin, Joshua J. Bouw, and Stephen Cobb. A Performance Evaluation of a Quantized Large Language Model on Various Smartphones, December 2023. URL http://arxiv.org/abs/2312.12472. arXiv:2312.12472 [cs]
2023 arXiv
-
[29]
Hl7 fhir: An agile and restful approach to healthcare information exchange
Duane Bender and Kamran Sartipi. Hl7 fhir: An agile and restful approach to healthcare information exchange. In Proceedings of the 26th IEEE Interna- tional Symposium on Computer-Based Medical Systems, pages 326–331, 2013. doi: 10.1109/CBMS.2013.6627810
2013
-
[30]
How many words do we read per minute? a review and meta- analysis of reading rate.Journal of Memory and Language, 109, 2019
Marc Brysbaert. How many words do we read per minute? a review and meta- analysis of reading rate.Journal of Memory and Language, 109, 2019. doi: 10.1016/ j.jml.2019.104047. URL https://www.sciencedirect.com/science/article/abs/ pii/S0749596X19300786#preview-section-cited-by. 19
2019
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.