REVIEW 4 major objections 5 minor 27 references
Bridging Language Barriers in Healthcare: A Study on Arabic LLMs
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Arabic medical AI needs task-specific language mixes; fine-tuning alone may not close the gap.
desk verdict Useful Arabic medical QA case study whose central claim about task-dependent language ratios is confounded by dataset composition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the bilingual fine-tuning grid: Llama 3.1 8B and 70B models trained for two epochs on a fixed 469.97M-token budget, with Arabic-English ratios 100/0, 80/20, 60/40, 50/50, 40/60, 20/80, and 0/100, assembled from native Arabic data (AHQAD, CIDAR), Qwen-translated Med42 and synthetic QA data, and English Med42 data. Evaluation runs on Arabic translations of PubMedQA, MedMCQA, MedQA, and MMLU using a Harness pipeline modified for right-to-left text and for scoring the full answer text rather than a single choice label. A second mechanism is the translation-comparison pipeline, which tests whether routing Arabic through English via various translators closes the gap; it does not.
What would settle it
Take a clinician-translated, back-translated Arabic version of PubMedQA and MedQA, then rerun the ratio grid; if the best ratios change or Arabic no longer beats English, the original task-dependent optima were translation artifacts.
Extended reading notes
Core claim
The paper's claim is that simply translating medical data does not guarantee strong target-language performance: the effectiveness of bilingual fine-tuning depends on the task's reasoning demands. In the paper's experiments, Llama 3.1 8B reaches its best Arabic accuracy with 100% Arabic data on PubMedQA (71.2), 80% Arabic on MedMCQA (35.1), 40% Arabic on MedQA (29.8), and 0% Arabic on MMLU (42.4), and these ordering patterns reproduce across base and instruct variants. For the 70B models, Arabic-only training is consistently best on Arabic tasks, while fine-tuned instruct models rarely surpass their already-aligned baselines. The paper concludes that data- and compute-intensive pretraining may still be necessary to reach native-level Arabic clinical performance.
Load-bearing premise
The findings depend on Arabic translations of English benchmarks being medically faithful and free of leakage; the paper did not independently validate the translated answer options, and some Arabic-above-English scores suggest the translations may be easier than the originals.
Editorial extensions
If this is right
- Deploying Arabic medical QA systems requires per-task data-curation strategies; a single best Arabic mix does not exist.
- For knowledge-heavy multiple-choice benchmarks like MMLU, English-only fine-tuning can improve Arabic scores, so language mixing is not always the lever that matters.
- Larger models respond more uniformly to language mixing, suggesting scale changes how language knowledge is acquired from fine-tuning data.
- Because fine-tuned instruct models rarely surpass their baselines, model developers should budget for continued pretraining when adding a low-resource language.
- Translation pipelines remain a partial remedy: general-purpose LLMs translate medical text better than specialized translation models, but still underperform native proficiency and add computational cost.
Reading between the lines
- Inference: The odd finding that several models score higher on Arabic PubMedQA than on English PubMedQA implies the Arabic translations may be easier or contain answer leakage; if so, the task-dependent optimum ratios partly measure translation properties, not clinical reasoning.
- Inference: The pattern suggests a two-stage recipe worth testing elsewhere: continued pretraining on target-language domain text to establish the language, followed by task-specific ratio tuning only as a fine adjustment.
- Inference: The same ratio-grid experiment could be repeated for other non-Latin-script languages, such as Hindi, Amharic, or Urdu, to test whether the task-dependence generalizes beyond Arabic's morphology.
- Inference: The full-answer scoring modification likely changes absolute accuracies; re-running the grid with choice-only scoring would reveal how much of the reported task-dependent ordering is an artifact of the scoring method.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates Arabic medical QA performance of several open-source LLMs on Arabic translations of PubMedQA, MedMCQA, MedQA, and Medical MMLU. It then studies a translation pipeline for Arabic-to-English processing and fine-tunes Llama 3.1 8B and 70B models on different Arabic-English data mixtures, reporting that the optimal mixture varies across tasks. The authors conclude that fine-tuning alone may be insufficient for adding a new language and that pretraining may be necessary.
Significance. If the empirical claims hold, the paper would provide useful evidence on cross-lingual medical QA and on practical data-mixture choices for Arabic medical LLMs. Its strengths include a broad zero-shot comparison across model families, an explicit comparison of translation-based adaptation against direct fine-tuning, and a relatively detailed description of the fine-tuning data pipeline. However, the central claim about task-dependent optimal language ratios currently rests on experiments in which language ratio is confounded with dataset composition, and the Arabic evaluation sets are not validated for translation quality or leakage; these issues are load-bearing for the main conclusions. The paper also reports no uncertainty quantification, so the observed differences may not be statistically reliable.
major comments (4)
- [Table 4 and 'Language Specific Finetuning'] The central claim that the optimal Arabic-English mixture is task-dependent is not identifiable from the fine-tuning experiments as designed. Table 3 shows that the Arabic data stream is a mixture of AHQAD, translated Med42, CIDAR, and synthetic open-ended QA, whereas the English stream is exclusively the English Med42 dataset. Consequently, changing the Arabic-English ratio in Table 4 also changes the dataset composition, task format, and source distribution. The pattern that PubMedQA favors 100% Arabic, for instance, could equally reflect the presence of AHQAD and synthetic QA pairs rather than the Arabic language; similarly, MMLU favoring 0% Arabic could reflect Med42's closer format match with the MMLU-style knowledge probe. The authors need a controlled comparison in which the only difference between streams is language, such as translating the same Med42 records into Arabic and mixing those, or a composition-controlled ablation that isolates the language effect.
- [Table 1 and 'Arabic Evaluation Datasets'] The Arabic evaluation datasets need validation before the reported performance gaps and mixture effects can be interpreted. Several models score substantially higher on Arabic PubMedQA than on English PubMedQA, e.g., Qwen2.5-3B-Instruct at 61.2 vs 29.2 and Qwen2.5-7B-Instruct at 74.4 vs 45.2. Such an inversion is surprising for zero-shot evaluation of models with stronger English capabilities and suggests translation artifacts, answer leakage, or an evaluation-pipeline bug. The paper describes the BiMediX semi-automated translation pipeline but reports no human fidelity checks, no post-translation validation, and no checks for leaked answer information. Because all zero-shot, translation-pipeline, and fine-tuning results depend on these Arabic datasets, the authors should provide translation quality statistics, sample-level manual inspection, and leakage analysis.
- [Table 4 and 'Results'] The paper reports no error bars, confidence intervals, or significance tests, yet many of the claimed task-dependent differences are small. For example, Llama 3.1 8B-Baseline Arabic MedMCQA accuracies range from 32.2 to 35.1 across ratios, and several adjacent differences are 1-2 points. Without variance estimates across seeds or repeated evaluation runs, these differences could be noise, and the claim that 'the best Arabic accuracy for MedMCQA comes from 80% Arabic' is not statistically supported. The authors should report standard deviations, paired significance tests, or at least confidence intervals for the fine-tuning runs.
- [Conclusion] The conclusion that 'data and computationally intensive pretraining methods may still be necessary' is not supported by the experiments. The paper only compares different fine-tuning mixtures; no continued-pretraining or pretraining-from-scratch arm is run. The statement is an untested hypothesis, and the conclusion should be rephrased as a motivation for future work rather than as an empirical finding of this study.
minor comments (5)
- [Modifications to Harness Pipeline] The Arabic example in this section is rendered as character escapes such as '/char48/char2e/char51/...' rather than readable Arabic text; this should be replaced with actual Arabic script or a transliteration.
- [Appendix] The 'Qwen Translation Example' in the appendix is also displayed as encoded '/char' sequences, making the example unusable for verifying translation quality; the appendix should show proper Arabic text.
- [Table 4] There are several typographical inconsistencies in Table 4, including '71.62' in the 8B-Baseline English Only row, '2 8.7' in the Strong English Majority row, and '66. 5' in the 70B-Instruct row; these should be corrected.
- [General experimental setup] The paper does not state the exact number of evaluation examples for each benchmark or the evaluation prompt/template used; this information is necessary for reproducibility and for interpreting percentage differences.
- [Fine-tuning pipeline] The fine-tuning section would benefit from reporting effective batch size, optimizer details, wall-clock time, and the choice of checkpoint (last checkpoint vs best on a validation set); the current description is not fully reproducible.
Circularity Check
No load-bearing circularity: the central fine-tuning and translation findings are empirical and self-contained, with only minor self-citations that do not drive the conclusions.
full rationale
I walked the derivation chain and found no step where a result reduces by construction to its own inputs. The zero-shot Arabic evaluations use the externally produced BiMediX translations of PubMedQA, MedMCQA, MedQA, and MMLU, and the models tested include a range of independent open-source checkpoints. The translation-pipeline comparison is a direct empirical measurement of accuracy after translating Arabic prompts to English, with no fitted parameter. The fine-tuning sweep varies the Arabic-English token ratio while holding total tokens constant; the outcome metrics are measured on held-out benchmarks that are not used to define the training mixture. The paper's central claim that the optimal language ratio varies by task is a post-hoc empirical observation from Table 4, not a quantity defined in terms of itself. The main self-citations are to Med42 and MEDIC, from overlapping authors, but Med42 is used only as a source of training data and MEDIC is mentioned as a future benchmark direction; neither is invoked as mathematical proof or as a uniqueness theorem. The confound between language ratio and dataset composition in Table 4 is a genuine identifiability threat and belongs under correctness risk, not circularity, because the paper does not define 'optimal ratio' via the evaluation score. Overall, the paper contains minor self-citation but no load-bearing circular step, so a score of 2 reflects that minor presence without implying that any derivation is circular.
Assumptions & free parameters
free parameters (4)
- AHQAD sample size =
100,000
- Synthetic QA retention threshold =
5
- Med42 sample size =
500,000
- Token budget for fine-tuning =
469.97M
assumptions (4)
- domain assumption The Arabic-translated evaluation datasets are valid, accurate, and measure the same clinical construct as the English originals.
- domain assumption The modified Harness pipeline correctly handles Arabic script and scores full answer text without introducing artifacts.
- ad hoc to paper The Arabic and English fine-tuning streams differ only in language, not in dataset composition.
- domain assumption Single-run accuracy scores without error bars are treated as reliable indicators of model capability.
Cite this review
Pith. "Pith review of Bridging Language Barriers in Healthcare: A Study on Arabic LLMs." pith.science (2026). https://pith.science/paper/RB3XXN2S
@misc{pith2026250109825,
author = {Pith},
title = {Pith review of: Bridging Language Barriers in Healthcare: A Study on Arabic LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/RB3XXN2S}},
note = {Machine review of arXiv:2501.09825}
}
read the original abstract
This paper investigates the challenges of developing large language models (LLMs) proficient in both multilingual understanding and medical knowledge. We demonstrate that simply translating medical data does not guarantee strong performance on clinical tasks in the target language. Our experiments reveal that the optimal language mix in training data varies significantly across different medical tasks. We find that larger models with carefully calibrated language ratios achieve superior performance on native-language clinical tasks. Furthermore, our results suggest that relying solely on fine-tuning may not be the most effective approach for incorporating new language knowledge into LLMs. Instead, data and computationally intensive pretraining methods may still be necessary to achieve optimal performance in multilingual medical settings. These findings provide valuable guidance for building effective and inclusive medical AI systems for diverse linguistic communities.
Reference graph
Works this paper leans on
-
[1]
Alyafeai, Z.; Almubarak, K.; Ashraf, A.; Alnuhait, D.; Alshahrani, S.; Abdulrahman, G. A. Q.; Ahmed, G.; Gawah, Q.; Saleh, Z.; Ghaleb, M.; Ali, Y.; and Al-Shaibani, M. S. 2024. CIDAR: Culturally Relevant Instruction Dataset For Arabic. arXiv:2402.03177
arXiv 2024
-
[2]
Ankit Pal, M. S. 2024. OpenBioLLMs: Advancing Open-Source Large Language Models for Healthcare and Life Sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B
work page 2024
-
[3]
Chen, Z.; Cano, A. H.; Romanou, A.; Bonnet, A.; Matoba, K.; Salvi, F.; Pagliardini, M.; Fan, S.; Köpf, A.; Mohtashami, A.; Sallinen, A.; Sakhaeirad, A.; Swamy, V.; Krawczuk, I.; Bayazit, D.; Marmet, A.; Montariol, S.; Hartley, M.-A.; Jaggi, M.; and Bosselut, A. 2023. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. arXiv preprint arXiv...
arXiv 2023
-
[4]
K.; Raha, T.; Khan, S.; and Pimentel, M
Christophe, C.; Kanithi, P. K.; Raha, T.; Khan, S.; and Pimentel, M. A. 2024. Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142
arXiv 2024
-
[5]
Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
arXiv 2024
-
[6]
Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac'h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2024. A framework for few-shot language model evaluation
2024
-
[7]
Helsinki-NLP. 2024. opus-mt-ar-en. https://huggingface.co/Helsinki-NLP/opus-mt-ar-en
work page 2024
-
[8]
Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR)
2021
Show all 27 references
-
[9]
Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D
Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825
2023 arXiv
-
[10]
Jin, D.; Pan, E.; Oufattole, N.; Weng, W.-H.; Fang, H.; and Szolovits, P. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14): 6421
2021
-
[11]
Jin, Q.; Dhingra, B.; Liu, Z.; Cohen, W.; and Lu, X. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language P...
2019
-
[12]
Jin, Y.; Chandra, M.; Verma, G.; Hu, Y.; De Choudhury, M.; and Kumar, S. 2024. Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. In Proceedings of the ACM Web Conference 2024, WWW '24, 2627–2638. New York, NY, USA: Association ...
2024
-
[13]
K.; Christophe, C.; Pimentel, M
Kanithi, P. K.; Christophe, C.; Pimentel, M. A. F.; Raha, T.; Saadi, N.; Javed, H.; Maslenkova, S.; Hayat, N.; Rajan, R.; and Khan, S. 2024. MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications. arXiv:arXiv:2409.07314
2024 arXiv
-
[14]
Lopez, M.; Parikh, A.; Yacouby, R.; and et al. 2023. GatorTron-M: Multilingual Clinical Language Models for Medical Natural Language Processing
2023
-
[15]
Lu, Y.; Zhu, W.; Li, L.; Qiao, Y.; and Yuan, F. 2024. LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages. arXiv preprint arXiv:2407.05975
2024 arXiv
-
[16]
M.; Carignan, D.; and Horvitz, E
Nori, H.; King, N.; McKinney, S. M.; Carignan, D.; and Horvitz, E. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375
2023 arXiv
-
[17]
K.; and Sankarasubbu, M
Pal, A.; Umapathi, L. K.; and Sankarasubbu, M. 2022. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In Flores, G.; Chen, G. H.; Pollard, T.; Ho, J. C.; and Naumann, T., eds., Proceedings of the Conference on Health, Inference, ...
2022
-
[19]
S.; Khan, F
Pieri, S.; Mullappilly, S. S.; Khan, F. S.; Anwer, R. M.; Khan, S.; Baldwin, T.; and Cholakkal, H. 2024 b . BiMediX: Bilingual Medical Mixture of Experts LLM. arXiv:2402.13253
2024 arXiv
-
[20]
K.; Jia, B.; Katipomu, S.; Li, H.; Koto, F.; Marshall, W.; Gosal, G.; Liu, C.; Chen, Z.; et al
Sengupta, N.; Sahu, S. K.; Jia, B.; Katipomu, S.; Li, H.; Koto, F.; Marshall, W.; Gosal, G.; Liu, C.; Chen, Z.; et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149
2023 arXiv
-
[21]
seyoungsong. 2024. flores101mm100175M. https://huggingface.co/seyoungsong/flores101-\_mm100\_175M
2024
-
[22]
Silma-AI. 2024. SILMA 1.0. https://huggingface.co/silma-ai/SILMA-9B-Instruct-v1.0
2024
-
[23]
Wang, X.; Chen, N.; Chen, J.; Hu, Y.; Wang, Y.; Wu, X.; Gao, A.; Wan, X.; Li, H.; and Wang, B. 2024. Apollo: Lightweight multilingual medical llms towards democratizing medical ai to 6b people. arXiv preprint arXiv:2403.03640
2024 arXiv
-
[24]
Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M.; Xue, M.;...
2024 arXiv
-
[25]
Yue, X.; Song, Y.; Asai, A.; Kim, S.; de Dieu Nyandwi, J.; Khanuja, S.; Kantharuban, A.; Sutawika, L.; Ramamoorthy, S.; and Neubig, G. 2024. Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages. arXiv preprint arXiv:2410.16153
2024 arXiv
-
[26]
Zhang, X.; Wang, Y.; and Chen, H. 2023. Large Language Models in Medicine: The Potentials and Pitfalls
2023
-
[27]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[28]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.