{"id":"6d99abe1-edd6-475d-8ce1-23084e5c79ef","arxiv_id":"2501.09825","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The optimal Arabic-English training-data ratio for a medical LLM varies by task, and fine-tuning alone does not reliably improve Arabic clinical performance.","lead":"This paper tests how well large language models handle Arabic medical questions, and tries to improve them by translating data and fine-tuning on different Arabic-English mixes. It finds that the best language mix depends on the task, and that fine-tuning alone rarely closes the gap with English.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's task-dependent optimal ratio is confounded: varying the Arabic-English ratio also varies dataset composition, so the central claim is not identifiable.","rationale":"The reader's weakest assumption—that the Arabic benchmarks are valid translations—is plausible, especially given the unusual Arabic-over-English scores on PubMedQA in Table 1. However, that is not the single most load-bearing issue for the paper's central claim: even if the translations are perfect, the fine-tuning comparison in Table 4 does not support the claim because language ratio is confounded with dataset composition. Table 3 makes this transparent: the Arabic side contains AHQAD, CIDAR, synthetic open-ended data, and translated Med42, while the English side is only Med42. The constant 469.97M token budget controls total size but not content, so the reported task-dependent optima could arise from the specific datasets assigned to each language. This is a correctable flaw: rerunning with parallel translated data would settle it. I agree with the reader's overall conditional posture, but for a different primary reason, hence partial agreement. The paper should also soften or remove the pretraining conclusion, since no pretraining experiment was performed. The work is otherwise useful and transparent about its data sources, and the zero-shot evaluation is informative; the issue is specifically that the headline ratio finding is not identifiable from the current design.","tokens_in":11805,"tokens_out":4165,"duration_ms":43250,"concrete_test":"Run a controlled ratio sweep using a single parallel corpus: take the same 500K Med42 samples, translate them to Arabic with Qwen2.5-72B as already done, then fine-tune Llama-3.1-8B on ratios 100/0, 80/20, 50/50, 20/80, and 0/100 of these identical samples in Arabic/English, keeping token count, format, epochs, and evaluation identical. If the task-dependent optimal ratios persist (PubMedQA best at 100% Arabic, MMLU best at 0% Arabic), the language effect is supported; if they flatten or reorder, the original Table 4 result is a dataset-composition artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper presents Table 4 as varying only the Arabic-English data ratio, but the two language streams are not parallel. Table 3 shows the Arabic stream is a mixture of AHQAD (8.33M tokens), translated Med42 (230.69M), CIDAR (1.34M), and synthetic open-ended QA (240M), while the English stream is exclusively the English Med42 dataset (464.97M). Consequently, moving from 0% to 100% Arabic also moves from Med42-only chat/CoT data to a different mix containing native Arabic QA, instruction data, and synthetic QA. The headline finding—PubMedQA favors 100% Arabic, MedMCQA favors 80%, MedQA favors 40%, and MMLU favors 0%—could therefore reflect task/format/source similarity rather than language. For example, AHQAD and synthetic QA may provide PubMedQA-style question-answer pairs, while Med42 may align better with MMLU's knowledge-probe format. Because language ratio and dataset composition are collinear, the central claim about optimal language mix varying by task is not identifiable from this experiment. Additionally, the conclusion that pretraining is necessary is not tested, since no pretraining arm is run; the paper only fine-tunes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates Arabic medical QA performance of several open-source LLMs on Arabic translations of PubMedQA, MedMCQA, MedQA, and Medical MMLU. It then studies a translation pipeline for Arabic-to-English processing and fine-tunes Llama 3.1 8B and 70B models on different Arabic-English data mixtures, reporting that the optimal mixture varies across tasks. The authors conclude that fine-tuning alone may be insufficient for adding a new language and that pretraining may be necessary.","tokens_in":12045,"tokens_out":3480,"duration_ms":38596,"significance":"If the empirical claims hold, the paper would provide useful evidence on cross-lingual medical QA and on practical data-mixture choices for Arabic medical LLMs. Its strengths include a broad zero-shot comparison across model families, an explicit comparison of translation-based adaptation against direct fine-tuning, and a relatively detailed description of the fine-tuning data pipeline. However, the central claim about task-dependent optimal language ratios currently rests on experiments in which language ratio is confounded with dataset composition, and the Arabic evaluation sets are not validated for translation quality or leakage; these issues are load-bearing for the main conclusions. The paper also reports no uncertainty quantification, so the observed differences may not be statistically reliable.","major_comments":[{"comment":"The central claim that the optimal Arabic-English mixture is task-dependent is not identifiable from the fine-tuning experiments as designed. Table 3 shows that the Arabic data stream is a mixture of AHQAD, translated Med42, CIDAR, and synthetic open-ended QA, whereas the English stream is exclusively the English Med42 dataset. Consequently, changing the Arabic-English ratio in Table 4 also changes the dataset composition, task format, and source distribution. The pattern that PubMedQA favors 100% Arabic, for instance, could equally reflect the presence of AHQAD and synthetic QA pairs rather than the Arabic language; similarly, MMLU favoring 0% Arabic could reflect Med42's closer format match with the MMLU-style knowledge probe. The authors need a controlled comparison in which the only difference between streams is language, such as translating the same Med42 records into Arabic and mixing those, or a composition-controlled ablation that isolates the language effect.","section":"Table 4 and 'Language Specific Finetuning'"},{"comment":"The Arabic evaluation datasets need validation before the reported performance gaps and mixture effects can be interpreted. Several models score substantially higher on Arabic PubMedQA than on English PubMedQA, e.g., Qwen2.5-3B-Instruct at 61.2 vs 29.2 and Qwen2.5-7B-Instruct at 74.4 vs 45.2. Such an inversion is surprising for zero-shot evaluation of models with stronger English capabilities and suggests translation artifacts, answer leakage, or an evaluation-pipeline bug. The paper describes the BiMediX semi-automated translation pipeline but reports no human fidelity checks, no post-translation validation, and no checks for leaked answer information. Because all zero-shot, translation-pipeline, and fine-tuning results depend on these Arabic datasets, the authors should provide translation quality statistics, sample-level manual inspection, and leakage analysis.","section":"Table 1 and 'Arabic Evaluation Datasets'"},{"comment":"The paper reports no error bars, confidence intervals, or significance tests, yet many of the claimed task-dependent differences are small. For example, Llama 3.1 8B-Baseline Arabic MedMCQA accuracies range from 32.2 to 35.1 across ratios, and several adjacent differences are 1-2 points. Without variance estimates across seeds or repeated evaluation runs, these differences could be noise, and the claim that 'the best Arabic accuracy for MedMCQA comes from 80% Arabic' is not statistically supported. The authors should report standard deviations, paired significance tests, or at least confidence intervals for the fine-tuning runs.","section":"Table 4 and 'Results'"},{"comment":"The conclusion that 'data and computationally intensive pretraining methods may still be necessary' is not supported by the experiments. The paper only compares different fine-tuning mixtures; no continued-pretraining or pretraining-from-scratch arm is run. The statement is an untested hypothesis, and the conclusion should be rephrased as a motivation for future work rather than as an empirical finding of this study.","section":"Conclusion"}],"minor_comments":[{"comment":"The Arabic example in this section is rendered as character escapes such as '/char48/char2e/char51/...' rather than readable Arabic text; this should be replaced with actual Arabic script or a transliteration.","section":"Modifications to Harness Pipeline"},{"comment":"The 'Qwen Translation Example' in the appendix is also displayed as encoded '/char' sequences, making the example unusable for verifying translation quality; the appendix should show proper Arabic text.","section":"Appendix"},{"comment":"There are several typographical inconsistencies in Table 4, including '71.62' in the 8B-Baseline English Only row, '2 8.7' in the Strong English Majority row, and '66. 5' in the 70B-Instruct row; these should be corrected.","section":"Table 4"},{"comment":"The paper does not state the exact number of evaluation examples for each benchmark or the evaluation prompt/template used; this information is necessary for reproducibility and for interpreting percentage differences.","section":"General experimental setup"},{"comment":"The fine-tuning section would benefit from reporting effective batch size, optimizer details, wall-clock time, and the choice of checkpoint (last checkpoint vs best on a validation set); the current description is not fully reproducible.","section":"Fine-tuning pipeline"}],"recommendation":"major_revision","confidential_remarks":"The fine-tuning data include the Med42 dataset, whose authors overlap with the present paper, but the evaluation sets come from BiMediX and are not used for training, so I do not see a direct train-evaluation contamination. Still, releasing the Arabic evaluation datasets and the fine-tuning data would substantially strengthen reproducibility; without them, the empirical claims are hard for the community to verify independently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical case study for anyone building Arabic clinical LLMs, but the headline claim about task-dependent language ratios is not actually identified by the experiment. The Arabic and English fine-tuning streams differ in dataset composition, so Table 4 varies two things at once.\n\nWhat's genuinely new: the paper gives a systematic comparison of Arabic-English fine-tuning ratios for medical QA on Llama 3.1 8B and 70B, evaluated on four translated benchmarks. That map—PubMedQA favoring 100% Arabic, MedMCQA 80%, MedQA 40%, MMLU 0% for the 8B base—is not in the prior work I know. The translation-pipeline comparison (Table 2) is also useful: general LLMs beat specialized MT models on medical terminology, and the appendix's worked examples make that concrete. The authors also deserve credit for acknowledging that close-ended QA doesn't capture generation, safety, or bias.\n\nThe soft spots are real, and the biggest one is the confound. Table 3 shows the Arabic stream is a mix of AHQAD, translated Med42, CIDAR, and synthetic QA, while the English stream is Med42 only. So moving from 0% to 100% Arabic also moves from Med42-style chat/CoT data to a different set of sources. The task-dependent optima in Table 4 could reflect format or source similarity rather than language per se. That's not a minor quibble; it undermines the central claim.\n\nOther issues: no error bars or significance tests, so differences like 32.2 vs 35.1 are hard to interpret. Several models score higher on Arabic PubMedQA than English PubMedQA, which is suspicious and suggests the Arabic translations may have artifacts or leakage. The BiMediX translation pipeline is used without independent validation of medical fidelity. And the conclusion that 'pretraining methods may still be necessary' is not supported by any pretraining experiment; only fine-tuning was run. The appendix also contains a garbled Arabic example (the Qwen translation is rendered as character codes), which doesn't help.\n\nWho this is for: practitioners working on Arabic medical NLP will find Table 4 and the translation results worth knowing even if the causal claim is shaky. It deserves a serious referee, but the authors should be asked to disentangle language from data source, validate the Arabic eval sets, and soften the pretraining claim.","headline":"Useful Arabic medical QA case study whose central claim about task-dependent language ratios is confounded by dataset composition.","tokens_in":12582,"tokens_out":2221,"would_cite":true,"duration_ms":20895,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Arabic medical AI needs task-specific language mixes; fine-tuning alone may not close the gap.","keywords":["Arabic medical NLP","multilingual large language models","cross-lingual medical QA","fine-tuning data mixtures","language transfer","translation pipelines","medical question answering","low-resource languages"],"falsifier":"Take a clinician-translated, back-translated Arabic version of PubMedQA and MedQA, then rerun the ratio grid; if the best ratios change or Arabic no longer beats English, the original task-dependent optima were translation artifacts.","tokens_in":11606,"feed_emoji":"🩺","tokens_out":6170,"duration_ms":55912,"temperature":0.7,"pith_summary":"This paper asks how to give large language models Arabic clinical competence without English-level resources. It evaluates open models on Arabic versions of four medical question benchmarks, then fine-tunes Llama 3.1 at seven Arabic-English data ratios. The central finding is that no single ratio is best: PubMedQA favors pure Arabic data, MedMCQA favors an 80/20 Arabic-English mix, MedQA favors an English-majority mix, and MMLU favors English-only data. The paper also finds that fine-tuning often fails to beat the original model, and argues that adding a language to a model may require continued pretraining rather than supervised fine-tuning alone.","feed_headline":"Best Arabic-English training mix for medical AI shifts by task","feed_subtitle":"For medical reasoning tasks pure Arabic data wins; for knowledge quizzes English-only data works best","key_machinery":"The load-bearing object is the bilingual fine-tuning grid: Llama 3.1 8B and 70B models trained for two epochs on a fixed 469.97M-token budget, with Arabic-English ratios 100/0, 80/20, 60/40, 50/50, 40/60, 20/80, and 0/100, assembled from native Arabic data (AHQAD, CIDAR), Qwen-translated Med42 and synthetic QA data, and English Med42 data. Evaluation runs on Arabic translations of PubMedQA, MedMCQA, MedQA, and MMLU using a Harness pipeline modified for right-to-left text and for scoring the full answer text rather than a single choice label. A second mechanism is the translation-comparison pipeline, which tests whether routing Arabic through English via various translators closes the gap; it does not.","core_discovery":"The paper's claim is that simply translating medical data does not guarantee strong target-language performance: the effectiveness of bilingual fine-tuning depends on the task's reasoning demands. In the paper's experiments, Llama 3.1 8B reaches its best Arabic accuracy with 100% Arabic data on PubMedQA (71.2), 80% Arabic on MedMCQA (35.1), 40% Arabic on MedQA (29.8), and 0% Arabic on MMLU (42.4), and these ordering patterns reproduce across base and instruct variants. For the 70B models, Arabic-only training is consistently best on Arabic tasks, while fine-tuned instruct models rarely surpass their already-aligned baselines. The paper concludes that data- and compute-intensive pretraining may still be necessary to reach native-level Arabic clinical performance.","pith_inferences":["Inference: The odd finding that several models score higher on Arabic PubMedQA than on English PubMedQA implies the Arabic translations may be easier or contain answer leakage; if so, the task-dependent optimum ratios partly measure translation properties, not clinical reasoning.","Inference: The pattern suggests a two-stage recipe worth testing elsewhere: continued pretraining on target-language domain text to establish the language, followed by task-specific ratio tuning only as a fine adjustment.","Inference: The same ratio-grid experiment could be repeated for other non-Latin-script languages, such as Hindi, Amharic, or Urdu, to test whether the task-dependence generalizes beyond Arabic's morphology.","Inference: The full-answer scoring modification likely changes absolute accuracies; re-running the grid with choice-only scoring would reveal how much of the reported task-dependent ordering is an artifact of the scoring method."],"forward_implications":["Deploying Arabic medical QA systems requires per-task data-curation strategies; a single best Arabic mix does not exist.","For knowledge-heavy multiple-choice benchmarks like MMLU, English-only fine-tuning can improve Arabic scores, so language mixing is not always the lever that matters.","Larger models respond more uniformly to language mixing, suggesting scale changes how language knowledge is acquired from fine-tuning data.","Because fine-tuned instruct models rarely surpass their baselines, model developers should budget for continued pretraining when adding a low-resource language.","Translation pipelines remain a partial remedy: general-purpose LLMs translate medical text better than specialized translation models, but still underperform native proficiency and add computational cost."],"supporting_citations":[{"why":"Provides the semi-automated Arabic translation pipeline and the Arabic evaluation datasets used for zero-shot and fine-tuning evaluation.","marker":"Pieri et al. 2024a"},{"why":"Defines the Llama 3.1 base and instruct models that are the paper's fine-tuning subjects and a key baseline.","marker":"Dubey et al. 2024"},{"why":"Qwen2.5 models are the strongest Arabic performers in zero-shot evaluation and the chosen translators for the Arabic fine-tuning data.","marker":"Yang et al. 2024"},{"why":"Med42 supplies the large English medical fine-tuning dataset that is translated into Arabic for the data-mixing experiments.","marker":"Christophe et al. 2024"},{"why":"Harness is the evaluation framework modified for Arabic script and full-answer log-likelihood scoring.","marker":"Gao et al. 2024"},{"why":"PubMedQA is one of the four medical benchmarks; its Arabic version drives the result that Arabic-only data is best.","marker":"Jin et al. 2019"},{"why":"MedMCQA is the benchmark where an 80/20 Arabic-English mix is optimal.","marker":"Pal, Umapathi, and Sankarasubbu 2022"},{"why":"MedQA is the benchmark where an English-majority mix is optimal.","marker":"Jin et al. 2021"},{"why":"MMLU's biomedical subsets form the benchmark where English-only fine-tuning is optimal.","marker":"Hendrycks et al. 2021"}],"fun_headline_variants":["Task, not size, sets ideal Arabic-English mix for medical AI","Arabic medical AI: Best language mix varies by task, not scale","Pure Arabic data for reasoning, English for knowledge quizzes","Translating data isn't enough: Tune language mix per medical task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings depend on Arabic translations of English benchmarks being medically faithful and free of leakage; the paper did not independently validate the translated answer options, and some Arabic-above-English scores suggest the translations may be easier than the originals.","fun_headline_variants_meta":{"raw":{"variants":["Task, not size, sets ideal Arabic-English mix for medical AI","Arabic medical AI: Best language mix varies by task, not scale","Pure Arabic data for reasoning, English for knowledge quizzes","Translating data isn't enough: Tune language mix per medical task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1309,"prompt_tokens":825,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":441,"tokens_out":484,"duration_ms":5754,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:37:29.636567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a clinician-translated, back-translated Arabic version of PubMedQA and MedQA, then rerun the ratio grid; if the best ratios change or Arabic no longer beats English, the original task-dependent optima were translation artifacts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PubMedQA is one of the four medical benchmarks; its Arabic version drives the result that Arabic-only data is best."},{"cited_title":"K.; and Sankarasubbu, M","cited_arxiv_id":null,"evidence_quote":"MedMCQA is the benchmark where an 80/20 Arabic-English mix is optimal."}],"review_version":1}