{"id":"b20053b5-15d1-4ed2-84a1-a4fc532ebdec","arxiv_id":"2509.10095","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning Mistral-7B on a new Arabic medical Q&A dataset yields BERTScore F1 of 68.50 percent, higher than LLaMA-2, GPT-2 Medium, AraGPT2, and BLOOM in this study.","lead":"This paper fine-tunes five large language models on Arabic medical conversations scraped from Facebook and reports that Mistral-7B produces the most semantically similar replies by BERTScore. It is a small benchmark study whose usefulness depends on whether the data and evaluation protocol are sound enough to trust the ranking.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation appears to use the fine-tuning dataset for scoring, so the reported Mistral advantage may reflect memorization rather than generalization; no held-out split is described.","rationale":"The reader and I converge on the same load-bearing weakness: the paper does not describe a train/test split, so the central comparative claim may rest on in-sample BERTScore values rather than measured generalization. I checked for additional internal inconsistencies that could independently undermine the numbers: Table I reports fine-tuned values that appear to be precision scores while Table II reports F1 values, and for Mistral the reported precision of 68.50 and recall of 69.08 imply an F1 of 68.79 under the paper's own harmonic-mean formula, not the reported 68.50. These are secondary but reinforce the need for a reproducible evaluation protocol. The paper is honest about some limitations, noting in Section V that traditional baselines were not comprehensively tested and in Section VI.C that user-generated data quality is a limitation; those admissions are consistent with my concern but do not repair the missing split. Because the data and code are unavailable and the evaluation procedure as written cannot support the model-ranking claim, I see no reason to change the reader's REJECT verdict.","tokens_in":8177,"tokens_out":3622,"duration_ms":32902,"concrete_test":"Inspect the authors' code or configuration for the exact data partition, or ask the authors for it. If no partition exists, re-run the LoRA fine-tuning of Mistral-7B-Instruct-v0.2 and LLaMA-2-7B on an 80/10/10 split with a documented random seed, ensuring no social-media thread appears in both train and test, then compute BERTScore only on the test split with 95% confidence intervals from multiple seeds. If the test-set gap is under 1.25 points or is not statistically significant, the ranking claim is unsupported. This check settles whether the reported 68.50 F1 is a generalization result or a training-fit artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central comparative claim, the BERTScore numbers in Table II must measure generalization to unseen patient questions. The paper never establishes this: Section IV introduces a curated dataset of roughly 20,000 QA pairs used for fine-tuning, and Section VI.A says the five models 'were evaluated on subsets of the dataset,' with no mention of a train/test split, random seed, or selection rule. Section V.E lists evaluation/save strategy, batch size, warmup, and mixed precision, but no data partition. If the evaluation subsets overlap the LoRA training data, BERTScore can be inflated by memorization, and the 1.25-point Mistral-vs-LLaMA F1 gap (68.50 vs 67.25) may be an in-sample artifact. The abstract's clinical accuracy claims require an entirely different kind of evidence; even the model ranking is not defensible as reported. This is not a dispute about consensus: it is a missing control that the paper's own protocol should have specified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Arabic medical text generation system by fine-tuning five generative LLMs (Mistral-7B-Instruct-v0.2, LLaMA-2-7B, AraGPT2-Base, GPT-2 Medium, and BLOOM-560M) on a newly scraped dataset of roughly 20,000 Arabic question–answer pairs collected from Facebook medical groups. The authors use LoRA for efficient fine-tuning and evaluate the models with BERTScore precision, recall, and F1. They report that Mistral-7B-Instruct-v0.2 achieves the best average F1 of 68.50% and, based on this and qualitative inspection, claim the system can provide accurate medical advice, diagnoses, drug recommendations, and treatment plans.","tokens_in":8478,"tokens_out":6153,"duration_ms":51070,"significance":"If the evaluation were sound, the paper would contribute a new Arabic medical conversation dataset and a comparative benchmark of generative LLMs under BERTScore. The BERTScore equations in Section VI.B are standard and correctly stated, and the choice of models spans useful size and language-coverage ranges. However, the central empirical claim is not supported as reported: the evaluation appears to reuse the fine-tuning dataset without a described held-out split, no clinician or expert validation is provided for the medical-accuracy claims, and the baseline tables are internally inconsistent and lack uncertainty estimates. The dataset itself could be a valuable resource if released with proper documentation, but the current manuscript does not provide enough detail to reproduce or trust the headline results.","major_comments":[{"comment":"The paper never specifies a train/test split. Section IV describes the curated dataset of approximately 20,000 QA pairs used for fine-tuning, and Section VI.A states only that 'Five models were evaluated on subsets of the dataset,' with no mention of a held-out partition, random seed, or selection rule. Since the same dataset is used for fine-tuning, the BERTScore values in Tables I and II may reflect memorization of training data rather than generalization to unseen patient questions. This undermines the central comparative claim that Mistral-7B-Instruct-v0.2 outperforms the other models; the authors must report the evaluation split and re-run the evaluation on a truly held-out subset.","section":"§IV, §V.E, §VI.A"},{"comment":"The abstract claims the system can provide 'accurate medical advice, diagnoses, drug recommendations, and treatment plans,' but no medical expert or clinician validation is reported. The only quantitative evidence is BERTScore, which measures semantic similarity to references scraped from Facebook posts without credential verification; BERTScore is not a measure of medical correctness. Section III itself states that ensuring accuracy requires validation mechanisms such as verified medical sources or expert reviews, yet none are implemented in the paper. The clinical accuracy claims should be removed or supported by appropriate human evaluation.","section":"Abstract, §III, §VI.D"},{"comment":"The baseline and fine-tuned scores are not reproducible or mutually consistent. Table I reports 'Base Model (%)' and 'Fine-Tuned (%)' but does not define which metric (precision, recall, or F1) is shown or how the base models were evaluated (prompt format, decoding strategy, evaluation set). Moreover, for LLaMA-2-7B and BLOOM-560M, the 'Fine-Tuned (%)' values in Table I (66.50 and 64.03) do not match the F1 scores in Table II (67.25 and 65.60). The authors also report single-run numbers with no error bars or significance tests, so the 1.25-point Mistral-versus-LLaMA F1 gap is not established as meaningful. Please clarify the metric and provide uncertainty estimates.","section":"Table I, Table II, §V.C"},{"comment":"The phrase 'subsets of the dataset' is too vague to be reproducible. The number of evaluation examples, the sampling procedure, and whether the subsets overlap the fine-tuning examples are not specified anywhere in the paper. This missing detail is directly tied to the circularity concern in the first major comment and must be documented in full.","section":"§VI.A"}],"minor_comments":[{"comment":"The symbol S_{i,j} is used in all three BERTScore equations but is never defined. It should be stated as the cosine similarity between contextual token embeddings, and the specific embedding model (e.g., which BERT variant and which language) should be identified for reproducibility.","section":"§VI.B, Eqs. (1)-(3)"},{"comment":"The paper alternates between 'BERT Score' and 'BERTScore'. Please use a single spelling consistently, and cite the original BERTScore paper (Zhang et al., 2020) rather than reference [6], which is a medical-chatbot case study and does not introduce the metric.","section":"Throughout"},{"comment":"Several citation-to-model mappings are incorrect: GPT-2 Medium is cited to [9] (Kühl et al., a general AI/ML paper), and BLOOM-560M is cited to [27] (Nova, a healthcare AI paper). These should point to the actual model resources (e.g., Hugging Face model cards).","section":"References"},{"comment":"The caption of Table I should explicitly state the evaluation metric, since Table II reports precision, recall, and F1 separately and the single 'Fine-Tuned (%)' column is ambiguous.","section":"Table I, §V.C"},{"comment":"Figure 3 is not referenced in the text. Please add a cross-reference and describe what is being compared, since the figure caption alone does not explain the qualitative evaluation.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"I considered reject, but the central empirical claim could in principle be repaired if the authors re-run the evaluation on a properly held-out split, report error bars, and remove or re-scope the clinical-accuracy claims. However, if the authors cannot provide the data split, evaluation details, or code upon request, this paper should not be accepted in any form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is an honest but incremental engineering report. The genuinely new piece is a roughly 20k QA dataset scraped from Arabic Facebook medical groups, plus a comparison of five fine-tuned models (Mistral, LLaMA-2, GPT-2 Medium, AraGPT2, BLOOM) on medical text generation. The BERTScore equations are correct, the tables are internally consistent, and the authors openly admit they did not run traditional baselines due to time constraints. That candor counts for something.\n\nThe soft spots are serious, though. The load-bearing problem is the evaluation split. Section VI.A says the models were \"evaluated on subsets of the dataset,\" Section VI.C mentions \"matching dataset entries,\" and the same Section IV dataset was used for fine-tuning. No train/test split, random seed, or selection rule is described anywhere. That means the reported BERTScore numbers—Mistral F1 68.50 vs LLaMA 67.25—may reflect memorization rather than generalization to unseen patient questions. This is not a minor quibble; it is the central comparative claim, and as reported it is not defensible. The abstract's claims about accurate medical advice, diagnoses, and treatment plans are also unsupported: BERTScore measures token-level semantic similarity, not clinical correctness. There is no clinician review, no error bars, and no released data or code, so the results are not reproducible.\n\nIn proportion: the paper is not nonsense. It is a standard fine-tuning pipeline applied to a domain where Arabic medical conversation data is genuinely scarce. The dataset, if released, would be a useful resource. But the evaluation protocol undercuts the headline result. I would not desk-reject on novelty grounds, but I would not send this to peer review in its current form. The fix is straightforward: specify the held-out split, report error bars, release the dataset, and drop the clinical-accuracy language. With those changes it becomes a citable engineering data point. As it stands, the main result should be treated as an in-sample artifact until proven otherwise.\n\nRecommendation: ask the authors for a revised version with a proper split and data release. Not worth referee time now.","headline":"A routine LoRA fine-tuning comparison on a new Arabic medical QA dataset, but the missing train/test split means the headline Mistral advantage may be in-sample memorization, and the clinical-accuracy claims outrun the evidence.","tokens_in":8878,"tokens_out":1799,"would_cite":false,"duration_ms":16956,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning Mistral-7B-Instruct-v0.2 on a 20,000-pair Arabic medical corpus scraped from social media gives it the best BERTScore among five generative models for Arabic medical text generation.","keywords":["Arabic large language models","medical text generation","BERTScore","LoRA fine-tuning","Arabic dialects","healthcare chatbot","social media medical corpus","low-resource NLP"],"falsifier":"Run the same fine-tuned models on a held-out Arabic medical QA set whose questions were never in the training corpus and whose reference answers were written independently by clinicians; if Mistral-7B no longer clearly beats LLaMA-2-7B, or if clinician review finds high-BERTScore answers are medically wrong, the reported ranking and accuracy claim are refuted.","tokens_in":7970,"feed_emoji":"🩺","tokens_out":8511,"duration_ms":64289,"temperature":0.7,"pith_summary":"The paper tries to establish that a general-purpose instruction-tuned language model can be adapted to generate useful Arabic medical text despite scarce domain data and informal, dialect-rich patient language. It curates about 20,000 question–answer pairs from Arabic medical conversations on social media, cleans them for dialectal variation, and fine-tunes five generative models with low-rank adaptation. The reported result is that Mistral-7B-Instruct-v0.2 reaches the highest average BERTScore F1 of 68.50%, ahead of LLaMA-2-7B, AraGPT2, GPT-2 Medium, and BLOOM-560M, and that its answers are judged coherent and contextually appropriate in qualitative review. If this holds, it gives a practical recipe for low-resource language medical assistants: collect real patient–doctor exchanges, clean them, and fine-tune an open generative model.","feed_headline":"Mistral-7B tops Arabic medical text generation at 68.5 BERTScore","feed_subtitle":"LoRA fine-tuning on 20,000 scraped Arabic health Q&As lets Mistral-7B beat four baselines on semantic similarity.","key_machinery":"The load-bearing object is the curated dataset: over 40,000 Arabic social-media medical posts, refined to roughly 20,000 question–answer pairs that retain informal, dialectal language. The method wraps around this corpus in two steps: low-rank adaptation (LoRA) fine-tunes decoder-only generative transformers cheaply, and BERTScore, which compares candidate and reference tokens by semantic embedding similarity, ranks the outputs. The corpus supplies the medical-domain signal, LoRA makes adaptation affordable, and BERTScore turns semantic closeness into a quantitative ranking.","core_discovery":"The central claim is that fine-tuning Mistral-7B-Instruct-v0.2 on a newly collected Arabic medical corpus makes it the best of five tested generative models for Arabic medical text generation. The paper reports average BERTScore precision of 68.50%, recall of 69.08%, and F1 of 68.50% for Mistral-7B, versus 67.25% F1 for LLaMA-2-7B, 65.07% for GPT-2 Medium, 65.04% for AraGPT2, and 65.60% for BLOOM-560M. The authors also claim the fine-tuned system can provide accurate medical advice, diagnoses, drug recommendations, and treatment plans in response to informal patient input, and that qualitative review supports the coherence and relevance of the generated replies.","pith_inferences":["The paper never states a train/test split for the BERTScore evaluation, so a fair reading treats the numbers as in-sample until a properly held-out split is run; this is the first experiment that would harden or weaken the claim.","Because the reference answers come from unverified social-media advice, even a high BERTScore cannot establish clinical safety; a clinician-rated gold standard is the missing link between fluent text and accurate medical advice.","The corpus-construction half of the contribution is probably the most transferable part: the same scrape-and-clean recipe should work for Swahili, Urdu, or other under-resourced medical dialects, possibly with stronger effect than the specific model choice.","The paper's own stated need for validation mechanisms suggests the 'accurate diagnoses, drugs, and treatment plans' language should be read as a design goal rather than a demonstrated clinical capability."],"forward_implications":["Mistral-7B-Instruct-v0.2, after LoRA fine-tuning on the Arabic medical corpus, becomes a practical candidate for generating replies in Arabic hospital-management and patient-assistance systems.","All five models improve their BERTScore after fine-tuning, so the collected corpus has value beyond the winning model, as adaptation data rather than a benchmark.","The small performance gap among mid-size models means deployment choices can weigh speed and cost: AraGPT2 at 148M parameters stays within about 3.5 F1 points of the 7B model.","The pipeline of scrape, clean, LoRA fine-tune, and BERTScore evaluation is directly reusable for other dialect-rich, low-resource medical languages."],"supporting_citations":[{"why":"It supplies the method and precedent for extracting health posts from Facebook groups.","marker":"[16]"},{"why":"It justifies treating medical social-media content as a valuable training source.","marker":"[15]"},{"why":"It provides the prior Arabic healthcare deep-learning system and the dataset/preprocessing references the work extends.","marker":"[11]"},{"why":"It is cited for the BERTScore equations and an Egyptian Arabic medical chatbot baseline.","marker":"[6]"},{"why":"It underpins the fine-tuning approach for generative LLMs in medical research and healthcare.","marker":"[5]"},{"why":"It supports the generative-AI-in-medicine approach and the need to validate generated advice.","marker":"[18]"},{"why":"It provides the review context and model-selection rationale for large language models in healthcare.","marker":"[4]"}],"fun_headline_variants":["Fine-tuned Mistral-7B wins Arabic medical text generation race","Mistral-7B beats four models in Arabic medical text generation","Arabic medical AI: Mistral-7B tops BERTScore benchmarks","Mistral-7B achieves top BERTScore for Arabic medical replies","Mistral-7B hits 68.5 BERTScore in Arabic medical generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the BERTScore figures measure how well a fine-tuned model answers new patient questions rather than how closely it repeats training examples, and that the social-media doctor replies used as reference answers are medically sound.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Mistral-7B wins Arabic medical text generation race","Mistral-7B beats four models in Arabic medical text generation","Arabic medical AI: Mistral-7B tops BERTScore benchmarks","Mistral-7B achieves top BERTScore for Arabic medical replies","Mistral-7B hits 68.5 BERTScore in Arabic medical generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3117,"prompt_tokens":1027,"completion_tokens":2090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1989}},"tokens_in":643,"tokens_out":2090,"duration_ms":11495,"temperature":1.0,"reasoning_tokens":1989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:56:37.236922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuned models on a held-out Arabic medical QA set whose questions were never in the training corpus and whose reference answers were written independently by clinicians; if Mistral-7B no longer clearly beats LLaMA-2-7B, or if clinician review finds high-BERTScore answers are medically wrong, the reported ranking and accuracy claim are refuted.","supporting_citations":[{"cited_title":"Health topics on facebook groups: content analysis of posts in multiple sclerosis communities,","cited_arxiv_id":null,"evidence_quote":"It supplies the method and precedent for extracting health posts from Facebook groups."},{"cited_title":"How valuable is medical social media data? content analysis of the medical web,","cited_arxiv_id":null,"evidence_quote":"It justifies treating medical social-media content as a valuable training source."},{"cited_title":"Deep learning for arabic healthcare: Medicalbot,","cited_arxiv_id":null,"evidence_quote":"It provides the prior Arabic healthcare deep-learning system and the dataset/preprocessing references the work extends."},{"cited_title":"Enhancing healthcare management: A case study of a medical chatbot in egypt,","cited_arxiv_id":null,"evidence_quote":"It is cited for the BERTScore equations and an Egyptian Arabic medical chatbot baseline."},{"cited_title":"A study of generative large language model for medical research and healthcare,","cited_arxiv_id":null,"evidence_quote":"It underpins the fine-tuning approach for generative LLMs in medical research and healthcare."},{"cited_title":"Generative ai in medicine and healthcare: promises, opportunities and challenges,","cited_arxiv_id":null,"evidence_quote":"It supports the generative-AI-in-medicine approach and the need to validate generated advice."}],"review_version":1}