{"id":"204b8e25-cfb3-41ec-97ba-93b270ae2ee1","arxiv_id":"2412.15236","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CareBot combines stable and boost continuous pretraining, supervised tuning, and DPO to make an 8B bilingual medical LLM that beats several prior medical models and ChatGPT on averaged benchmarks.","lead":"CareBot is a medical chatbot that uses a two-stage pretraining recipe, data filtering, and preference alignment to improve an 8-billion-parameter language model for Chinese and English health questions. The paper reports average gains over ChatGPT and earlier open-source medical models, but the model and data are promised for later release.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 may be inflated: CMB, MedQA, and Huatuo26M are both SFT sources and evaluation benchmarks, with no stated split or decontamination.","rationale":"The reader's weak assumption is well-founded but not the most load-bearing. Even if GPT-4 labels were validated by humans, the Table 1 averages would still be suspect if evaluation benchmarks were used in training. SFT data construction explicitly lists CMB, MedQA, and Huatuo26M as sources; the evaluation section uses CMB, MedQA, and Huatuo26M-test. There is no sentence stating 'we remove all test-set instances' or 'we use only the official train split.' Under the page-facing rule, the absence of this statement is an in-scope limitation. This is not an accusation of deliberate leakage; it is a missing control that determines whether the central claim is meaningful. The concrete test is simple and decisive. If the overlap is zero and the split is documented, the remaining objections (no released weights, missing base-model baseline, GPT-4 judge bias) are enough for a conditional acceptance, not rejection. If overlap is nonzero, the headline comparison must be recomputed and likely changes. Therefore UNCHANGED (still CONDITIONAL) is appropriate, with the reason sharpened.","tokens_in":19235,"tokens_out":5523,"duration_ms":57080,"concrete_test":"Use the released SFT dataset (or ask authors for the exact train/test split metadata) and run exact-match plus MinHash n-gram overlap between the SFT corpus and the CMB test set, MedQA test set, and Huatuo26M-test queries. If any test question or near-duplicate appears in SFT, remove those instances and recompute Table 1 and Figure 3. A clean result would require the authors to state that CMB/MedQA/Huatuo26M entries came only from official training splits and to report the overlap statistics; without that, the 6.69%/8.65% margins are not independently verifiable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the SFT Data Construction section, the paper states that the training corpus 'integrates data from seven publicly available sources: Chinese Medical Dialogue Data, Huatuo26M, MedDialog, ChatMed Consult Dataset, ChatDoctor, CMB, and MedQA.' Three of these names are also evaluation sets used for the headline comparison: CMB (Table 1 and CMB-Clin), MedQA (Table 1), and Huatuo26M-test (Figure 3 single-turn consultation). The manuscript never says that only official train splits were used, and it reports no overlap check between the SFT corpus and these test sets. If any CMB test items, MedQA test questions, or Huatuo26M-test queries are present in the SFT data, CareBot-Chat's average advantage over HuatuoGPT II (6.69%) and ChatGPT (8.65%) in Table 1 is partly or wholly an artifact of memorization. This is more load-bearing than the GPT-4-label concern: the MCQ tables are the quantitative core of the 'state-of-the-art' claim, while GPT-4 preference bias mainly affects the dialogue tables and DPO training. The missing split documentation also matters because the paper promises open-source artifacts that are not yet available, so the overlap cannot be checked externally.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CareBot, a bilingual (Chinese/English) medical language model built on LLaMA-3-8B, trained with a three-stage pipeline: continuous pre-training (CPT), supervised fine-tuning (SFT), and DPO-based RLHF. The main methodological contributions are a two-stage CPT scheme (Stable CPT followed by Boost CPT), a GPT-4-trained data quality assessor called DataRater, and a multi-turn dialogue filtering metric called ConFilter. The authors evaluate CareBot on Chinese and English medical multiple-choice benchmarks (Table 1) and on single-turn and multi-turn medical consultation benchmarks (Tables 2 and 3, Figure 3), reporting that CareBot-Chat outperforms ChatGPT and the open-source medical model HuatuoGPT II on average. The paper promises to release datasets and models.","tokens_in":19495,"tokens_out":6799,"duration_ms":65659,"significance":"If the results are correct, CareBot would be a strong open-source bilingual medical LLM, and the training pipeline—especially the two-stage CPT and ConFilter—would be a useful recipe for domain adaptation. The paper has several genuine strengths: the core MCQ claims are benchmarked against external ground truth, the authors report a t-test for Table 1, they provide ablation-style comparisons for one-stage versus two-stage CPT and for ConFilter versus random selection, and the appendix contains substantial training and evaluation detail. However, the current evidence has important gaps: the SFT corpus overlaps with several evaluation sets without documented decontamination; Table 1 lacks a LLaMA-3-8B or plain-SFT baseline; key thresholds for Deita and ConFilter are missing; the multi-turn dialogue evaluation relies on GPT-4 as both generator and judge without human agreement; and DataRater has no downstream ablation. These gaps do not necessarily invalidate the central claim, but they are load-bearing for reproducibility and for attributing the reported gains to the proposed components.","major_comments":[{"comment":"The SFT corpus is described as integrating data from seven public sources, explicitly including CMB, MedQA, and Huatuo26M (Section 'Supervised Fine-Tuning', Data Construction). The same three collections are used as evaluation benchmarks in Table 1 and Figure 3 (CMB, MedQA, Huatuo26M-test). The manuscript never states that only the official training splits were used and reports no overlap check or decontamination for these sets. Because the models and datasets are not yet released, this overlap cannot be checked externally. If any test items are present in the SFT data, the reported average advantages of CareBot-Chat over HuatuoGPT II (6.69%) and ChatGPT (8.65%) in Table 1 would be partly memorization artifacts. Please document the exact splits, perform and report a decontamination analysis (e.g., n-gram overlap), or re-evaluate on held-out subsets.","section":"Supervised Fine-Tuning / Experimental Setup"},{"comment":"Table 1 compares CareBot-Chat with medical LLMs and ChatGPT, but it does not report the LLaMA-3-8B base model or a plain SFT-only CareBot. Without these baselines, the gains in Table 1 cannot be attributed to the proposed CPT and data-filtering pipeline rather than to the choice of base model. In addition, Figure 4 is captioned 'Comparison of the loss between our proposed two-stage CPT and the plain CPT,' while the text of 'One Stage CPT vs Two Stage CPT' repeatedly refers to 'Acc'; if the figure plots loss, the claim of a 'marked and consistent improvement in Acc' is not directly supported. Please clarify the metric and add the missing baselines or an ablation.","section":"Table 1 / One Stage CPT vs Two Stage CPT"},{"comment":"The selection thresholds for Deita and ConFilter are never specified. The text says 'we set a score threshold' for the Deita score s_i and 'We also set a threshold' for CF in Eq. (3), but no values, normalization, or selection criterion are given. The ConFilter definition also leaves open which model computes L_theta and whether it is frozen. Without these details, the 110k-dialogue selection in Figure 5 and the multi-turn SFT pipeline cannot be reproduced, and the reported benefits of ConFilter cannot be separated from threshold choice. Please report the thresholds and, ideally, a sensitivity analysis.","section":"Multi-turn Medical Dialogue Data (ConFilter)"},{"comment":"GPT-4 is used to construct the subjective DPO preference pairs (it writes the doctor-style response and judges which response is superior) and the same style of four-dimension rating (fluency, relevance, completeness, proficiency) is used to evaluate multi-turn dialogues in Tables 2 and 3. The paper reports no human agreement study on either the preference labels or the dialogue judgments. Since the preferred responses come from GPT-4, an evaluation by GPT-4 on the same dimensions risks confirming the model's own stylistic preferences rather than measuring medical quality. Please add a human evaluation on a sample of dialogues or report agreement statistics (e.g., Cohen's kappa) between GPT-4 and human raters.","section":"RLHF / Results for Medical Consultation Ability"},{"comment":"DataRater is presented as a key contribution, but the paper provides no ablation showing its effect on the final CareBot. Section 'LLM-based Data Quality Filtering' describes the 15k GPT-4-labeled training set and Appendix B reports the MSE of the learned predictor, while Figure 9 compares CPT strategies. There is no experiment that trains CareBot with and without DataRater-filtered CPT data. Without such an ablation, the claim that DataRater 'effectively mitigates data bias' and is responsible for CareBot's performance is not supported. Please add this comparison or weaken the claim accordingly.","section":"LLM-based Data Quality Filtering"}],"minor_comments":[{"comment":"In the turn-10 row, the average for HuatuoGPT II is listed as 3.18, but the four dimension scores (4.96, 3.41, 3.47, 4.27) average to 4.03; please correct the table and re-check the other rows for the same calculation error.","section":"Table 4"},{"comment":"The t-test statement ('p<0.05 under t-test') is underspecified: no test type, comparison group, or number of observations is given; please report the test details so the significance claim can be assessed.","section":"Table 1"},{"comment":"There are numerous typos and inconsistencies, including 'develope' in the abstract, 'pipline' in Figure 1, 'mdedical' in Appendix E, and inconsistent capitalization of 'Carebot' versus 'CareBot'; a careful proofread is needed.","section":"Abstract and Introduction"},{"comment":"The claim that the SFT corpus is 'the largest open-source bilingual medical SFT dataset available' is not supported by any dataset statistics or a comparison with prior datasets; please provide the dataset size, topic coverage, and a quantitative comparison.","section":"Supervised Fine-Tuning, Data Construction"},{"comment":"The paper says 'We will open-source the datasets and models later,' but the GitHub link is not yet populated; please provide an availability statement with a concrete timeline or an anonymous link for review.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The most serious risk is the potential overlap between the SFT training sources and the evaluation sets (CMB, MedQA, Huatuo26M). If the authors cannot document decontamination, the headline numbers should not be taken at face value. The other concerns—missing baselines, missing thresholds, GPT-4-as-judge without human validation, and no DataRater ablation—are addressable with additional experiments and reporting. The paper also currently lacks the promised artifacts, which makes independent verification impossible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine systems effort: a two-stage CPT schedule, a DataRater quality model for CPT, a ConFilter metric for multi-turn SFT selection, and a large bilingual SFT set covering single-turn, multi-turn, and MCQ data. The CPT results in Figure 2 show the two-stage approach does something real, and the ConFilter ablation in Figure 5 is a legitimate internal check. I believe the recipe is coherent and the authors are not sloppy about their own experiments.\n\nThe load-bearing problem is data decontamination. The SFT section says the training corpus integrates Huatuo26M, CMB, and MedQA. The evaluation section uses Huatuo26M-test, CMB test questions, and MedQA test set. The paper never says only official train splits were used, and no overlap check is reported. If any test items leaked into SFT, the Table 1 gains over HuatuoGPT II and ChatGPT are partly memorization. That is more serious than the GPT-4-as-judge circularity, which mainly affects the dialogue tables. The MCQ tables are the quantitative core of the SOTA claim, and they are currently unverifiable. The authors promise open-source artifacts \"later,\" so the community cannot check this independently.\n\nOther soft spots are more minor. The GPT-4 loop—GPT-4 rewriting SFT content, building DPO pairs, and grading dialogue—deserves a human agreement study; without one, the dialogue gains might partly reflect style matching rather than medical quality. Missing baselines (plain LLaMA-3-8B, SFT without CPT) make it hard to isolate what the recipe contributes. Figure 4's caption says loss while the text discusses accuracy; that is a mechanical error but should be fixed. ConFilter is a ratio of two cross-entropy losses, which is a simple idea, but the paper presents it as a metric and tests it, so the novelty is modest but honest.\n\nFor a venue that values complete systems work, this deserves a serious referee, not a desk reject. The recipe is useful to the open-source medical LLM community. But the referee should ask for: documented provenance of every SFT source with explicit train/test split declaration, an n-gram or embedding-level overlap check with all evaluation sets, a human judge study on at least a sample of the dialogue evaluations, and the missing baselines. If the contamination check comes back clean, the paper could be a solid contribution. As it stands, the headline numbers are not trustworthy enough to cite.\n\nRecommendation: send to review, but require the decontamination evidence and artifact release before acceptance.","headline":"A coherent full-pipeline medical LLM recipe whose headline numbers are undercut by likely benchmark contamination, so the performance claims should be read with real caution until the authors document train/test splits and release artifacts.","tokens_in":20049,"tokens_out":1753,"would_cite":false,"duration_ms":17876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims its 8-billion-parameter open model CareBot outperforms ChatGPT and the best open-source medical models on the average of five Chinese and English medical benchmarks.","keywords":["bilingual medical LLM","continuous pre-training","data quality filtering","multi-turn dialogue filtering","direct preference optimization","Chinese medical benchmarks","open-source medical LLM","LLaMA3-8B"],"falsifier":"Have a panel of licensed clinicians blindly score CareBot-Chat and HuatuoGPT II on, say, 200 held-out multi-turn consultations from CMB-Clin using the same four dimensions; if clinicians do not prefer CareBot-Chat overall, the paper's attribution of its consultation advantage to the training recipe is not supported. A second check would re-run the pipeline once with human expert labels substituted for GPT-4 labels at a single stage and compare benchmark scores.","tokens_in":18977,"feed_emoji":"🩺","tokens_out":14368,"duration_ms":123240,"temperature":0.7,"pith_summary":"CareBot is an attempt to show that a fully open-source, bilingual medical assistant can be built from an 8-billion-parameter base model by controlling data quality at every training stage rather than relying on a larger proprietary model. The paper claims that its recipe—two-stage continuous pre-training, a GPT-4-trained quality rater, a large bilingual instruction dataset (described as the largest open-source bilingual medical SFT dataset available) filtered for complexity and dialogue coherence, and direct preference optimization—raises the model above ChatGPT and the strongest open-source medical models on average Chinese and English medical knowledge benchmarks. The authors also claim that the multi-turn dialogue filter, ConFilter, is what gives CareBot its edge in sustained consultations, where rival models drift off topic. If these claims hold, the contribution is a reproducible open-source pipeline for medical LLMs, together with datasets and models the paper says it will release. The broader point is that domain competence can be installed into a small open model by deliberate data curation, which matters for applications where sending patient data to closed APIs is undesirable.","feed_headline":"8B open medical model CareBot beats ChatGPT on benchmark averages","feed_subtitle":"A two-stage CPT recipe, filtered SFT data, and DPO push an 8B open model past proprietary rivals in Chinese and English.","key_machinery":"The central machinery is a two-stage continuous pre-training schedule. Stable CPT first mixes medical pre-training data with general data at 19:1 (token-level Chinese:English 1:9) to insert domain knowledge without degrading general language ability; Boost CPT then mixes very high-quality medical pre-training data with open-source medical SFT data at 1:1 (Chinese:English 4:6) to close the gap toward instruction tuning. Around this sit two data filters: DataRater, a quality-scoring model built on the multilingual bge-m3 encoder and trained on GPT-4 scores, which scores CPT candidates on grammar, information density, semantic consistency, and domain relevance; and ConFilter, which computes the ratio of a turn's loss conditioned on dialogue history to its loss in isolation, $CF^\\theta(H,T)=L_\\theta(t_i|H)/L_\\theta(t_i)$, to reject multi-turn dialogues whose context is either irrelevant or redundantly repetitive. The final stage applies DPO on 12,727 preference pairs, 9,019 subjective (GPT-4-generated and judged) and 3,708 objective (ground-truth multiple-choice answers), to align style while limiting alignment tax.","core_discovery":"Using LLaMA3-8B as the base, the paper trains CareBot in three phases and measures the result on five medical knowledge benchmarks (MedQA, MMLU-Med, CMB, CMMLU-Med, C-Eval-Med) and on Chinese consultation benchmarks. On the average of the five knowledge benchmarks, CareBot-Chat reaches 61.58, exceeding HuatuoGPT II, the best open-source medical model tested, by 6.69 percentage points and ChatGPT by 8.65 percentage points; the DPO-finetuned variant scores 61.39. On multi-turn consultation, CareBot-Chat averages 4.53 on CMtMedQA and 4.22 on CMB-Clin, ahead of every baseline, with its advantage concentrated in relevance and completeness as conversations lengthen. The paper attributes the consultation gains to its ConFilter selection of multi-turn SFT data and the knowledge gains to the two-stage CPT schedule, not to a larger model.","pith_inferences":["Because GPT-4 supplies nearly every quality judgment in the pipeline, the reported gains are best read as gains against GPT-4's standard of medical quality; a human-clinician agreement study would show how much of the win is clinical rather than stylistic.","The DataRater and ConFilter machinery is not medicine-specific, so the same two-stage CPT plus filtered SFT recipe could transfer to other expertise-heavy domains where high-quality open data is scarce.","The pattern of large Chinese gains and slight English losses on some benchmarks suggests that starting from a stronger multilingual base, or rebalancing the CPT data ratios, could shift the trade-off; this is an untested extrapolation.","A direct comparison of CareBot against GPT-4 on the same benchmarks would clarify what remains closed-source advantage, but the paper does not report that comparison."],"forward_implications":["An 8-billion-parameter open-source model can outperform ChatGPT on the average of five medical knowledge benchmarks, suggesting model size is not the binding constraint for domain competence.","The two-stage CPT schedule improves Chinese medical benchmarks substantially while keeping English medical knowledge roughly stable, so the recipe can deliberately shift a base model's language-domain balance.","ConFilter-filtered multi-turn data yields higher relevance and completeness in long consultations, where one-stage and SFT-only baselines degrade.","DPO after SFT leaves knowledge benchmarks roughly unchanged while slightly improving multi-turn consultation scores, consistent with alignment affecting style more than factual knowledge.","The released datasets and models let others reproduce or extend the pipeline, including applying the same data-quality filters to other expertise-heavy domains."],"supporting_citations":[{"why":"Supplies the HuatuoGPT-II baseline and the HuatuoEval single-turn evaluation framework, and represents the one-stage training alternative the paper argues against.","marker":"Chen et al. 2023"},{"why":"Provides the DPO algorithm used in the RLHF stage to align CareBot with human preferences.","marker":"Rafailov et al. 2023"},{"why":"Supplies the Deita complexity-quality scoring used to select single-turn SFT data.","marker":"Liu et al. 2024"},{"why":"Contributes the Huatuo26M dataset used as SFT data and as the single-turn consultation test set.","marker":"Li et al. 2023a"},{"why":"Provides the CMB benchmark and the CMB-Clin multi-turn evaluation protocol with its four rating dimensions.","marker":"Wang et al. 2024"},{"why":"Supplies the MedQA benchmark used to measure English medical concept knowledge.","marker":"Jin et al. 2021"},{"why":"Supplies the MMLU-Med benchmark used to measure English medical concept knowledge.","marker":"Hendrycks et al. 2021"},{"why":"Supplies the CMMLU-Med benchmark used to measure Chinese medical concept knowledge.","marker":"Li et al. 2024"},{"why":"Supplies the C-Eval-Med benchmark used to measure Chinese medical concept knowledge.","marker":"Huang et al. 2023"},{"why":"Supplies the Zhongjing full-pipeline baseline and the CMtMedQA multi-turn dialogue benchmark.","marker":"Yang et al. 2023b"}],"fun_headline_variants":["Open 8B CareBot tops ChatGPT on medical benchmarks","CareBot: open 8B LLM beats ChatGPT in medical tests","Two-stage CPT helps 8B CareBot outscore ChatGPT","Bilingual CareBot, 8B, surpasses ChatGPT on med averages","Open-source CareBot edges ChatGPT on five med benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Throughout the pipeline, GPT-4 supplies the labels, quality scores, rewritten answers, preference pairs, and dialogue judgments that decide what counts as good medical text, and the paper reports no human agreement study on any of those judgments.","fun_headline_variants_meta":{"raw":{"variants":["Open 8B CareBot tops ChatGPT on medical benchmarks","CareBot: open 8B LLM beats ChatGPT in medical tests","Two-stage CPT helps 8B CareBot outscore ChatGPT","Bilingual CareBot, 8B, surpasses ChatGPT on med averages","Open-source CareBot edges ChatGPT on five med benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2603,"prompt_tokens":1024,"completion_tokens":1579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1491}},"tokens_in":640,"tokens_out":1579,"duration_ms":10626,"temperature":1.0,"reasoning_tokens":1491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:46.021583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of licensed clinicians blindly score CareBot-Chat and HuatuoGPT II on, say, 200 held-out multi-turn consultations from CMB-Clin using the same four dimensions; if clinicians do not prefer CareBot-Chat overall, the paper's attribution of its consultation advantage to the training recipe is not supported. A second check would re-run the pipeline once with human expert labels substituted for GPT-4 labels at a single stage and compare benchmark scores.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Deita complexity-quality scoring used to select single-turn SFT data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CMB benchmark and the CMB-Clin multi-turn evaluation protocol with its four rating dimensions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MedQA benchmark used to measure English medical concept knowledge."}],"review_version":1}