{"id":"c1418a88-d4e0-43b5-b6d1-bad3657a8ff0","arxiv_id":"2501.09484","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A dialogue-strategy-trained patient simulator improves realism in AI medical consultations and shows that inquiry quality and diagnostic skill jointly limit diagnostic accuracy.","lead":"Researchers built a patient simulator chatbot from real online doctor-patient dialogues, then used it to test AI doctors. They find that good questioning and good diagnosis are both needed: weak inquiry caps diagnostic accuracy even for strong models, and vice versa.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inquiry-diagnosis conclusion may reflect patient-simulator compatibility with GPT-4o rather than a general property of inquiry quality.","rationale":"The reader's weakest assumption concerns information access: the proposed simulator receives structured patient medical records while baselines may rely only on prompt engineering. That is a real evaluation concern, but Figure 1's caption states the illustrative comparison uses identical patient records, and Section 2.2's phrase 'during training' suggests the records are in the training prompt, not necessarily an inference-time advantage. The reader's specific confound is therefore not the most secure place to attack. The strongest load-bearing concern is the simulator-compatibility confound: the patient simulator is trained exclusively on GPT-4o-generated doctor turns, and the central inquiry-diagnosis claim is derived from comparing GPT-4o-family and Claude inquiry models through that same simulator. If the simulator responds less coherently to Claude's questioning style, the observed 'inferior inquiry quality' of Claude and the resulting Liebig's-law conclusion could be a byproduct of training distribution rather than a true property of the inquiry-diagnosis relationship. This concern is concrete, internal to the paper's methodology, and directly testable by varying the patient simulator or using real patient dialogues. Because the paper's second central claim requires that the inquiry-model ranking generalize beyond the specific patient model, the manuscript needs an additional control experiment before the Liebig-law claim can be accepted. This does not change the reader's CONDITIONAL verdict, but it identifies a different condition that should be met.","tokens_in":14510,"tokens_out":10417,"duration_ms":107781,"concrete_test":"Re-run the Section 3.2 experiment with the patient role played by (a) the proposed simulator, (b) AgentClinic's prompt-engineered patient given the same records, and (c) a held-out set of real doctor-patient transcripts, holding diagnosis (e.g., o1-preview) and number of rounds fixed. Compute diagnostic accuracy for GPT-4o, GPT-4o-mini, and Claude-3.5 inquiries. If Claude is not consistently lowest under (b) or (c), the Liebig conclusion is confounded by simulator compatibility. As a secondary check, measure the simulator's IRR separately for each inquiry model; a significantly higher IRR with Claude would indicate off-distribution patient behavior.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The downstream inquiry-diagnosis result may be an artifact of patient-simulator compatibility rather than a general inquiry-quality effect. Section 2.1 trains the patient simulator entirely on synthetic doctor-patient dialogues generated by GPT-4o through in-context learning, so the only doctor turns the patient model has seen are GPT-4o-style turns. Section 3.1 then pairs this simulator with three inquiry models (GPT-4o, GPT-4o-mini, Claude-3.5). If Claude phrases questions differently, as the paper itself documents in Figure 5, the simulator is off-distribution and may return less informative or less coherent responses, lowering the diagnostic accuracy of the resulting transcript. The paper attributes Claude's consistently lower accuracy to 'inquiry capability' and uses this cross-model ranking to support Liebig's law, but it never checks whether the ranking survives when the patient role is played by a different simulator or by real patient data. Without that control, the second central claim rests on a single, potentially biased patient model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a patient simulator for online medical consultations, trained by supervised fine-tuning on synthetic doctor-patient dialogues produced from real dialogue strategy flows extracted from MedDialog and from CCKS 2019 Chinese medical records. The authors report that their simulator has a lower hallucination rate and higher anthropomorphism than prompt-engineered baselines, then use the simulator in multi-round interactions with three inquiry models and five diagnostic models to study how inquiry quality affects diagnostic accuracy. They conclude that inquiry and diagnosis obey Liebig's law, i.e., each can be a binding constraint on final accuracy, and they propose a four-type taxonomy of physician inquiries to explain cross-model differences in inquiry behavior.","tokens_in":14722,"tokens_out":7540,"duration_ms":73091,"significance":"The topic is timely: most medical LLM evaluations focus on diagnosis with complete information and neglect the inquiry stage, and the paper proposes a concrete alternative to prompt-engineered patient agents. The authors release the LoRA weights, ground the training in real dialogue strategies from MedDialog, and offer a simple four-type inquiry taxonomy that could inform future interactive clinical benchmarks. These are genuine assets for the community. However, the headline simulator gains and the law-like inquiry-diagnosis conclusion currently rest on a confounded baseline comparison and on averages without variance estimates, and the cross-model inquiry ranking may reflect simulator compatibility rather than general inquiry ability. The significance is therefore conditional on addressing these evaluation gaps.","major_comments":[{"comment":"The simulator evaluation is confounded by information access. The proposed model is given the full patient medical record in its system prompt (Figure 7), whereas the AgentClinic and Qwen2.5-72B-Instruct baselines use only a core system prompt without the record. The authors themselves state in §2.2 that the improvement 'largely stems from incorporating patient medical records into the system prompt during training.' As a result, the reported hallucination-rate reduction (0.31% vs. 3.71%) and anthropomorphism advantage (0.87 vs. 0.31) do not isolate the contribution of the training paradigm; they could reflect the baselines' lack of access to the ground-truth patient record. Please add ablations in which the baselines receive the same medical record (or an equivalent scenario description) and re-report the three metrics.","section":"§2.2, Table 1, Appendix C"},{"comment":"All claims of 'significant differences' in inquiry capability and of diagnosis ceilings rest on only three runs with the average reported; no error bars, standard deviations, or significance tests are provided. For instance, at five inquiry rounds under o1-preview, the three inquiry models achieve accuracies of 0.439, 0.481, and 0.500; gaps of 0.019 to 0.061 accuracy points could easily be within sampling noise, especially with only three trials. Please report per-run values, confidence intervals, and an appropriate significance test, or temper the 'significant differences' language throughout §3.2.","section":"§3.2, Figure 3"},{"comment":"The Liebig's law conclusion is not demonstrated as a minimum-factor law. The experiments show main effects — more inquiry rounds generally improve accuracy, and stronger diagnosticians generally achieve higher accuracy — but they never test the predicted interaction, such as whether the benefit of a stronger diagnostician is smaller when inquiry quality is low (or vice versa). Moreover, inquiry quality is operationally defined by the very downstream diagnostic accuracy it is said to limit, so the statement that 'poor inquiry constrains accuracy' is close to a restatement of the ranking used to define inquiry quality. The four-type distribution analysis in §4.2 is independent evidence, but it is based on GPT-4o annotation with no reported annotation agreement. Please either define inquiry quality with an independent measure (e.g., information gain or coverage of required history items) or explicitly reframe the conclusion as an observed correlation between inquiry behavior and downstream accuracy rather than as a law.","section":"§3.2, §4.2"},{"comment":"The patient simulator is trained exclusively on synthetic doctor-patient dialogues generated by GPT-4o through in-context learning (Figure 2), and the only doctor turns it has seen are GPT-4o-style turns. Section 3.1 then pairs this simulator with Claude-3-5-sonnet as an inquiry model. Because the paper itself documents that Claude asks questions with a different type distribution (Figure 5), the Claude interactions are likely off-distribution for the simulator, so the low diagnostic accuracy of Claude-generated records in Figure 3 may reflect simulator incompatibility rather than a generally poorer inquiry capability. Please add a control in which the same inquiry models interact with a different patient simulator, or with fixed real-patient responses, and check whether the ranking of inquiry models survives.","section":"§2.1, §3.1"},{"comment":"The paper does not specify the language or format of the AgentClinic MedQA-Extend medical records used in the inquiry-diagnosis experiments, nor how they were adapted for a simulator that, per §2.1 and the Limitations section, was trained only on Chinese CCKS records and is limited to Chinese dialogues. If MedQA-Extend records are in English, the simulator is being applied out of its declared language scope and all accuracy numbers in Figure 3 would need re-evaluation. Please clarify the language, source, and preprocessing of these records, and confirm that the simulator's Chinese-only limitation does not apply to the experimental setup.","section":"§3.1"}],"minor_comments":[{"comment":"The Anthropomorphism Score is defined in §2.2 as being 'on a scale from 0 to 1,' but Table 1 reports values of 28.00, 31.00, and 87.00. Please state explicitly whether these are percentages or values on a different scale, and make the table consistent with the definition.","section":"Table 1"},{"comment":"The caption states that each experiment is conducted three times and the average is reported, but it does not say whether the three runs use the same patient records or resample records. Please clarify and, if possible, show per-run variability in the figure.","section":"Figure 3"},{"comment":"The patient dialogue strategy label 'Disagree' appears twice in the list. It should likely appear once.","section":"Appendix A"},{"comment":"The reference 'Herhert M Adler' contains a typo; the correct spelling is 'Herbert M. Adler.'","section":"References"},{"comment":"The description of the SFT dataset (1000 multi-turn dialogues, 8:2 split) is brief; please clarify whether any held-out test dialogues were used in the §2.2 evaluation or whether the evaluation was performed on newly generated interactions, and report the number and source of the evaluation dialogues.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CL and the authors are transparent about the system-prompt confound in §2.2, which makes me hopeful that the evaluation can be repaired. The more serious concern is the cross-model inquiry ranking in §3, which needs either an alternative patient simulator or a real-patient control to rule out compatibility effects, and the Liebig's law claim, which needs an interaction analysis or a reframing. I recommend major revision rather than rejection because the proposed fixes are concrete and the underlying resources (weights, strategy flows, taxonomy) are useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, it is a genuinely new attempt to build a patient simulator by learning dialogue strategies from real doctor-patient exchanges, rather than relying on prompt engineering. The authors release the weights, and the simulator produces more anthropomorphic, lower-hallucination behavior than the baselines they compare against. Second, the headline evaluation is confounded, and the authors admit it: their model gets the patient's medical record in the system prompt while the AgentClinic and Qwen baselines do not. So the large HR and AS gaps cannot be cleanly attributed to their training paradigm—they may just reflect who has the chart.\n\nWhat the paper does well: the idea of extracting a \"dialogue strategy flow\" from real consultations and using that to synthesize training data is sensible and original. The four-type inquiry taxonomy is a useful descriptive lens, and the observation that Claude's inquiries skew toward symptom specification is a nice case study. The authors are also unusually candid: they state the record-access confound themselves and list limitations (Chinese-only, no follow-up visits, no multimodal).\n\nWhere it gets shaky: the \"Liebig's law\" conclusion rests on coupling their simulator with three inquiry models, one being Claude. But the simulator was trained exclusively on GPT-4o-generated doctor turns. When Claude phrases questions differently, the patient model is off-distribution, and that alone could lower downstream diagnostic accuracy. The paper never tests with a second simulator or with real patient data, so the cross-model ranking may be an artifact of simulator compatibility rather than a general property. The law is also somewhat circular: they define inquiry quality by diagnostic performance under strong diagnosticians, then use that same data to assert that poor inquiry limits diagnosis. On top of that, the accuracy gaps between inquiry models are small (0.44 vs 0.48 vs 0.50) and come from only three runs with no error bars or significance testing. That is thin support for a law.\n\nBottom line: I would cite this paper for the simulator and the taxonomy, and I would send it to a serious referee—the idea deserves a hearing. But the evaluation needs redoing: give the baselines the same patient record, test the simulator with non-GPT-4o inquiry models, and report variance. As it stands, the two strong claims (simulator superiority and Liebig's law) are both conditionally true at best.","headline":"A promising patient simulator undermined by a confounded comparison and a downstream claim that may just reflect simulator-model compatibility.","tokens_in":15190,"tokens_out":3161,"would_cite":true,"duration_ms":30071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A patient simulator trained on real dialogue shows that inquiry quality caps AI diagnostic accuracy, no matter how strong the diagnostician.","keywords":["patient simulator","medical dialogue","large language models","inquiry-diagnosis relationship","Liebig's law","dialogue strategy","hallucination rate","anthropomorphism"],"falsifier":"Run a controlled rerun where AgentClinic and Qwen2.5-72B-Instruct receive the same patient medical record in their system prompts as the proposed simulator does; the training-paradigm claim predicts the simulator still beats them on hallucination rate and anthropomorphism by roughly the same margin. Separately, find any inquiry-diagnosis pair where a model with clearly poorer inquiry (as the paper defines it for Claude) yields higher accuracy than a better inquiry model under the same strong diagnostician, or where a weak diagnostician with excellent inquiries beats a strong diagnostician with poor inquiries — that would violate Liebig's law.","tokens_in":14365,"feed_emoji":"🩺","tokens_out":2746,"duration_ms":27708,"temperature":0.7,"pith_summary":"This paper argues that prompt-engineered patient agents fail to mimic real patients, and that a patient simulator trained on dialogue strategies extracted from real doctor-patient conversations behaves more realistically and hallucinate less. Using this simulator as a fixed patient, the authors test how well different large language models gather information and then diagnose, and they claim the two stages obey Liebig's law: the weaker of inquiry quality and diagnostic capability sets the ceiling on final accuracy. They also categorize doctors' questions into four types and show that differences in how models distribute their questions help explain why some models diagnose worse than others. If the claims hold, future online-consultation AI should be optimized for inquiry, not just diagnosis.","feed_headline":"Weak questions cap AI diagnosis, no matter the doctor model","feed_subtitle":"A patient simulator trained on real dialogue shows inquiry and diagnosis obey Liebig's law of the minimum.","key_machinery":"The central object is the trained patient simulator, built by (1) annotating real doctor-patient dialogues with dialogue strategy tags, (2) manually curating high-quality strategy flows, (3) synthesizing doctor-patient dialogues from Chinese medical records and randomly sampled strategy flows, and (4) supervised fine-tuning Qwen2.5-72B-Instruct with LoRA on those dialogues. The simulator runs with only a system prompt containing the patient record, so it must learn to produce realistic patient turns — emotions, proactive questions, occasional refusal to answer — without being told the strategy tags at inference time. This fixed patient then plays the role of the controlled variable in the inquiry-diagnosis experiments.","core_discovery":"The central discovery is that a patient simulator fine-tuned on synthesized dialogues — generated from real medical records and dialogue strategy flows tagged from authentic consultations — achieves a hallucination rate of 0.31% versus 3.71% for a prompt-engineered baseline, and an anthropomorphism score of 0.87 versus 0.31. The paper then uses this simulator as a fixed patient to generate inquiry records from several doctor models and diagnose them with several others, showing that models with poor inquiry consistently yield low accuracy even with the strongest diagnostic model, and models with weak diagnosis stay low even with good inquiries. The authors conclude that inquiry and diagnosis follow Liebig's law of the minimum: insufficient inquiry constrains diagnostic effectiveness regardless of diagnostic skill, and vice versa.","pith_inferences":["The reported advantage of the simulator over baselines may depend on the fact that the simulator receives the patient's medical record in its system prompt while the baselines do not; a fair comparison would give the same record to the baselines before attributing the difference to the training paradigm.","The higher 'irrelevant response rate' of the simulator (4.79% versus 0.93%) may actually be a realistic feature rather than a flaw, since real patients sometimes refuse to answer; if so, benchmarks should stop treating irrelevance as strictly undesirable.","One testable extension is to use the simulator as a training environment for doctor models, where the doctor must learn to ask the right mix of questions — a direct test of whether the four-type inquiry distribution can be shaped by reinforcement learning.","The Liebig's-law claim suggests a measurable benchmark property: for a fixed set of patient records, diagnostic accuracy should be approximately the minimum of an inquiry-quality score and a diagnostic-capability score; this could be checked on new models without running a full dialogue study."],"forward_implications":["Benchmarks that evaluate AI doctors through simulated consultations will be more trustworthy if the patient side can hold back information and act emotionally, rather than dumping all symptoms upfront.","Diagnostic accuracy in online consultations can be improved by training or selecting models specifically for the inquiry stage, since inquiry quality is an independent binding constraint.","Because model inquiry ability and diagnostic ability appear uncorrelated, a system that separates the questioning role from the diagnosing role could outperform a single generalist model.","The allocation of inquiry questions across chief complaints, symptom specification, accompanying symptoms, and history-taking within 3 to 5 rounds is a concrete optimization target for future medical AI.","The simulator itself can generate realistic synthetic doctor-patient dialogues at scale, which could serve as training data for other medical dialogue systems."],"supporting_citations":[{"why":"Provides the MedDialog dataset of real doctor-patient conversations from which dialogue strategies are extracted.","marker":"(Zeng et al., 2020)"},{"why":"Supplies the CCKS 2019 Chinese medical record set used as the patient-record source for synthesizing training dialogues.","marker":"(Han et al., 2020)"},{"why":"AgentClinic is the prompt-engineered patient-agent baseline the simulator is compared against, and its MedQA-Extend provides patient records for the inquiry-diagnosis experiments.","marker":"(Schmidgall et al., 2024)"},{"why":"Qwen2.5-72B-Instruct is the base model that the patient simulator is fine-tuned from and one of the baseline patient agents.","marker":"(Yang et al., 2024)"},{"why":"LoRA is the parameter-efficient fine-tuning method used to train the simulator weights.","marker":"(Hu et al., 2021)"},{"why":"GPT-4o is used for dialogue strategy annotation, synthetic dialogue generation, metric scoring, and inquiry-type annotation.","marker":"(OpenAI, 2024a)"}],"fun_headline_variants":["Weak questions cap AI diagnosis, no matter the model","Patient simulator shows poor inquiry caps diagnosis","Inquiry limits diagnosis: Liebig's law in medical AI","Bad questions doom diagnosis, even with top doctor models","Inquiry quality gates diagnosis: patient simulator proof"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison that establishes the simulator's superiority gives the proposed model the patient's medical record in the system prompt while the baselines do not; if the baselines were also given that record, the hallucination and anthropomorphism gaps could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Weak questions cap AI diagnosis, no matter the model","Patient simulator shows poor inquiry caps diagnosis","Inquiry limits diagnosis: Liebig's law in medical AI","Bad questions doom diagnosis, even with top doctor models","Inquiry quality gates diagnosis: patient simulator proof"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00045,"raw_usage":{"total_tokens":2244,"prompt_tokens":898,"completion_tokens":1346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1272}},"tokens_in":514,"tokens_out":1346,"duration_ms":10784,"temperature":1.0,"reasoning_tokens":1272,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:57:46.777697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled rerun where AgentClinic and Qwen2.5-72B-Instruct receive the same patient medical record in their system prompts as the proposed simulator does; the training-paradigm claim predicts the simulator still beats them on hallucination rate and anthropomorphism by roughly the same margin. Separately, find any inquiry-diagnosis pair where a model with clearly poorer inquiry (as the paper defines it for Claude) yields higher accuracy than a better inquiry model under the same strong diagnostician, or where a weak diagnostician with excellent inquiries beats a strong diagnostician with poor inquiries — that would violate Liebig's law.","supporting_citations":[],"review_version":1}