{"id":"90d7cbb7-8e2b-4dc9-a69c-a5ba4ff92d4a","arxiv_id":"2505.12888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GAP constructs a patient-centric graph from dialogue and generates knowledge-grounded prompts, achieving 39.28 Jaccard on DialMed, above LLM-based baselines but below the supervised DDN model.","lead":"The paper introduces GAP, a framework that builds a patient-specific medical graph from a doctor-patient dialogue and uses it to create prompts from medical knowledge bases, improving the medications an LLM recommends. For general readers, it shows a practical way to reduce dangerous drug recommendations in conversational medical AI by making the model attend to conditions like pregnancy or bronchitis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety-critical patient-state extraction (pregnancy, blood pressure, age) is not specified in Eqs. (1)-(2) and is never evaluated, so the graph and safety claims depend on an unverified component.","rationale":"The paper's central claim is that GAP reduces non-factual or unsafe medication recommendations by building a patient-centric graph and grounding prompts in external knowledge. For this to hold, every medically relevant state must be faithfully extracted. The reader identified extraction reliability as the weakest assumption; I agree and sharpen it: the extraction pipeline described in Eqs. (1)-(2) and Tables 4-5 does not even specify how patient characteristics such as pregnancy are extracted, despite Appendix A.1 listing them as graph components. This is an internal gap, not just an unmeasured quantity. If the motivating example cannot be handled by the published prompts, the framework as written cannot deliver its advertised safety benefit. The proposed test directly checks both the specification gap and the empirical accuracy. I also note the base-model inconsistency (DeepSeek-V3 in Section 4.2 vs ChatGPT-3.5-Turbo in A.3) as a secondary reproducibility concern, but the extraction gap is more load-bearing because it affects the conceptual validity of the method, not just the reproducibility of numbers. Since the paper could be fixed with an explicit extraction description and evaluation, a conditional verdict remains appropriate.","tokens_in":13760,"tokens_out":5361,"duration_ms":51913,"concrete_test":"Run the exact prompts in Tables 4-5 on the Figure 1 dialogue; check whether 'pregnant' or '30 weeks pregnant' is returned. If not, Eqs. (1)-(2) cannot construct the patient-centric graph as illustrated, confirming that an undocumented patient-characteristic extractor is required. Then annotate 100 randomly sampled DialMed dialogues for pregnancy, age, and blood-pressure mentions and compute recall of the full GAP pipeline for these states; report extraction F1. If recall is below 95%, the framework's safety claim is not supported by evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines extraction as two LLM calls: Eq. (1) extracts medical concepts (disease, symptom, medication) and Eq. (2) extracts slot-value pairs for each concept, with slots defined per concept category. Appendix A.1 then states that patient characteristics (gender, age, blood pressure, pregnant state) are extracted as part of the patient-centric graph. No equation, prompt, or instruction for this patient-characteristic extraction appears in the paper; Tables 4-5 show only disease/symptom/medication extraction and disease/symptom state judgment. If pregnancy is captured by a separate, undocumented LLM call, its accuracy is never measured anywhere. The Figure 1 motivating example requires recognizing '30 weeks pregnant' to avoid recommending Losartan; a single missed state directly contaminates the patient-centric graph, neighborhood prompts, and path-based prompts. The paper reports no extraction recall or F1 on DialMed or any other corpus, so the central mechanism by which GAP claims to reduce non-factual or unsafe responses is unverified. This makes the safety and accuracy improvements attributable to an unmeasured upstream component rather than to the graph-assist design itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GAP, a framework for dialogue-based medication recommendation. GAP extracts medical concepts and their states from a dialogue using LLM-based two-stage extraction, builds a patient-centric graph, and generates neighborhood prompts and path-based prompts by querying external knowledge sources (CMeKG, Disease-KB, LLM reasoning, and Internet search). The framework is evaluated on the DialMed medication recommendation dataset and on a diagnostic interviewing task using an LLM-as-Patients simulator, reporting Jaccard/F1 improvements over LLM baselines and a diagnostic score of 0.63. The paper includes ablations showing the contribution of each prompt type and an error analysis.","tokens_in":14055,"tokens_out":9837,"duration_ms":87130,"significance":"If the results hold, GAP provides a practical plug-and-play method for grounding LLM-based medication recommendations in an explicit patient-specific representation, with consistent gains over KG-RAG and few-shot CoT prompting on DialMed, and it demonstrates a promising direction for integrating structured dialogue memory with heterogeneous knowledge sources. The ablation study is well-designed and shows clear contributions from both neighborhood and path-based prompts. The main contributions are the patient-centric graph construction and the multi-source prompt generation. However, the central mechanism depends on upstream extraction components that are not specified or evaluated, and the reported configuration contains a direct inconsistency about the underlying LLM, so the current evidence is insufficient to support the paper's safety-related claims.","major_comments":[{"comment":"The extraction equations (1)-(2) and the prompts in Tables 4-5 cover medical concepts and their states, but the patient-characteristic extraction (gender, age, blood pressure, pregnancy) that Appendix A.1 lists as part of the patient-centric graph is never specified with a prompt, an equation, or an evaluation. The motivating safety example in Figure 1 and the path-based prompts in Figure 3 depend on detecting pregnancy correctly; a missed pregnancy directly contaminates the graph, neighborhood prompts, and path-based prompts. The paper reports no extraction precision, recall, or F1 on DialMed or any corpus, so the mechanism claimed to reduce unsafe recommendations is unverified.","section":"§3.2, Appendix A.1, Table 5"},{"comment":"The foundation LLM is inconsistent. Section 4.1 states 'We use DeepSeek-V3 [4] as the foundation LLM' and Section 4.2 says DeepSeek-V3 is utilized for extraction and prompt generation, while Appendix A.3 states 'We use ChatGPT-3.5-Turbo (version 1106) for the experiments.' This is a direct contradiction that affects the reproducibility and interpretation of all reported results; the authors must clarify which model was used for which component and reconcile the two statements.","section":"§4.1, §4.2, Appendix A.3"},{"comment":"The paper reports means and standard deviations over five runs but no statistical significance tests (e.g., paired bootstrap or t-test). The central claim that 'GAP outperforms other LLM-based baselines on all department cases' (Section 4.3) requires statistical validation, especially for comparisons where the gaps are smaller (e.g., Respiratory Jaccard: GAP 29.33 vs. KG-RAG 27.48). Please report significance tests or confidence intervals.","section":"Tables 1 and 3"},{"comment":"The abstract claims that GAP 'retrieving information from multiple sources to reduce the non-factual responses.' However, the evaluation only measures medication-recommendation accuracy (Jaccard/F1) and a diagnostic score; it does not measure factual correctness or hallucination in the generated text. The error analysis (Figure 4) is conducted on CoT errors, not on GAP errors. Thus, the specific benefit of 'non-factual' reduction is not demonstrated; please provide direct evidence or temper the claim.","section":"Abstract, §4.3"},{"comment":"The linking of extracted mentions to KG entities is described as 'simple linking methods (e.g., edit distance, synonym list)' but its accuracy is never measured. Since neighborhood prompts are retrieved via this linking, incorrect normalization will propagate to the retrieved knowledge. Please evaluate the linking component or discuss its failure modes in the context of the safety claims.","section":"§3.3"}],"minor_comments":[{"comment":"The sentence 'The context sliding window size of extraction is set to 4 (i.e., k = 1 in Equation 2), and set to ∞ in diagnostic task and the recommendation task, respectively' is ambiguous; please clarify which value applies to which task and how the window size relates to k.","section":"§4.2"},{"comment":"The diagnostic evaluation relies on the authors' own LLM-as-Patients framework [16] and LLM-based scoring; please discuss the potential for self-evaluation bias and provide details on the agreement of the five LLM evaluators.","section":"§4.1, Table 2"},{"comment":"The error analysis in Figure 4 is conducted on CoT errors; an error analysis of GAP itself would help identify remaining failure modes for this framework.","section":"§4.3"},{"comment":"The prompt in Table 7 uses the term 'Respirator' for the department; this should be 'Respiratory'.","section":"Table 7"},{"comment":"The hyperparameters k1, k2, and the context window size are set heuristically; a sensitivity analysis would strengthen the robustness of the reported results.","section":"§3.4"},{"comment":"The path schema S is said to be defined by medical professionals, but no details about the panel, their instructions, or inter-annotator agreement are provided; please add these details.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper uses the authors' own LLM-as-Patients framework for the secondary diagnostic evaluation; this is not a blocker, but the authors should be transparent about potential bias. The main concerns are the unverified extraction components and the LLM inconsistency, both of which are fixable in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: GAP is a clean integration of known pieces—LLM extraction, patient-centric graph, KG/LLM/internet prompts—that consistently beats KG-RAG on DialMed and makes a plausible safety argument. But the paper asks the reader to trust the extraction step that carries the whole safety claim, and never measures it.\n\nWhat's new: the components are not new, but the patient-centric graph as a cross-turn memory, plus two prompt families, is a sensible integration. The ablations in Table 3 are the strongest evidence: removing both prompts drops F1 from 54.27 to 36.24, and each prompt contributes independently. The case study with pregnancy and loratadine is exactly the right illustration.\n\nSoft spots: the safety-critical part—extracting patient characteristics like pregnancy, age, and blood pressure—is described in Appendix A.1 but no prompt or equation covers it. Eq. (2) extracts slots for disease, symptom, and medication states; patient characteristics are just asserted to be extracted. So the Figure 1 scenario, where missing \"pregnant\" would produce a contraindicated Losartan recommendation, hinges on an unevaluated LLM call. No extraction F1 or recall is reported anywhere. That is not a minor omission; it is the load-bearing step for the safety claim.\n\nAlso, Section 4.1 names DeepSeek-V3 as the foundation LLM, while Appendix A.3 says ChatGPT-3.5-Turbo (version 1106) was used. These cannot both be right. No code or data is released, so reproducibility is limited. The diagnostic interviewing task uses the authors' own LLM-as-Patients framework and LLM evaluators, so the 0.63 score is suggestive but self-referential.\n\nMinor: no significance tests, though the error bars in Table 1 are small enough that the headline improvement over KG-RAG is plausibly real. The comparison to supervised methods is fair, and the abstract's \"competitive\" claim is accurate.\n\nWho this is for: people building LLM-based medical dialogue systems, especially with RAG over knowledge graphs. The framework is plug-and-play and could be adapted broadly. It deserves a serious referee, because the core idea has merit and the ablations are informative, but it needs substantial revision: specify and evaluate the extraction step, resolve the model inconsistency, and release artifacts.\n\nRecommendation: send to peer review, but reviewers should demand extraction evaluation and artifact release before acceptance.","headline":"GAP is a promising integration of graph-construction over LLM extraction with KG/LLM/internet prompts for medication recommendation, but the unevaluated extraction of pregnancy-related states is a load-bearing gap that blocks acceptance as-is.","tokens_in":14531,"tokens_out":1846,"would_cite":false,"duration_ms":18786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph-assisted prompts lift LLM medication-recommendation accuracy by encoding the patient's dialogue as an explicit graph, then querying external knowledge along physician-defined paths.","keywords":["dialogue-based medication recommendation","patient-centric graph","graph-assisted prompting","knowledge graph retrieval","retrieval-augmented generation","large language models","medical dialogue systems","drug contraindication safety"],"falsifier":"Run the extraction step alone on the DialMed dialogues and compare its concept and state recall against a clinician-verified gold set, especially for states such as pregnancy, allergy history, and past medication use. If extraction recall is substantially below 1.0, or if a dialogue whose gold recommendation changes with the state 'pregnant' produces the same graph with and without that state, then GAP's safety advantage would fail exactly where the paper claims it.","tokens_in":13585,"feed_emoji":"💊","tokens_out":5757,"duration_ms":55554,"temperature":0.7,"pith_summary":"GAP sets out to prove that a doctor-side large language model can recommend medications more accurately and safely if it first turns the patient's dialogue history into an explicit graph of medical concepts and their states, and then uses that graph to query external medical knowledge. The paper claims that across multi-turn dialogue, plain LLM prompting loses fine-grained facts such as pregnancy or bronchitis history, so the graph acts as a structured memory that keeps those facts in view. On the dialogue-based medication benchmark, GAP raises Jaccard from 30.15 (a knowledge-graph RAG baseline) to 39.28 and F1 from 42.44 to 54.27, and in simulated diagnostic interviews it scores 0.63 versus 0.51 for a strong general LLM.","feed_headline":"Graph memory lifts LLM medication recommendation accuracy","feed_subtitle":"A patient-centric graph plus multi-source retrieval raises DialMed Jaccard to 39.28 and F1 to 54.27.","key_machinery":"The patient-centric graph $G_p$: a graph whose root is a patient node, with edges to normalized medical concepts (disease, symptom, medication) and directed edges from each concept to its slot-value states (e.g., treatment effective, past medical history). It is constructed from LLM-extracted concepts and states across dialogue turns, and it is then joined with an external medical knowledge graph to form a neighborhood graph $N_{G_p}$. From this structure GAP generates neighborhood prompts (top-1 relation-selected facts from $N_{G_p}$) and path-based prompts (paths matched against pre-defined medical schemas, converted into queries for KG verification, LLM reasoning, and Internet search). The graph and prompts are concatenated with the dialogue history as the final LLM input.","core_discovery":"The central claim is that dialogue-based medication recommendation improves when the LLM is prompted not only with the raw dialogue but with a patient-centric graph built from extracted medical concepts (diseases, symptoms, medications) and their attributed states (pregnancy, duration, past medical history), plus two kinds of derived prompts: neighborhood prompts that pull nearby facts from an external medical knowledge graph, and path-based prompts that query the knowledge graph, the LLM itself, and the Internet along physician-defined schemas. GAP reports that this combination outperforms existing LLM-based recommendation methods on the DialMed benchmark with Jaccard 39.28 and F1 54.27, and improves diagnostic interviewing information density (0.19) and score (0.63). The paper presents this as evidence that explicit graph memory plus multi-source retrieval can reduce non-factual and contraindicated medication responses.","pith_inferences":["The safety claim rests entirely on the LLM extraction step: the paper never measures extraction accuracy, so if a state like 'pregnant' is missed, the graph omits it and a contraindicated drug can still be recommended; a testable extension is to measure concept and state extraction recall separately and to inject a global state summary from an independent extractor.","The graph is a lossy textual summary of the dialogue; replacing the linearized triple string with a graph neural network, or storing uncertainty about extracted states, could improve robustness to extraction noise.","Because results are reported only on a Chinese dialogue dataset with a specific set of Chinese medical knowledge sources, the framework's portability to English sources and other knowledge graphs is open; a replication with English KGs would test whether the benefit is language- or KG-dependent.","The diagnostic-interviewing gain is a single aggregate score; decomposing it into aspects and information catches would show whether GAP's improvement comes from asking more complete questions or from eliciting more key items from the simulated patient."],"forward_implications":["If GAP's gains hold, any black-box LLM can be turned into a more reliable recommender by adding a dialogue-derived graph plus retrieval prompts, with no retraining or parameter updates.","Ablation results show that both prompt types matter: removing neighborhood prompts drops F1 by about 19%, and removing path-based prompts drops F1 by about 11%, so the retrieval design carries much of the benefit.","The path schema is modular: the paper notes it can be re-targeted from acquiring treatments to suitable food, medical tests, or lifestyle advice, so the same architecture extends to other dialogue-based health services.","Because GAP is orthogonal to training-time methods, it can be stacked on top of fine-tuned medical LLMs and chain-of-thought prompting, potentially compounding their gains."],"supporting_citations":[{"why":"Supplies the DialMed dataset and the DDN baseline for dialogue-based medication recommendation, which GAP is evaluated against.","marker":"[10]"},{"why":"Provides the LLM-as-Patients simulated diagnostic interviewing evaluation and its scoring metric, used for GAP's second task.","marker":"[16]"},{"why":"Serves as the KG-RAG baseline that GAP compares against and as the design source for knowledge-graph-enhanced prompt generation.","marker":"[27]"},{"why":"Introduces the retrieval-augmented generation paradigm that GAP's multi-source prompt retrieval builds on.","marker":"[15]"},{"why":"Provides the chain-of-thought prompting baseline used in the medication-recommendation experiments.","marker":"[32]"},{"why":"Motivates the filtering of retrieved neighborhood facts to avoid irrelevant context, a key design choice in GAP's neighborhood prompts.","marker":"[36]"}],"fun_headline_variants":["GAP framework uses graphs to boost dialogue medication recs","Graph-assisted prompts improve LLM medication recommendations","Patient-centric graphs cut non-factual medication suggestions","Multi-source retrieval plus graph memory lifts med rec accuracy","GAP: graphs and external knowledge improve dialogue med recs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM extraction step catches all medically relevant concepts and states in the dialogue, since the paper never measures extraction accuracy and a missed state like pregnancy drops out of the graph entirely.","fun_headline_variants_meta":{"raw":{"variants":["GAP framework uses graphs to boost dialogue medication recs","Graph-assisted prompts improve LLM medication recommendations","Patient-centric graphs cut non-factual medication suggestions","Multi-source retrieval plus graph memory lifts med rec accuracy","GAP: graphs and external knowledge improve dialogue med recs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1487,"prompt_tokens":994,"completion_tokens":493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":417}},"tokens_in":610,"tokens_out":493,"duration_ms":5460,"temperature":1.0,"reasoning_tokens":417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:24:24.106836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the extraction step alone on the DialMed dialogues and compare its concept and state recall against a clinician-verified gold set, especially for states such as pregnancy, allergy history, and past medication use. If extraction recall is substantially below 1.0, or if a dialogue whose gold recommendation changes with the state 'pregnant' produces the same graph with and without that state, then GAP's safety advantage would fail exactly where the paper claims it.","supporting_citations":[{"cited_title":"In: ICCL","cited_arxiv_id":null,"evidence_quote":"Supplies the DialMed dataset and the DDN baseline for dialogue-based medication recommendation, which GAP is evaluated against."},{"cited_title":"In: NeurIPS (2020)","cited_arxiv_id":null,"evidence_quote":"Introduces the retrieval-augmented generation paradigm that GAP's multi-source prompt retrieval builds on."},{"cited_title":"In: NeurIPS (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought prompting baseline used in the medication-recommendation experiments."}],"review_version":1}