REVIEW 4 major objections 6 minor 28 references
DoPI: Doctor-like Proactive Interrogation LLM for Traditional Chinese Medicine
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read DoPI separates asking questions from making a diagnosis, and reports 84.68% accuracy on TCM consultations.
desk verdict The architecture is sensible, but the 84.68% accuracy rests on a circular evaluation that reads the label out of the knowledge graph. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the TCM knowledge graph, whose nodes are symptoms and diseases and whose edges carry weights $w_{ki}$ representing how strongly symptom $k$ indicates disease $i$. The guidance model maps the patient's words to symptom nodes; the graph computes a cosine similarity $S_i = D_i \cdot P / (\|D_i\| \|P\|)$ between the patient's symptom vector and each disease's symptom vector, uses those scores to rank candidate diseases, then scores every unasked symptom by $\text{Score}(j) = \sum_i w_{ji} S_i$ and asks about the top two or three. A decaying Gaussian perturbation term is added to the symptom scores for exploration, and the dialogue stops when the top disease's cosine similarity passes a threshold $\epsilon$, at which point the expert model is invoked.
What would settle it
Run DoPI on a held-out set of real TCM consultations in which patients give incomplete, mistaken, or extra symptoms; if the accuracy falls close to the general-baseline level, the reported 84.68% reflects the synthetic test rather than genuine interrogation skill.
Extended reading notes
Core claim
The paper's central claim is that proactive interrogation can be decomposed and delegated: a guidance model asks questions, a knowledge graph ranks which questions matter, and an expert model diagnoses. The authors argue that this two-model architecture avoids the 'data conflict' seen when a single model is fine-tuned on both medical knowledge and multi-turn dialogues, so DoPI keeps professional TCM knowledge while gaining the ability to gather incomplete patient information through purposeful questioning. On the constructed dataset of more than two thousand doctor-patient dialogues, DoPI reaches 84.68% diagnostic accuracy, with a Q&A ratio of 21.31 and an interrogation distance of 1.84, beating general large language models and existing medical chatbots on all three metrics. The paper further claims that this evaluation method, built from synthetic dialogues rather than manually collected consultations, can assess interrogation capability without expensive real-world data collection.
Load-bearing premise
The knowledge graph and the test dialogues appear to be built from the same disease-symptom data, so the high accuracy may simply mean the model recovers labels used to generate the test set.
Editorial extensions
If this is right
- A medical LLM can conduct purposeful multi-turn interrogation without being fine-tuned on dialogue data, preserving its domain expertise.
- Knowledge-graph-guided questioning can outperform general-purpose LLMs on diagnostic accuracy in a specialized domain like TCM.
- The proposed evaluation methodology offers a low-cost template for measuring proactive interrogation skills without manually collected consultation transcripts.
- The two-model split allows the question-selection component and the diagnostic component to be upgraded or replaced independently.
Reading between the lines
- Editorial extension: the knowledge graph and the test dialogues appear to be built from the same disease-symptom data, so the 84.68% accuracy likely overstates performance on real consultations unless an independent test set confirms it.
- Editorial extension: the same two-model architecture could transfer to other specialties where diagnostic criteria are well defined, such as Western primary care or veterinary triage, if a reliable symptom-disease knowledge graph exists.
- Editorial extension: the 'honest patient' assumption is the largest gap between the synthetic setting and reality; an obvious test is to make simulated patients misreport or omit symptoms and see how much accuracy drops.
- Editorial extension: the tongue-coating image input is described but never evaluated in the experiments, so a direct test would compare accuracy with and without that visual modality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DoPI, a TCM diagnostic system in which a small 'guidance' LLM conducts multi-turn proactive questioning using edge weights from a disease-symptom knowledge graph, and a separate 'expert' model (Sunsimiao-based) provides the final diagnosis and treatment plan. The authors construct a synthetic multi-turn doctor-patient dialogue dataset by having qwen-plus simulate consultations from 'disease + symptom list' tuples, under explicit assumptions that patients are honest and accurate. They evaluate DoPI against Qwen2.5-Max, ChatGPT-4o, DeepSeek-v3, and several TCM models, reporting 84.68% diagnostic accuracy for DoPI-7B versus 58.74% for DeepSeek-v3, together with Q&A-ratio, interrogation-distance, and LLM-as-judge metrics. The central claim is that this demonstrates a significant improvement in proactive interrogation and diagnostic accuracy for TCM.
Significance. If the 84.68% accuracy figure were valid, the paper would make a useful architectural contribution: decoupling the questioning model from the expert model and grounding questions in a knowledge graph is a sensible way to avoid the expertise loss that often accompanies multi-turn fine-tuning. The pseudocode in Algorithm 1 and the explicit definitions of symptom vectors, cosine similarity, and importance scoring (Eqs. 1-5) give a concrete, implementable method. The authors are also transparent about the 'honesty' assumption in Section 4. However, the evaluation as presented does not support the headline claim: the knowledge graph and the synthetic test dialogues are built from the same disease-symptom associations, and no independent validation, confidence intervals, or real-patient data are provided. No code or dataset is released, which further limits verification.
major comments (4)
- [§3.2, §4, §5.1, Eq. (3)] The evaluation is circular. Section 3.2 states that the knowledge graph is built from 'existing data on TCM disease types and corresponding symptoms' with edge weights based on frequency, while Section 4 generates the test dialogues from 'binary tuples of disease + symptom list' drawn from the same kind of disease-symptom associations. Algorithm 1 makes the final selection by cosine similarity (Eq. 3) against disease vectors in that same knowledge graph, and Section 5.1 evaluates on 'the previously mentioned dataset' without any described split that separates test tuples from the data used to initialize the KG. Consequently, the 84.68% accuracy is consistent with retrieving the generative label rather than with diagnostic skill; the authors must evaluate on an independent test set, or at minimum on a held-out set of disease-symptom pairs that demonstrably did not inform the KG weights, with the KG construction and test generation separated.
- [§4] The 'Honesty' rule states that the simulated patient 'can accurately judge the symptoms inquired by the doctor and will not report any non-existent symptoms.' This assumption removes exactly the uncertainty, omission, and misreporting that the introduction identifies as the motivation for proactive interrogation. As a result, the reported accuracy cannot be interpreted as performance under realistic consultation conditions, and the paper's claim that the synthetic dialogues 'simulate a comprehensive real-world clinical scenario' is contradicted by its own modeling assumption. The authors should re-run the evaluation with noisy, incomplete, or confabulated patient responses, or validate on real consultation transcripts.
- [§5.2, Eq. (8)] The 'Interrogation Distance' metric compares the model's question and answer counts with Qd_i and Ad_i attributed to 'the specialized physician,' but these reference counts come from the LLM-generated dialogues of Section 4, not from professional TCM doctors. The metric therefore measures agreement with a synthetic simulation, not closeness to genuine clinical interrogation behavior, so the claim that DoPI's questioning is 'aligned with that of a professional physician' is unsupported by the evidence presented.
- [§5.2, Table 3] All results are reported as point estimates from what appears to be a single evaluation run, with no confidence intervals, significance tests, or error bars. Given the synthetic, circular test set and the stochasticity of both the dialogue generation and the LLM-as-judge evaluation, the single 84.68% figure cannot be taken as a reliable estimate of system performance; the authors should report variance across runs and, if possible, split-half or bootstrap statistics.
minor comments (6)
- [Algorithm 1] There are typographical issues: 'Konwledge' should be 'Knowledge,' 'ask epochs' is used without definition, and 'M odel U pdate' appears to be a line-wrapping artifact that should be cleaned up.
- [Eq. (5) and Table 2] The symbol epsilon is used both for the diagnosis threshold in Table 2 and for the Gaussian noise term in Eq. (5); rename the noise term (e.g., xi) to avoid ambiguity.
- [§5.1] The dataset is described only as 'more than two thousand pieces of high-quality data'; the paper should give the exact number of dialogues, the number of distinct diseases and symptoms, the train/test split sizes, and the generation prompts for reproducibility.
- [Data Availability] The statement that 'the data from the original dataset is not publicly available' conflicts with the paper's stated goal of proposing an evaluation method that avoids manual data collection; at minimum, the synthetically generated dialogues and the KG construction code should be released.
- [Figure 3] The LLM-as-judge results in Figure 3 lack detail: the axes, the number of pairwise comparisons, the agreement rate of the five repeated judgments, and the prompt template used for evaluation should all be specified.
- [References] Several references are malformed, notably [27] ('Qwen, :, Yang, A., ...'), and the author lists for [25] and [26] are incomplete; these should be corrected.
Circularity Check
The 84.68% diagnostic accuracy reduces to knowledge-graph retrieval consistency: the synthetic test dialogues and the KG are built from the same disease-symptom tuples, and Algorithm 1's cosine-similarity diagnosis (Eq. 3) can recover the label used to generate the test case.
-
fitted input called prediction
[Section 4 (dataset construction) and Section 5.1 (evaluation methodology)]
"We used the previously mentioned dataset of multiple rounds of doctor-patient conversations containing more than two thousand pieces of high-quality data."
The dialogue dataset is constructed in Section 4 from 'multiple binary tuples of disease + symptom list', with the patient assumed to suffer exclusively from the specified disease and exhibit all listed symptoms. Section 5.1 then evaluates DoPI on this 'previously mentioned dataset' without introducing any independent test set or describing a split that separates test tuples from the disease-symptom data used to build the KG. The reported 84.68% accuracy is therefore measured on data generated from the same disease-symptom associations that define the model's knowledge graph, so the number is a measure of self-consistency, not a prediction on unseen diagnostic cases.
-
self definitional
[Algorithm 1 (Section 3.2), Eqs. (1)-(3)]
"Similarity = Cosine(Symptoms, P D) if Similarity ≥ ϵ then Break"
The diagnostic loop terminates when the cosine similarity between the patient symptom vector P and a KG disease vector PD exceeds a threshold. P is a 0/1 vector of confirmed symptoms (Eq. 1), and PD is built from the weights between that disease and symptoms in the KG (Eq. 2), where the KG is constructed from 'existing data on TCM disease types and corresponding symptoms' with weights initialized from frequency. Since each synthetic test patient is generated from exactly a disease plus its symptom list, and the Honesty rule guarantees the patient confirms every queried symptom, the correct disease's KG vector overlaps precisely with the confirmed symptom set. Thus Eq.
full rationale
The central claim of the paper is the 84.68% interrogation accuracy in Table 3. Following the derivation chain: Section 3.2 builds the TCM knowledge graph from disease-symptom data with frequency-based weights; Section 4 constructs the multi-turn dialogue dataset from 'disease + symptom list' tuples using the same kind of TCM knowledge; Section 5.1 evaluates on that constructed dataset with no described independent split; and Algorithm 1 stops when cosine similarity (Eq. 3) between the patient's confirmed symptom vector and a KG disease vector exceeds a threshold. The synthetic patients are defined to have exactly the listed symptoms and to answer honestly, so the model's questions and the KG's disease vectors are co-derived from the same tuples. The diagnostic accuracy therefore reduces by construction to retrieving the disease label from the KG that generated the test dialogue. This is a fitted-input-called-prediction and self-definitional circularity for the headline result. The Q&A Ratio and Interrogation Distance metrics are less affected by this overlap, and the use of Sunsimiao as the expert model is external, but those do not rescue the core accuracy claim. No self-citation chain is involved; the circularity is in the data-generation and evaluation design. Because the central advertised number is forced by the experimental construction, the score is 8 rather than a lower value.
Assumptions & free parameters
free parameters (4)
- epsilon (diagnosis threshold) =
not reported
- sigma (Gaussian noise standard deviation) =
not reported
- number of questions per round =
2 or 3
- KG edge weights w =
set by dataset frequency, then updated
assumptions (4)
- domain assumption The TCM knowledge graph accurately captures disease-symptom relations needed for diagnosis.
- ad hoc to paper LLM-generated dialogues with the honesty constraint are representative of real doctor-patient conversations.
- domain assumption LLM-as-judge is a reliable evaluation of dialogue quality.
- domain assumption Frequency-based initialization of KG weights is a valid prior for diagnostic relevance.
Cite this review
Pith. "Pith review of DoPI: Doctor-like Proactive Interrogation LLM for Traditional Chinese Medicine." pith.science (2026). https://pith.science/paper/DYW7YMI5
@misc{pith2026250704877,
author = {Pith},
title = {Pith review of: DoPI: Doctor-like Proactive Interrogation LLM for Traditional Chinese Medicine},
year = {2026},
howpublished = {\url{https://pith.science/paper/DYW7YMI5}},
note = {Machine review of arXiv:2507.04877}
}
read the original abstract
Enhancing interrogation capabilities in Traditional Chinese Medicine (TCM) diagnosis through multi-turn dialogues and knowledge graphs presents a significant challenge for modern AI systems. Current large language models (LLMs), despite their advancements, exhibit notable limitations in medical applications, particularly in conducting effective multi-turn dialogues and proactive questioning. These shortcomings hinder their practical application and effectiveness in simulating real-world diagnostic scenarios. To address these limitations, we propose DoPI, a novel LLM system specifically designed for the TCM domain. The DoPI system introduces a collaborative architecture comprising a guidance model and an expert model. The guidance model conducts multi-turn dialogues with patients and dynamically generates questions based on a knowledge graph to efficiently extract critical symptom information. Simultaneously, the expert model leverages deep TCM expertise to provide final diagnoses and treatment plans. Furthermore, this study constructs a multi-turn doctor-patient dialogue dataset to simulate realistic consultation scenarios and proposes a novel evaluation methodology that does not rely on manually collected real-world consultation data. Experimental results show that the DoPI system achieves an accuracy rate of 84.68 percent in interrogation outcomes, significantly enhancing the model's communication ability during diagnosis while maintaining professional expertise.
Reference graph
Works this paper leans on
-
[1]
Advances in neural information processing systems 33, 1877–1901 (2020)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
2020
-
[2]
arXiv preprint arXiv:2302.13971 (2023)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
arXiv 2023
-
[3]
Ieee Network 33(5), 156–165 (2019)
Wang, X., Han, Y., Wang, C., Zhao, Q., Chen, X., Chen, M.: In-edge ai: Intelligen- tizing mobile edge computing, caching and communication by federated learning. Ieee Network 33(5), 156–165 (2019)
work page 2019
-
[4]
arXiv preprint arXiv:2305.15075 (2023)
Zhang, H., Chen, J., Jiang, F., Yu, F., Chen, Z., Li, J., Chen, G., Wu, X., Zhang, Z., Xiao, Q., et al.: Huatuogpt, towards taming language model to be a doctor. arXiv preprint arXiv:2305.15075 (2023)
arXiv 2023
-
[5]
arXiv preprint arXiv:2310.15896 (2023)
Chen, Y., Wang, Z., Xing, X., Xu, Z., Fang, K., Wang, J., Li, S., Wu, J., Liu, Q., Xu, X., et al.: Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt. arXiv preprint arXiv:2310.15896 (2023)
arXiv 2023
-
[6]
In: Proceedings of the AAAI Conference on Artificial Intelligence, vol
Yang, S., Zhao, H., Zhu, S., Zhou, G., Xu, H., Jia, Y., Zan, H.: Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 19368–19376 (2024)
work page 2024
-
[7]
MedChatZH: a Better Medical Adviser Learns from Better Instructions
Tan, Y., Li, M., Huang, Z., Yu, H., Fan, G.: Medchatzhx: a better medical adviser learns from better instructions. arXiv preprint arXiv:2309.01114 (2023)
work page Pith review arXiv 2023
- [8]
Show all 28 references
-
[9]
arXiv preprint 16 arXiv:2403.16056 (2024)
Zhang, H., Wang, X., Meng, Z., Chen, Z., Zhuang, P., Jia, Y., Xu, D., Guo, W.: Qibo: A large language model for traditional chinese medicine. arXiv preprint 16 arXiv:2403.16056 (2024)
2024 arXiv
-
[10]
Computer Methods and Programs in Biomedicine Update 6, 100158 (2024)
Yang, G., Liu, X., Shi, J., Wang, Z., Wang, G.: TCM-GPT: Efficient pre-training of large language models for domain adaptation in traditional chinese medicine. Computer Methods and Programs in Biomedicine Update 6, 100158 (2024)
2024
-
[11]
Journal of Chinese Information Processing 33(10), 1–9 (2019)
Byambasuren, O., Yang, Y., Sui, Z., Dai, D., Chang, B., Li, S., Zan, H.: Prelim- inary study on the construction of chinese medical knowledge graph. Journal of Chinese Information Processing 33(10), 1–9 (2019)
2019
-
[12]
He, J., Guo, Y., Lam, L.K., Leung, W., He, L., Jiang, Y., Chen, H.: OpenTCM: A GraphRAG-Empowered LLM-based System for Traditional Chinese Medicine Knowledge Retrieval and Diagnosis (2025)
2025
-
[13]
arXiv preprint arXiv:2304.08177 (2023)
Cui, Y., Yang, Z., Yao, X.: Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177 (2023)
2023 arXiv
-
[14]
arXiv preprint arXiv:2303.08774 (2023)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
2023 arXiv
-
[15]
arXiv preprint arXiv:2409.18486 (2024)
Zhong, T., Liu, Z., Pan, Y., Zhang, Y., Zhou, Y., Liang, S., Wu, Z., Lyu, Y., Shu, P., Yu, X., et al.: Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486 (2024)
2024
-
[16]
arXiv preprint arXiv:2412.19437 (2024)
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)
2024 arXiv
-
[17]
Le Scao, T., Fan, A., Akiki, C., Pavlick, E., Ili´ c, S., Hesslow, D., Castagn´ e, R., Luccioni, A.S., Yvon, F., Gall´ e, M., et al.: Bloom: A 176b-parameter open-access multilingual language model (2023)
2023
-
[18]
arXiv preprint arXiv:2307.15020 7, 3 (2023)
Sun, T., Zhang, X., He, Z., Li, P., Cheng, Q., Yan, H., Liu, X., Shao, Y., Tang, Q., Zhao, X., et al.: Moss: Training conversational language models from synthetic data. arXiv preprint arXiv:2307.15020 7, 3 (2023)
2023 arXiv
-
[19]
Computers in biology and medicine 172, 108290 (2024)
Tan, Y., Zhang, Z., Li, M., Pan, F., Duan, H., Huang, Z., Tang, Y.: Medchatzh: A tuning llm for traditional chinese medicine consultations. Computers in biology and medicine 172, 108290 (2024)
2024
-
[20]
Journal of the American Medical Informatics Association 31(9), 2019–2029 (2024)
Hua, R., Dong, X., Wei, Y., Shu, Z., Yang, P., Hu, Y., Zhou, X.: Lingdan: enhanc- ing encoding of traditional chinese medicine knowledge for clinical reasoning tasks with large language models. Journal of the American Medical Informatics Association 31(9), 2019–2029 (2024)
2024
-
[21]
Journal of Evidence-Based Medicine 18(1), 12658 (2025)
Ren, Y., Luo, X., Wang, Y., Li, H., Zhang, H., Li, Z., ADV ANCED Working 17 Group: Large language models in traditional chinese medicine: A scoping review. Journal of Evidence-Based Medicine 18(1), 12658 (2025)
2025
-
[22]
arXiv preprint arXiv:2304.08247 (2023)
Han, T., Adams, L.C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., L¨ oser, A., Truhn, D., Bressem, K.K.: Medalpaca–an open-source collec- tion of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247 (2023)
2023 arXiv
-
[23]
https://arxiv.org/abs/2303.14070
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., Zhang, Y.: ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge (2023). https://arxiv.org/abs/2303.14070
2023 arXiv
-
[24]
Nature Medicine, 1–8 (2025)
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S.R., Cole-Lewis, H., et al.: Toward expert-level medical question answering with large language models. Nature Medicine, 1–8 (2025)
2025
-
[25]
arXiv preprint arXiv:2106.09685 (2021)
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
2021 arXiv
-
[26]
https://arxiv.org/abs/1609.02907
Kipf, T.N., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks (2017). https://arxiv.org/abs/1609.02907
2017 arXiv
-
[27]
https://arxiv.org/abs/2412.15115
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C.: Qwen2.5 Technical Report (2025). https://arxiv.org/abs/2412.15115
2025 arXiv
-
[28]
In: Proceedings of the 37th International Con- ference on Neural Information Processing Systems
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging llm-as-a-judge with mt-bench and chatbot arena. In: Proceedings of the 37th International Con- ference on Neural Information...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.