Pith. sign in

REVIEW 5 major objections 6 minor 25 references

DocCHA: Towards LLM-Augmented Interactive Online diagnosis System

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DocCHA's confidence-guided three-stage pipeline lifts diagnostic accuracy over direct LLM prompting, reaching 95.86% on IMCS21 and 94.14% on DX.

desk verdict DocCHA is a sensible modular design for LLM diagnostic dialogue, but the reported gains are not statistically established and a few headline claims don't match the table. read the letter →

arxiv 2507.07870 v1 pith:OH7IRODY submitted 2025-07-10 cs.CL

Xinyi Liu , Dachun Sun , Yi R. Fung , Dilek Hakkani-Tür , Tarek Abdelzaher This is my paper
classification cs.CL
keywords conversationalhealthagentsLLM-augmenteddiagnosisinteractivediagnosticdialogueconfidence-guidedquestioningcausalgraphreasoningsymptomelicitationmedicalevaluationpatientsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DocCHA is a conversational diagnostic agent that does not just prompt a large language model for an answer. Instead it breaks the consultation into three stages — symptom collection, medical history acquisition, and causal-graph reasoning — and uses explicit confidence scores at each stage to decide what to ask next. On two real-world Chinese consultation datasets the full pipeline raises diagnostic accuracy to 95.86% and 94.14%, beating direct GPT-4o prompting by about five and four percentage points respectively, while extracting more clinically relevant symptoms at similar turn counts. The paper argues these gains come from the structured pipeline, not from a larger model, because ablating any stage degrades accuracy. If that is right, interactive diagnosis does not need to wait for bigger models; it can be improved by giving current models a confidence-aware conversational structure.

What carries the argument

The load-bearing object is the trio of confidence functions that gate when and what DocCHA asks. Symptom confidence combines coverage of a candidate diagnosis's canonical symptom set with how many required attributes (onset, severity, duration) have been filled; history confidence blends coverage, semantic relevance, and expression certainty; causal-graph confidence aggregates coherence, medical plausibility measured against UMLS-style knowledge, and entailment. Whenever a score falls below its threshold within a per-module question quota, the system asks a targeted follow-up, and each symptom also carries a discriminative-power score so that questions are aimed at the clues most likely to separate competing diagnoses. A separate Patient Agent Simulator generates answers grounded in real transcripts, so the evaluation can measure how much medically relevant information the agent's questions actually extract.

What would settle it

Replace the Patient Agent Simulator with responses written by human annotators or recorded from real patients who have not seen the gold diagnosis, and rerun the comparison against GPT-4o direct prompting at matched turn counts. If DocCHA's accuracy advantage (about five points on IMCS21) shrinks below statistical significance, the simulator's grounding rule, not the pipeline, was carrying the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that decomposing a diagnostic dialogue into three confidence-scored modules — symptom elicitation, history acquisition, and causal graph construction — yields measurably better diagnoses than letting an LLM converse freely. DocCHA's symptom module prompts for the most discriminative missing signs; its history module probes relevant background categories until coverage, relevance, and certainty scores pass a threshold; its causal module builds an explicit graph linking evidence to hypotheses and follows up on the weakest link. The paper reports 95.86% accuracy with GPT-4o on IMCS21 versus 90.68% for direct prompting, and 94.14% versus 89.96% on DX, with ablation studies attributing the difference to the pipeline rather than to prompt engineering alone.

Load-bearing premise

The evaluation assumes that the Patient Agent Simulator, which answers only when a response is grounded in the real transcript, behaves like a real patient well enough that the measured accuracy and recall gains will transfer to actual consultations.

Editorial extensions

If this is right

  • If the gains are real, current LLMs can be made more reliable at interactive diagnosis without fine-tuning, simply by wrapping them in confidence-scored question-selection logic.
  • The per-module confidence scores give a natural stopping rule: the system can end the interview when its causal graph passes the thresholds, limiting patient burden.
  • Because the causal graph exposes which links are weak, the system can justify its diagnosis and retry or repair unsupported reasoning instead of outputting a flat answer.
  • The method transfers across languages and datasets: the same modular design produces large gains on two different Chinese consultation benchmarks.
  • Removing any module costs accuracy, making structured end-to-end design a requirement for these gains rather than a cosmetic addition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Patient Agent Simulator's answerability gate — a response is grounded only if the question is answerable from the transcript — is the most fragile link: if that gate implicitly encodes the gold diagnosis, then the accuracy and recall numbers could overstate what a real patient would provide.
  • The same confidence-and-quota pattern could apply to other interactive elicitation tasks beyond medicine, such as troubleshooting, legal intake, or requirements gathering, where an LLM must decide what to ask rather than answer.
  • A direct test that the paper does not report: compare DocCHA's question efficiency against experienced physicians on the same transcripts with the same turn budget; the information-recall metric would then show whether the gains come from asking the right questions or from the simulator's permissive grounding.
  • The sensitivity analysis shows plateaus after five turns, suggesting a deployment could cap consultations near that length; this is a design consequence the authors state as a property of the method rather than an operational recommendation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DocCHA, a modular, confidence-aware framework for LLM-based conversational diagnosis. The system decomposes diagnostic dialogue into three stages: symptom collection, history acquisition, and causal graph construction and refinement, with each stage using interpretable confidence scores to select follow-up questions. A Patient Agent Simulator is introduced to evaluate information elicitation from real consultation transcripts. Experiments on two Chinese medical dialogue datasets (IMCS21 and DX) compare DocCHA against direct prompting baselines (LLaMA-3, GPT-3.5, GPT-4o) and ablation variants, reporting higher diagnostic accuracy and information recall.

Significance. If the results are validated, DocCHA would be a useful contribution to structured and interpretable conversational diagnosis: the modular confidence-guided questioning is a plausible alternative to free-form prompting, the causal graph module provides a mechanism for transparent reasoning, and the evaluation introduces a reproducible patient simulator. However, the current evidence is not yet convincing because the headline recall improvement is not supported by the table, turn budgets are not actually matched, the reported differences lack statistical significance testing, and the module weights were selected on the same test sets used for the main comparisons. These issues are fixable, so the framework itself remains promising.

major comments (5)
  1. [Abstract and §1, §4.4, Table 1] The abstract and introduction claim 'over 30% improvement in symptom recall', but Table 1 does not support this. In the Recallinfo column, DocCHA (GPT-4o) improves over GPT-4o direct prompting by only 2.37 absolute points (52.12 to 54.49, about 4.5% relative), and over LLaMA-3 by 6.77 absolute points (47.72 to 54.49, about 14.2% relative). The 29.63% figure in the introduction refers to accuracy relative to LLaMA-3, not symptom recall. This overstatement should be corrected.
  2. [§4.6 and §4.5, Figure 3 and Table 1] The sensitivity analysis in §4.6 selects the default weights (α=0.5, λ=(0.4,0.3,0.3), μ=(0.4,0.3,0.3)) as the configurations that peak on the same test sets used in Table 1, and the ablations in §4.5 then remove modules while retaining these test-selected weights. This creates an optimism bias in both the absolute results and the attributed module contributions. The authors should validate the chosen weights on a held-out set or report results across the full sensitivity grid, and at minimum acknowledge this selection effect when interpreting the ablation magnitudes.
  3. [§4.2, §4.4, Table 1] The text states that 'All baselines ask the same number of questions' and that turn count is matched to DocCHA, but Table 1 reports average turns of 7.1 for DocCHA versus 6.8 for GPT-4o, 6.4 for GPT-3.5, and 5.6 for LLaMA-3. The extra turns available to DocCHA could partly explain its gains, so the comparison is not budget-matched as claimed. The authors should either enforce equal turn budgets in the experiments or report accuracy as a function of turn count, and should correct the text in §4.2.
  4. [§4.4, Table 1] No error bars, confidence intervals, or significance tests are reported. With only 100 IMCS21 sessions and 120 DX sessions, the +5.18 accuracy gap over GPT-4o corresponds to roughly five additional correct cases out of 100, which could be within sampling noise. The authors should report paired significance tests (e.g., McNemar's test) or bootstrap confidence intervals, and ideally multiple random seeds given the stochastic nature of LLM prompting.
  5. [§3.4, Eq. (11)] The Patient Agent Simulator uses an undisclosed LLM to generate responses and to judge whether a question is 'answerable from T'. If this simulator uses the same underlying model family as the DocCHA backbone, or if its gating is systematically over-permissive, the evaluation could be self-referential and favor the modular pipeline. The authors should disclose the simulator's model, validate its outputs against human judgments or real patient behavior, and test how the main results change when the simulator is replaced or its gating threshold is varied.
minor comments (6)
  1. [Table 1] The table header contains a typo: 'ICMS' should be 'IMCS21'.
  2. [§3.4 and §4.3] The definition of Recallinfo in Eq. (13) is identical to Coverage in Eq. (12); the authors should clarify whether these are indeed the same metric or whether one is meant to be precision or a different quantity.
  3. [§4.1] The description of using Google Translate to convert Chinese consultations to English is brief; the authors should state whether the gold diagnoses and UMLS matching were also translated, and how translation errors were handled.
  4. [References] Several references are incomplete, for example 'Meng Chen et al. 2023', 'Or Honovich et al. 2022', and 'Yiheng Yang et al. 2023' lack full author lists and page numbers; the reference list should be completed to journal standards.
  5. [§3, Appendix A] The sentence 'The prompts are shown in Session A' should read 'Appendix A'.
  6. [§4.6 and Figure 3] The sensitivity analysis text refers to 'Bold bars' and 'Black dotted lines' in Figure 3, but the figure is not included in the manuscript text; the authors should ensure the figure is legible and clearly labeled.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported accuracy gains and module attributions rest on weights selected on the same test sets in §4.6, so the headline prediction is partly a fitted value.

  1. fitted input called prediction [Section 4.6 Sensitivity Analysis / Figure 3 caption; default weights used in Table 1 (§4.4) and §4.2 Weight Settings.]
    "Bold bars indicate the configuration used as the default setting for each module. ... Symptom Collection (α): Performance peaks at α=0.5 ... History Acquisition (λ): The default weighting (coverage=0.4, relevance=0.3, certainty=0.3) yields optimal results. ... Causal Reasoning (µ): Giving coherence the highest weight (µ1=0.4) produces the most accurate and interpretable reasoning."

    The default weights used in Table 1 are selected as the peak-performing configurations in a sensitivity scan run on the same IMCS21 and DX test sets. Therefore the reported DocCHA gains (+5.18 accuracy over GPT-4o) and the per-module ablation deltas are in-sample fitted values, not out-of-sample predictions. The paper's own §4.2 says weights are 'fixed based on clinical reasoning principles', but §4.6 identifies them empirically by peak performance on the evaluation sets. No holdout tuning split or significance test is reported, so the central attribution of the gains to the modular pipeline partly reduces to the fitted parameter choice.

full rationale

The only concrete circular step is the weight-selection procedure: Table 1's default configuration (α=0.5, λ=(0.4,0.3,0.3), μ=(0.4,0.3,0.3)) is explicitly the bold-bar configuration that peaks on the same IMCS21 and DX evaluation sets in §4.6. Because the same 100/120 sessions are used both to pick those weights and to report DocCHA's +5.18 accuracy and module-ablation contributions, those numbers are in-sample fitted values rather than predictions; the §4.2 statement that weights are 'fixed based on clinical reasoning principles' does not remove the selection. The remaining concerns—the LLM-based Patient Agent Simulator, the absence of significance tests, and the abstract's unsupported 'over 30%' recall claim—are validity and reporting issues, not circular reductions: the simulator is used consistently for all systems and Eq. 11 is an evaluation proxy, not a fitted input to DocCHA's inference. There is no self-citation chain or definitional identification of X with Y. Hence partial circularity, score 6.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The system's reported performance depends on several manually chosen weights (α, λ, μ) that the sensitivity analysis suggests were tuned on the test set, plus unspecified thresholds, quotas, and top-K values. The evaluation further assumes that LLM-generated confidence scores and LLM-based patient responses are reliable, and that UMLS/SemMedDB grounding is a valid proxy for medical plausibility.

free parameters (7)
  • α (symptom module weight) = 0.5
    Section 4.2 states α=0.5 balances coverage and detail; Section 4.6 says performance peaks at α=0.5, indicating test-set selection.
  • λ (history weights) = λ1=0.4, λ2=0.3, λ3=0.3
    Section 4.2 sets these defaults; Section 4.6 says the default weighting 'yields optimal results', implying it was chosen based on sensitivity analysis.
  • μ (causal weights) = μ1=0.4, μ2=0.3, μ3=0.3
    Chosen as default; sensitivity analysis says 'giving coherence the highest weight produces the most accurate and interpretable reasoning'.
  • τ_sym, τ_hist, τ_causal (confidence thresholds)
    Thresholds control when follow-up questions are triggered (Sections 3.1-3.3) but their values are never reported.
  • q_sym, q_hist, q_causal (question quotas) = up to 4 questions per module (loose)
    Only loosely specified; exact quota per module is not given.
  • K (number of candidate diagnoses)
    Used in Eq. 1 to compute discriminative power; value not specified.
  • Turn limit T
    Sensitivity analysis varies T but the default value used in Table 1 is not explicitly stated.
assumptions (5)
  • domain assumption Patient Agent Simulator responses generated by an LLM from real transcripts faithfully represent real patient behavior
    Section 3.4, Eq. 11: when a query is 'answerable from T', the response is LLM(T,q_i), otherwise a fixed negative string; this assumes the LLM-produced answer is equivalent to what a real patient would say and that the gating is unbiased.
  • domain assumption Confidence scores computed from LLM outputs are reliable indicators of diagnostic information sufficiency
    Equations 2-4, 5-8, 9-10 use LLM-derived symptom sets, history relevance, certainty, and entailment to decide when to stop asking questions; the paper does not validate these scores against clinician judgment.
  • domain assumption UMLS and SemMedDB provide sufficient and accurate medical grounding for causal plausibility
    Eq. 9 scores causal edges by overlap with UMLS/SemMedDB; this assumes the knowledge bases contain the relevant relations and that absence means implausibility.
  • domain assumption The top-K candidate diagnoses generated by the LLM form a complete hypothesis space
    Eq. 1 computes discriminative power only over D = {d1,...,dK}; if the true diagnosis is not in the top-K, the scoring may mislead.
  • standard math Standard probability and linear algebra operations are valid
    Variance (Eq. 1), cosine similarity (Eq. 6), and weighted sums are used without formal proof, which is acceptable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DocCHA: Towards LLM-Augmented Interactive Online diagnosis System." pith.science (2026). https://pith.science/paper/OH7IRODY

@misc{pith2026250707870,
  author       = {Pith},
  title        = {Pith review of: DocCHA: Towards LLM-Augmented Interactive Online diagnosis System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OH7IRODY}},
  note         = {Machine review of arXiv:2507.07870}
}
read the original abstract

Despite the impressive capabilities of Large Language Models (LLMs), existing Conversational Health Agents (CHAs) remain static and brittle, incapable of adaptive multi-turn reasoning, symptom clarification, or transparent decision-making. This hinders their real-world applicability in clinical diagnosis, where iterative and structured dialogue is essential. We propose DocCHA, a confidence-aware, modular framework that emulates clinical reasoning by decomposing the diagnostic process into three stages: (1) symptom elicitation, (2) history acquisition, and (3) causal graph construction. Each module uses interpretable confidence scores to guide adaptive questioning, prioritize informative clarifications, and refine weak reasoning links. Evaluated on two real-world Chinese consultation datasets (IMCS21, DX), DocCHA consistently outperforms strong prompting-based LLM baselines (GPT-3.5, GPT-4o, LLaMA-3), achieving up to 5.18 percent higher diagnostic accuracy and over 30 percent improvement in symptom recall, with only modest increase in dialogue turns. These results demonstrate the effectiveness of DocCHA in enabling structured, transparent, and efficient diagnostic conversations -- paving the way for trustworthy LLM-powered clinical assistants in multilingual and resource-constrained settings.

Figures

Figures reproduced from arXiv: 2507.07870 by the authors.

Figure 1
Figure 1. Overview of the DocCHA framework. The left panel shows a multi-turn dialogue between the user and the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An end-to-end diagnosis example with DocCHA. The left shows the multi-turn interaction between [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Sensitivity of DocCHA’s modules across Accuracy, Cosine Similarity, Information Recall, and Turn Count [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Prompt used in Module 2 to elicit missing [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Prompt set used in Module 3 for generating [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 15 canonical work pages

  1. [1]

    Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain. 2023. Conversational health agents: A personalized llm-powered agent framework. arXiv preprint arXiv:2310.02374

    arXiv 2023

  2. [2]

    Anastasiya Belyaeva, Justin Cosentino, Farhad Hormozdiari, Krish Eswaran, Shravya Shetty, Greg Corrado, Andrew Carroll, Cory Y McLean, and Nicholas A Furlotte. 2023. Multimodal llms for health grounded in individual-specific data. In Workshop on Machine Learning for Multimodal Healthcare Data, pages 86--102. Springer

    work page 2023

  3. [3]

    Som Biswas. 2023. Chatgpt and the future of medical writing

    work page 2023

  4. [4]

    Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucleic Acids Research, 32(suppl\_1):D267--D270

    work page 2004

  5. [5]

    Meng Chen et al. 2023. Assessing the utility of chatgpt for cancer treatment information. JAMA Oncology

    work page 2023

  6. [6]

    Arthur S Elstein and Alan Schwarz. 2002. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. Bmj, 324(7339):729--732

    work page 2002

  7. [7]

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander L \"o ser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca--an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247

    arXiv 2023

  8. [8]

    Or Honovich et al. 2022. True or false? measuring factual consistency in natural language generation with entailment. Transactions of the Association for Computational Linguistics

    work page 2022

Show all 25 references
  1. [9]

    Halil Kilicoglu et al. 2012. Semmeddb: a pubmed-scale repository of biomedical semantic predications. Bioinformatics, 28(23):3158--3160

    2012

  2. [10]

    Haotian Li et al. 2023. Chatdoctor: A medical dialogue llm fine-tuned on doctor-patient conversations. arXiv preprint arXiv:2305.09617

    2023 arXiv

  3. [11]

    Kaiyu Lin, Fei Liu, et al. 2023. Carechat: Llm-based medical dialogue agent with multilevel memory and external evidence. arXiv preprint arXiv:2312.04587

    2023 arXiv

  4. [12]

    Chunyu Liu, Yongpei Ma, Kavitha Kothur, Armin Nikpour, and Omid Kavehei. 2023. Biosignal copilot: Leveraging the power of llms in drafting reports for biomedical signals. medRxiv, pages 2023--06

    2023

  5. [13]

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409

    2022

  6. [14]

    Michael McTear. 2022. Conversational ai: Dialogue systems, conversational agents, and chatbots. Springer Nature

    2022

  7. [15]

    Joao Luis Zeni Montenegro, Cristiano Andr \'e da Costa, and Rodrigo da Rosa Righi. 2019. Survey of conversational agents in health. Expert Systems with Applications, 129:56--67

    2019

  8. [16]

    Yifan Peng et al. 2023. Self-aligning pretraining for ehr question answering and dialogue generation. In ACL

    2023

  9. [17]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857--16867

    2020

  10. [18]

    Saeid Ashraf Vaghefi, Dominik Stammbach, Veruska Muccione, Julia Bingler, Jingwei Ni, Mathias Kraus, Simon Allen, Chiara Colesanti-Senni, Tobias Wekhof, Tobias Schimanski, et al. 2023. Chatclimate: Grounding conversational ai in climate science. Communications Earth & Environm...

    2023

  11. [19]

    Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Attila Kiraly, Sahar Kazemzadeh, Zakkai Melamed, et al. 2023. Elixr: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and ...

    2023 arXiv

  12. [20]

    Yiheng Yang et al. 2023. Meddialog: Llm-based medical consultation simulation. arXiv preprint arXiv:2306.05389

    2023 arXiv

  13. [21]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems (NeurIPS)

    2021

  14. [22]

    Jiayi Zhang, Xiao Wang, and Yifan Peng. 2022. Bioie: Uncertainty detection in clinical text. In ACL

    2022

  15. [23]

    Jinsung Zhou and Jimeng Sun. 2019. Learning to diagnose with lstm recurrent neural networks. arXiv preprint arXiv:1701.07875

    2019 arXiv

  16. [24]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  17. [25]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.