Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DoctorRAG claims that fusing knowledge-base retrieval with nearest-neighbour patient-case retrieval and iterative textual-gradient refinement outperforms strong RAG baselines across multilingual medical tasks.

desk verdict The patient-base construction leaks the answer into the prompt, so the headline gains don't identify the method's contribution; the framework idea is still worth a look. read the letter →

arxiv 2505.19538 v1 pith:77OJZXES submitted 2025-05-26 cs.CL cs.AIcs.CEcs.IRcs.MA

Yuxing Lu , Gecheng Fu , Wei Wu , Xukai Zhao , Sin Yee Goi , Jinzhuo Wang This is my paper
classification cs.CLcs.AIcs.CEcs.IRcs.MA
keywords retrieval-augmentedgenerationmedicalquestionansweringcase-basedreasoningpatientanalogytextualgradientsdiseasediagnosismultilingualNLPknowledgegrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DoctorRAG argues that medical retrieval-augmented generation should imitate two complementary human reasoning modes: consulting formal medical knowledge and recalling similar past patients. The paper claims that fusing both sources—through concept-tagged declarative knowledge retrieval and nearest-neighbour patient retrieval—makes language-model answers more accurate, relevant, and faithful than knowledge-only RAG systems. It also claims that a multi-agent textual-gradient loop, Med-TextGrad, iteratively improves the generated answer against two criteria: consistency with retrieved medical context and relevance to the patient query. The experiments span Chinese, English, and French and cover diagnosis, question answering, treatment recommendation, and text generation; for example, English DDXPlus diagnosis accuracy reaches 98.27%, versus 92.37% for the strongest baseline. If these results hold, the paper offers a practical recipe for injecting experiential, case-based evidence into medical language-model systems.

What carries the argument

The load-bearing object is the dual database: a Knowledge Base of declarative sentences tagged with first-level ICD-10 concept labels, and a Patient Base of de-identified records with embedding indices. Retrieval uses a concept-constrained cosine score for knowledge (a candidate is excluded when its concept tags do not intersect the query's tags) and plain cosine similarity for patients. The second main mechanism is Med-TextGrad, a multi-agent textual-gradient loop: a Generator produces an answer, then a Context Criterion and a Patient Criterion produce critiques, and optimizer agents turn those critiques into textual gradients that update the prompt for the next iteration. This loop is the framework's mechanism for faithfulness: it repeatedly grounds the output in the retrieved context and the patient's query.

What would settle it

Count, for evaluation queries in a dataset like English DDXPlus, how often the correct disease appears among the top-k retrieved patient records that pass the 0.99 similarity filter; if that rate is high, or if removing the target disease and reference answer from the retrieved patient records drops the reported 98.27% accuracy toward the strongest baseline's 92.37%, the advantage is explained by label leakage rather than patient-analogy reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that adding a patient-case memory to RAG—not as a replacement for knowledge bases but as a parallel retrieval channel—substantially improves medical answers. DoctorRAG's hybrid retrieval filters knowledge sentences by first-level ICD-10 concept tags and retrieves patient records by embedding similarity, giving the generator both expertise and experience. The generated answer is then refined by Med-TextGrad, an iterative optimization loop in which separate context and patient criteria produce textual critiques, answer-level textual gradients, and prompt-level gradients, analogous to backpropagation. Across tasks and backbones, DoctorRAG consistently beats Vanilla RAG, Proposition RAG, and the strongest graph-based baseline, with the largest margins on diagnosis; ablations attribute the gain to the combination of patient-base retrieval, knowledge-base retrieval, concept tagging, and declarative statement transformation.

Load-bearing premise

The evaluation assumes that removing patient records whose text embeddings are nearly identical (similarity above 0.99) from the patient base is enough to stop test answers from leaking, but the retrieved similar-patient text includes the target disease and a full reference answer, so other same-disease records can still hand the model the answer.

Editorial extensions

If this is right

  • Medical RAG systems that only retrieve from static knowledge bases can improve by adding a retrievable store of anonymized patient records, even across different languages and diseases.
  • Iterative refinement should be run with a small, fixed number of steps; the paper's pairwise comparisons show the largest gain at iteration one and diminishing or slightly negative returns by iteration three.
  • Each architectural component contributes: removing patient retrieval, knowledge retrieval, concept tagging, or declarative statement transformation lowers accuracy, so the full pipeline rather than any single component explains the reported gains.
  • Retrieval quality saturates with context size: performance on the MuZhi dataset converges once roughly four retrieved items are used, while token cost rises linearly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the patient-analogy channel is what drives the gain, the same dual-retrieval-plus-refinement pattern could transfer to other decision domains that accumulate cases, such as legal advice or technical support; the paper only evaluates medical data.
  • Editorial inference: a cleaner leakage check would rerun the evaluation with each retrieved patient record's target disease and reference answer removed; the paper does not report this condition.
  • Editorial inference: a deployment-oriented next step would directly measure memorization risk, namely whether nearest-neighbour retrieval can regurgitate identifiable patient text; the paper does not address this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DoctorRAG, a medical retrieval-augmented generation framework that retrieves both explicit knowledge (from a knowledge base) and implicit experiential knowledge (from a patient base), and then refines answers with Med-TextGrad, a multi-agent textual-gradient optimization procedure. The system is evaluated on Chinese, English, and French datasets covering diagnosis, question answering, treatment recommendation, and text generation, with the central claim being consistent, large improvements over Vanilla RAG, Proposition RAG, and Graph RAG.

Significance. If the empirical evaluation were sound, the paper would make a useful contribution: integrating similar-patient cases into medical RAG is a plausible and underexplored direction, and the Med-TextGrad refinement loop is an interesting adaptation of textual-gradient optimization. The manuscript also provides detailed prompts, an ablation study, and a token-cost analysis, which aid reproducibility and future work. However, the central evidence is compromised by a likely label-leakage problem in the patient-base construction, and the iterative-refinement results rely on a circular LLM-judge setup. As presented, the reported superiority over baselines cannot be attributed to the proposed method with confidence.

major comments (3)
  1. [§3.1, §4.1, Appendix E.1] The evaluation protocol leaks the ground-truth answer into the prompted context. Section 3.1 constructs the Patient Base from approximately 80% of the same benchmark datasets used for testing, excluding only the exact record and records with cosine similarity above 0.99. Equation (2) then retrieves top-k patients with no disease-aware exclusion, and Appendix E.1 shows that each retrieved patient record contains a 'Target Disease' field and a full reference 'Answer.' Since datasets such as DDXPlus contain many patients with the same disease, and Figure 3 confirms disease-specific embedding clusters, the top-k retrieved cases will often contain the correct label and a reference answer for the test patient. Baselines as described in Section 4.1 do not have access to this patient-base signal, so the large gaps in Table 2, e.g., 98.27% versus 92.37% on DDXPlus (EN), and the 'Patient Base Retrieval' ablation in Table 3 are confounded: removing that module removes both the hypothesized experiential reasoning and the leaked label. The authors should re-run experiments with an external patient corpus, or at minimum strip 'Target Disease' and 'Answer' fields from retrieved records, exclude same-disease neighbors, and report results for all baselines and ablations.
  2. [Table 2, §4.1] All claims of 'significantly outperforming' in Section 4.1 rest on single-run point estimates in Table 2, with no confidence intervals, standard errors, significance tests, or per-cell sample sizes. Several differences are small, e.g., 94.96% versus 92.87% on DialMed, or 72.64% versus 71.73% on NEJM-QA, and without repeated runs or bootstrapped intervals it is impossible to assess whether these differences are stable. The authors should provide bootstrapped confidence intervals and/or paired significance tests for the main comparisons and for the ablation results in Table 3.
  3. [§4.3, Appendix D.3] The pairwise comparison supporting the Med-TextGrad claim uses DeepSeek-V3 both as generator and as evaluator, which creates a self-referential evaluation loop. The 'human verification' is described only as '2 human experts' with no protocol, no inter-annotator agreement, and no adjudication details, and the evaluation is based on only 50 COD samples. Given that the judge is the same model family that produces the answers, the observation that DoctorRAG's outputs beat ground truth in over 90% of comparisons could reflect stylistic self-preference rather than genuine quality. The authors should use independent judges, including human annotators with reported agreement, and provide the full evaluation protocol.
minor comments (5)
  1. [§4.2, Figure 3] The UMAP clustering of patient embeddings by disease actually reinforces the leakage concern in the major comment, because it shows that same-disease patients form tight clusters and will frequently be retrieved for a test patient. The authors should explicitly address this in their response.
  2. [Table 3, Figure 5] The manuscript contains garbled Unicode tokens around Table 3 and Figure 5, e.g., '/uni00000014/uni00000015/...' which should be fixed before any resubmission.
  3. [§A.2] Reference [28] is cited for both SelfCheckGPT and Chain-of-Verification; these are distinct methods and should have separate citations.
  4. [Figure 4] The description of the pairwise comparison matrices ('cell (Y,X) in the lower-left') is confusing; please clarify whether rows or columns correspond to the reference answer and report the exact number of ties.
  5. [Table 2] The table reports percentages but not the number of evaluation samples per cell; adding N per dataset or explicitly stating equal sample sizes would make the comparisons interpretable.

Circularity Check

1 steps flagged · score 6.0 of 10

DoctorRAG's headline gains are confounded by a patient-base evaluation loop that puts ground-truth disease labels and reference answers into the prompt.

  1. fitted input called prediction [Section 3.1; Eq. (2); Appendix E.1; Prompt D.1]
    "approximately 80% of the patient records from each dataset were allocated for constructing DoctorRAG’s patient base, while the remaining were held out as a distinct evaluation set. To ensure an unbiased assessment and prevent data leakage, we guarantee that for any given sample within this evaluation set, its corresponding full patient record and extreme similar ones (similarity > 0.99) were strictly excluded from DoctorRAG’s patient base during evaluation of that specific sample."

    Patient Retrieval (Eq. 2) is a pure top-k cosine search with no threshold, and the Patient Base is an 80% split of the same benchmark, so a test query's nearest neighbors are often other patients with the same disease—the paper's own UMAP (Fig. 3) shows same-disease clusters. The >0.99 exclusion removes only near-duplicates, not these same-disease neighbors. Appendix E.1 shows each retrieved record includes 'Target Disease' and a full reference 'Answer'; Prompt D.1 injects this as 'Similar Patients Information' and asks the model to output a disease from options. Thus the ground-truth label is available in the context for many test samples, while the RAG baselines retrieve only from the knowledge base and lack this signal.

full rationale

DoctorRAG is an empirical systems paper, so there is no closed-form derivation chain whose output equals its input; I found no load-bearing self-citation or imported uniqueness theorem. The significant circularity is in the evaluation loop. Section 3.1 builds the Patient Base from an 80% split of the very same benchmark datasets and excludes only near-duplicates with cosine similarity above 0.99. Patient retrieval (Eq. 2) is an unthresholded top-k search, and the paper's own UMAP analysis shows that same-disease patients form tight clusters. Appendix E.1 shows that each retrieved patient record contains 'Target Disease' and the full reference 'Answer', and Prompt D.1 feeds these into the prompt as 'Similar Patients Information' while asking the model to select the disease from options. Consequently, for many test queries the ground-truth disease label is present in the prompt context, whereas the Vanilla, Proposition, and Graph RAG baselines retrieve only from the knowledge base and do not receive this label-in-context signal. This makes the central comparisons—such as 98.27% versus 92.37% on DDXPlus (EN)—and the Table 3 ablation for Patient Base Retrieval partly artifacts of leaked answer information rather than evidence for the proposed dual-retrieval and Med-TextGrad reasoning. There is also a secondary self-evaluation risk: DeepSeek-V3 serves both as a generator and as the pairwise judge in Section B.7, though human verification is reported for the sampled comparisons. Overall, the central empirical 'prediction' is substantially forced by the way the prompt context is constructed, so the partial circularity score is 6 rather than 0-2; this is an evaluation-loop circularity, not an equation-level derivation collapse.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework rests on embedding similarity as a proxy for clinical analogy, on same-dataset splits being leakage-free, on coarse ICD-10 tags being sufficient, and on LLM self-critique being reliable. The first and fourth are common domain assumptions; the second is the most fragile and is contradicted by the prompt format in E.1.

free parameters (3)
  • k (top-k retrieval size) = Not stated; Figure 5 implies k around 4
    Controls how many knowledge sentences and patient records enter the context; chosen from token-performance analysis, not fixed in main text.
  • T (Med-TextGrad iterations) = 3
    Predetermined number of refinement rounds (Section 2.2.3); pairwise evaluation on 50 samples suggests gains plateau at T=2.
  • Similarity exclusion threshold = 0.99
    Records with cosine similarity above 0.99 to a test sample are removed from the patient base; this threshold is chosen ad hoc and does not exclude lower-similarity same-disease records (Section 3.1).
assumptions (5)
  • domain assumption Cosine similarity between text embeddings of patient complaints captures clinically meaningful patient analogy.
    Used in Eq. (2); UMAP clustering (B.6) is suggestive but not external clinical validation.
  • ad hoc to paper A patient base drawn from the same datasets as the test set, with only >0.99 similarity exclusions, gives an unbiased evaluation.
    Section 3.1; same-disease patients remain and retrieved records carry target-disease labels, so this premise is questionable.
  • domain assumption First-level ICD-10 categories are sufficient concept tags for medical retrieval.
    B.4; coarse categories may either admit irrelevant knowledge or exclude relevant fine-grained matches.
  • domain assumption LLM-generated critiques and textual gradients consistently improve answer quality.
    Section 2.2 and C.1; the paper itself notes this depends on LLM capability and prompt design.
  • domain assumption Chain-rule notation in Eq. (13) is a valid model for prompt optimization.
    C.1 clarifies these are conceptual, not mathematical gradients; the notation is an analogy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients." pith.science (2026). https://pith.science/paper/77OJZXES

@misc{pith2026250519538,
  author       = {Pith},
  title        = {Pith review of: DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77OJZXES}},
  note         = {Machine review of arXiv:2505.19538}
}
read the original abstract

Existing medical RAG systems mainly leverage knowledge from medical knowledge bases, neglecting the crucial role of experiential knowledge derived from similar patient cases -- a key component of human clinical reasoning. To bridge this gap, we propose DoctorRAG, a RAG framework that emulates doctor-like reasoning by integrating both explicit clinical knowledge and implicit case-based experience. DoctorRAG enhances retrieval precision by first allocating conceptual tags for queries and knowledge sources, together with a hybrid retrieval mechanism from both relevant knowledge and patient. In addition, a Med-TextGrad module using multi-agent textual gradients is integrated to ensure that the final output adheres to the retrieved knowledge and patient query. Comprehensive experiments on multilingual, multitask datasets demonstrate that DoctorRAG significantly outperforms strong baseline RAG models and gains improvements from iterative refinements. Our approach generates more accurate, relevant, and comprehensive responses, taking a step towards more doctor-like medical reasoning systems.

Figures

Figures reproduced from arXiv: 2505.19538 by the authors.

Figure 1
Figure 1. General and Medical RAGs retrieve solely from knowledge bases, whereas DoctorRAG [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed DoctorRAG & Med-TextGrad framework. Medical knowl￾edge is transformed into declarative statements and tagged to match the patient’s query. Dual-database retrieval generates a response incorporating both clinical expertise and experience. A two-way multi￾agent Med-TextGrad optimization process refines the answer through an iterative computation graph involving textual gradient backpropagation… view at source ↗
Figure 3
Figure 3. UMAP visualizations of patient embeddings on four distinct medical datasets. Each point [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pairwise comparison scores for ground truth (GT), original answer (OA), refined answers [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Performance v.s. Token consumption. The ablation study ( [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Language-Guided Tuning: Enhancing Numeric Optimization with Textual Feedback

    cs.AI 2025-08 reject novelty 4.0 of 10

    A multi-agent LLM framework uses natural-language 'textual gradients' to jointly tune architecture, features, training strategy, and hyperparameters.

Reference graph

Works this paper leans on

78 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  4. [4]

    Practical recommendations for gradient-based training of deep architectures

    Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade: Second edition, pages 437–478. Springer, 2012

    2012

  5. [5]

    Quality assurance during the rapid implementation of an ai-assisted clinical documentation support tool

    Carol H Cain, Anna C Davis, Benjamin Broder, Eugene Chu, Amanda Hauser DeHaven, Anthony Domenigoni, Nancy Gin, Anuj Kapoor, Vincent Liu, Ainsley MacLean, et al. Quality assurance during the rapid implementation of an ai-assisted clinical documentation support tool. NEJM AI, 2(4):AIcs2400977, 2025

    work page 2025

  6. [6]

    Cod, towards an interpretable medical agent using chain of diagnosis, 2024

    Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, and Benyou Wang. Cod, towards an interpretable medical agent using chain of diagnosis, 2024

    work page 2024

  7. [7]

    Can textual gradient work in federated learning? arXiv preprint arXiv:2502.19980, 2025

    Minghui Chen, Ruinan Jin, Wenlong Deng, Yuanyuan Chen, Zhi Huang, Han Yu, and Xiaoxiao Li. Can textual gradient work in federated learning? arXiv preprint arXiv:2502.19980, 2025

    arXiv 2025

  8. [8]

    Dense x retrieval: What retrieval granularity should we use?, 2024

    Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense x retrieval: What retrieval granularity should we use?, 2024

    work page 2024

Show all 78 references
  1. [9]

    Mrd-rag: Enhancing medical diagnosis with multi-round retrieval-augmented generation

    Yixiang Chen, Penglei Sun, Xiang Li, and Xiaowen Chu. Mrd-rag: Enhancing medical diagnosis with multi-round retrieval-augmented generation. arXiv preprint arXiv:2504.07724, 2025

    2025 arXiv

  2. [10]

    A survey on case-based reasoning in medicine

    Nabanita Choudhury and Shahin Ara Begum. A survey on case-based reasoning in medicine. International Journal of Advanced Computer Science and Applications, 7(8):136–144, 2016

    2016

  3. [11]

    The future landscape of large language models in medicine

    Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. The future landscape of large language models in medicine....

    2023

  4. [12]

    Two-layer retrieval- augmented generation framework for low-resource medical question answering using reddit data: Proof-of-concept study

    Sudeshna Das, Yao Ge, Yuting Guo, Swati Rajwal, JaMor Hairston, Jeanne Powell, Drew Walker, Snigdha Peddireddy, Sahithi Lakamana, Selen Bozkurt, et al. Two-layer retrieval- augmented generation framework for low-resource medical question answering using reddit data: Proof-of-c...

    2025

  5. [13]

    From local to global: A graph rag approach to query-focused summarization, 2025

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization, 2025

    2025

  6. [14]

    Ddxplus: A new dataset for automatic medical diagnosis

    Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. Ddxplus: A new dataset for automatic medical diagnosis. Advances in neural information processing systems, 35:31306–31318, 2022

    2022

  7. [15]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1, 2023

    2023 arXiv

  8. [16]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  9. [17]

    Dialmed: A dataset for dialogue-based medication recommendation

    Zhenfeng He, Yuqiang Han, Zhenqiu Ouyang, Wei Gao, Hongxu Chen, Guandong Xu, and Jian Wu. Dialmed: A dataset for dialogue-based medication recommendation. arXiv preprint arXiv:2203.07094, 2022

    2022 arXiv

  10. [18]

    Rag and rau: A survey on retrieval-augmented language model in natural language processing

    Yucheng Hu and Yuxing Lu. Rag and rau: A survey on retrieval-augmented language model in natural language processing. arXiv preprint arXiv:2404.19543, 2024

    2024 arXiv

  11. [19]

    Tc-rag: Turing-complete rag’s case study on medical llm systems

    Xinke Jiang, Yue Fang, Rihong Qiu, Haoyu Zhang, Yongxin Xu, Hao Chen, Wentao Zhang, Ruizhe Zhang, Yuchen Fang, Xu Chu, et al. Tc-rag: Turing-complete rag’s case study on medical llm systems. arXiv preprint arXiv:2408.09199, 2024

    2024 arXiv

  12. [20]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021

    2021

  13. [21]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023

    2023 arXiv

  14. [22]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems...

    2020

  15. [23]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  16. [24]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  17. [25]

    Biomedical knowledge graph: A survey of domains, tasks, and real-world applications

    Yuxing Lu, Sin Yee Goi, Xukai Zhao, and Jinzhuo Wang. Biomedical knowledge graph: A survey of domains, tasks, and real-world applications. arXiv preprint arXiv:2501.11632, 2025

    2025 arXiv

  18. [26]

    Clinicalrag: Enhancing clinical decision support through heterogeneous knowledge retrieval

    Yuxing Lu, Xukai Zhao, and Jinzhuo Wang. Clinicalrag: Enhancing clinical decision support through heterogeneous knowledge retrieval. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 64–68, 2024

    2024

  19. [27]

    Rjua-qa: A comprehensive qa dataset for urology, 2023

    Shiwei Lyu, Chenfei Chi, Hongbo Cai, Lei Shi, Xiaoyan Yang, Lei Liu, Xiang Chen, Deng Zhao, Zhiqiang Zhang, Xianguo Lyu, Ming Zhang, Fangzhou Li, Xiaowei Ma, Yue Shen, Jinjie Gu, Wei Xue, and Yiran Huang. Rjua-qa: A comprehensive qa dataset for urology, 2023

    2023

  20. [28]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896, 2023. 11

    2023 arXiv

  21. [29]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  22. [30]

    Rag in health care: a novel framework for improving communication and decision-making by addressing llm limitations

    Karen Ka Yan Ng, Izuki Matsuba, and Peter Chengming Zhang. Rag in health care: a novel framework for improving communication and decision-making by addressing llm limitations. NEJM AI, 2(1):AIra2400380, 2025

    2025

  23. [31]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  24. [32]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  25. [33]

    A rag-based medical assistant especially for infectious diseases

    Mahimai Raja, E Yuvaraajan, et al. A rag-based medical assistant especially for infectious diseases. In 2024 International Conference on Inventive Computation Technologies (ICICT), pages 1128–1133. IEEE, 2024

    2024

  26. [34]

    Factalign: Fact-level hallucination detection and classification through knowledge graph align- ment

    Mohamed Rashad, Ahmed Zahran, Abanoub Amin, Amr Abdelaal, and Mohamed AlTantawy. Factalign: Fact-level hallucination detection and classification through knowledge graph align- ment. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024),...

    2024

  27. [35]

    Knowledge editing on black-box large language models

    Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Yutao Mou, Jinxu Zhao, and Weiran Xu. Knowledge editing on black-box large language models. arXiv preprint arXiv:2402.08631, 2024

    2024 arXiv

  28. [36]

    Biorag: A rag-llm framework for biological question reasoning

    Chengrui Wang, Qingqing Long, Meng Xiao, Xunxin Cai, Chengjun Wu, Zhen Meng, Xuezhi Wang, and Yuanchun Zhou. Biorag: A rag-llm framework for biological question reasoning. arXiv preprint arXiv:2408.01107, 2024

    2024 arXiv

  29. [37]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  30. [38]

    Task-oriented dialogue system for automatic diagnosis

    Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuan-Jing Huang, Kam-Fai Wong, and Xiang Dai. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pap...

    2018

  31. [39]

    Medical graph rag: Towards safe medical large language model via graph retrieval- augmented generation

    Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min Xu, Filippo Menolascina, and Vicente Grau. Medical graph rag: Towards safe medical large language model via graph retrieval- augmented generation. arXiv preprint arXiv:2408.04187, 2024

    2024 arXiv

  32. [40]

    Automedprompt: A new framework for optimizing llm medical prompts using textual gradients

    Sean Wu, Michael Koo, Fabien Scalzo, and Ira Kurtz. Automedprompt: A new framework for optimizing llm medical prompts using textual gradients. arXiv preprint arXiv:2502.15944, 2025

    2025 arXiv

  33. [41]

    Benchmarking retrieval-augmented generation for medicine

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024, pages 6233–6251, 2024

    2024

  34. [42]

    Im- proving retrieval-augmented generation in medicine with iterative follow-up questions

    Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, and Aidong Zhang. Im- proving retrieval-augmented generation in medicine with iterative follow-up questions. In Biocomputing 2025: Proceedings of the Pacific Symposium, pages 199–214. World Scientific, 2024

    2025

  35. [43]

    Optimizing generative ai by backpropagating language model feedback

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055):609–616, 2025. 12

    2025

  36. [44]

    Almanac—retrieval- augmented language models for clinical medicine

    Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R Dalal, Jennifer L Kim, Michael Moor, Robyn Fong, Curran Phillips, Kevin Alexander, Euan Ashley, et al. Almanac—retrieval- augmented language models for clinical medicine. Nejm ai, 1(2):AIoa2300068, 2024

    2024

  37. [45]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  38. [46]

    Medrag: Enhancing retrieval- augmented generation with knowledge graph-elicited reasoning for healthcare copilot

    Xuejiao Zhao, Siyan Liu, Su-Yin Yang, and Chunyan Miao. Medrag: Enhancing retrieval- augmented generation with knowledge graph-elicited reasoning for healthcare copilot. In Proceedings of the ACM on Web Conference 2025, pages 4442–4457, 2025

    2025

  39. [47]

    A00-B99 Certain infectious and parasitic diseases,

    Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, et al. Realm: Rag-driven enhancement of multimodal electronic health records analysis via large language models. arXiv preprint arXiv:2402.07016, 2024. 13 A Relat...

    2024 arXiv

  40. [48]

    Answer Generation / Prompt Update

    Ensure your refined answer is factually sound based on the context, empathetic, and easy for a patient to understand. 6. IMPORTANT: Your output must be ONLY 20 the refined medical answer itself. Do not include any preamble, conversational phrases, meta-commentary, or any text ...

  41. [49]

    **Topical corticosteroids** (e.g., 1% hydrocortisone) to reduce skin inflammation and itching

  42. [50]

    **NSAIDs** (e.g., ibuprofen) for joint pain and swelling

  43. [51]

    Avoid scratching to prevent secondary infection

    **Antihistamines** (e.g., cetirizine) to address allergic or pruritic components. Avoid scratching to prevent secondary infection. If symptoms persist or worsen, or if fever develops, seek urgent evaluation for possible bacterial infection (e.g., Erysipelas) or referral to a d...

  44. [52]

    Carefully read the ’Patient Query (q)’ and the ’Supporting Context (C)’

  45. [53]

    Critically evaluate the ’current answer (A)’ (provided below) against this information

  46. [54]

    Generate an improved and refined version of the ’current answer (A)’

  47. [55]

    Focus on addressing any shortcomings in the ’current answer (A)’ regarding accuracy, completeness, clarity, and direct relevance to the patient’s query

  48. [56]

    Ensure your refined answer is factually sound based on the context, empathetic, and easy for a patient to understand

  49. [57]

    really uncomfortable and concerning

    IMPORTANT: Your output must be ONLY the refined medical answer itself. Do not include any preamble, conversational phrases, meta-commentary, or any text other than the refined answer. Input Answer: It sounds like you may be experiencing a submandibular space infection, given t...

  50. [58]

    Add Infratemporal Space Infection to Differential Diagnosis: • Insert a sentence mentioning infratemporal space infection as another potential diagnosis, noting its presentation with swelling in temporal/masseter/parotid regions, as referenced in the context

  51. [59]

    • Add loss of appetite as a symptom more typical of acute suppurative parotitis, as noted in the context

    Clarify Distinctions Between Diagnoses: • Differentiate submandibular space infection and acute suppurative parotitis by specifying that high fever is more characteristic of acute suppurative parotitis (per context). • Add loss of appetite as a symptom more typical of acute su...

  52. [60]

    Maintain Existing Valid Information: • Retain all accurate details about concerning symptoms (fever, difficulty swallow- ing/breathing), treatment (antibiotics, drainage, imaging), and urgency (emer- gency evaluation)

  53. [61]

    Preserve Structure and Flow: 28 • Keep the original warning about rapid progression and complications (e.g., airway obstruction) but ensure the added distinctions between diagnoses are integrated smoothly

  54. [62]

    Avoid Overlap or Ambiguity: • Ensure the revised text clearly separates the symptoms and diagnostic clues for each condition (submandibular space infection, acute suppurative parotitis, infratemporal space infection) to reduce ambiguity

  55. [63]

    Gradient Answer (Patient):

    No Major Restructuring: • Do not alter the overall structure or tone of the answer; focus only on targeted additions and clarifications to address the critiques. Gradient Answer (Patient):

  56. [64]

    While your symptoms align with these conditions, other causes like dental infections or salivary gland blockages may also be considered during evaluation

    Clarify Symptom-Diagnosis Alignment: • Add a brief statement explaining that the described symptoms (infraorbital, masseter, parotid, and submandibular swelling/pain) are consistent with the suggested diagnoses (submandibular space infection or acute suppurative paroti- tis), ...

  57. [65]

    I understand this is uncomfortable and worrying, but seek- ing care now can help prevent complications and speed up recovery

    Address Patient Concerns with Reassurance: • Insert a sentence acknowledging the patient’s discomfort and concern, empha- sizing that prompt care improves outcomes. • Example addition: "I understand this is uncomfortable and worrying, but seek- ing care now can help prevent co...

  58. [66]

    Amoxicillin-clavulanate is often used because it targets bacteria common in oral infections. A CT scan helps determine if drainage is needed

    Explain Treatment Rationale: • Briefly justify the antibiotic choice (amoxicillin-clavulanate) by mentioning its broad-spectrum coverage for oral bacteria. • Clarify that imaging (CT) helps identify abscesses or infection spread, guiding treatment decisions. • Example addition...

  59. [67]

    why" behind immediate action by explicitly linking rapid pro- gression to risks like airway obstruction. • Example addition:

    Strengthen Actionability with Urgency: • Reinforce the "why" behind immediate action by explicitly linking rapid pro- gression to risks like airway obstruction. • Example addition: "These infections can worsen quickly, potentially affecting breathing, so don’t delay evaluation."

  60. [68]

    Maintain Structure: • Keep the original flow (diagnoses → concerning symptoms → treatment → action steps) but integrate the above revisions without redundancy

  61. [69]

    Focus on clarity and motivation to act

    Avoid Overloading: • Ensure additions are concise and do not overwhelm the patient with excessive detail. Focus on clarity and motivation to act. Updated Prompt: You are a medical consultation AI tasked with refining the ’current answer (A)’ to improve its accuracy, completene...

  62. [70]

    - Explicitly map key symptoms (e.g., fever, loss of appetite, swelling locations) to their most characteristic diagnoses using context-provided linkages

    **Comprehensive Differential Diagnosis** - Include ALL relevant conditions from the supporting context (e.g., submandibular space infection, acute suppurative parotitis, infratemporal space infection) and retain clinically justified differentials (e.g., dental abscess, salivar...

  63. [71]

    submandibular infection)

    **Diagnostic Distinctions** - Highlight distinguishing clinical features between similar conditions (e.g., high fever in acute suppurative parotitis vs. submandibular infection). - Avoid diagnostic overlap by clearly attributing symptoms to specific conditions (e.g., temporal/...

  64. [72]

    - Justify diagnostic/treatment choices by linking them directly to the patient’s symptoms (e.g., ¨CT scans are prioritized for multi-area swelling to localize infection¨)

    **Symptom-Diagnosis Alignment** - Clarify how the patient’s symptoms align with suggested diagnoses while briefly mentioning other plausible conditions. - Justify diagnostic/treatment choices by linking them directly to the patient’s symptoms (e.g., ¨CT scans are prioritized f...

  65. [73]

    - Maintain empathetic yet direct language, avoiding vague reassurances

    **Emotional Acknowledgment** - Use explicit validation (e.g., ¨This sounds really uncomfortable and concerning¨) to acknowl- edge patient distress. - Maintain empathetic yet direct language, avoiding vague reassurances

  66. [74]

    **Treatment Rationale** - Justify treatment choices (e.g., antibiotics like amoxicillin-clavulanate for oral bacteria) and tests (e.g., CT for abscess detection) with simple, patient-friendly explanations

  67. [75]

    **Urgency Reinforcement** - Clearly state why immediate evaluation is critical, citing risks like airway obstruction or rapid infection progression. - Confirm that localized swelling/pain in critical facial areas (e.g., infraorbital, submandibular) alone warrants urgent evalua...

  68. [76]

    - Integrate new diagnostic additions seamlessly without disrupting urgency warnings or treatment advice

    **Structural Guidance** - Maintain the logical sequence: diagnoses → red flags → treatment → action. - Integrate new diagnostic additions seamlessly without disrupting urgency warnings or treatment advice

  69. [77]

    - Prioritize anatomical precision and symptom-diagnosis alignment while avoiding unnecessary changes to accurate, clinically sound information

    **Preservation Clause** - Retain all valid existing content that aligns with clinical relevance, even if not explicitly mentioned in the context. - Prioritize anatomical precision and symptom-diagnosis alignment while avoiding unnecessary changes to accurate, clinically sound ...

  70. [78]

    track treatment response and report any new or changing symptoms

    **Gentle skincare**: Avoid harsh scrubs or irritants. 5. **Monitoring**: Track treatment response and report any new or changing symptoms. *To help narrow the cause**: - Have you noticed if the spots worsen with sun exposure or at certain times in your menstrual cycle? - Are y...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.