Pith. sign in

REVIEW 4 major objections 5 minor 64 references

A new benchmark shows that rare-disease diagnostic systems must be judged not only on whether the true disease ranks high, but also on whether they outrank plausible alternatives and on what evidence they use.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:00 UTC pith:2LDGYXPM

load-bearing objection A solid benchmark that adds evidence auditing to rare-disease diagnosis evaluation, but the clinical validity of its 'hard' confounders needs external support. the 4 major comments →

arxiv 2607.24878 v1 pith:2LDGYXPM submitted 2026-07-27 q-bio.QM cs.AIcs.LG

GraphRareBench: An Auditable Graph-Evidence Benchmark for Phenotype-Driven Rare-Disease Diagnosis

Guiling Guo , Jia Yang , Jiahao Xu , Shuyuan Zheng , Zhonghai Sun , Qiyuan Li This is my paper
classification q-bio.QM cs.AIcs.LG
keywords rare disease diagnosisphenotype-driven rankingbenchmarktarget-over-confounder accuracyhard confoundersHPO phenotype queriesevidence auditLLM agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GraphRareBench argues that phenotype-driven rare-disease diagnosis systems should not be evaluated solely by top-k retrieval of the true disease. It provides 2,365 ontology-derived cases with fixed candidate pools, graph-defined hard confounders, and source-linked evidence records, enabling audits of a system's ability to distinguish the target from close alternatives and to access relevant evidence. On a gene-component-disjoint test split, supervised rankers reach MRRs of 0.640–0.740 and target-over-confounder accuracies of 0.898–0.916, while two tool-using agents achieve statistically indistinguishable MRRs yet differ by 0.561 in target-evidence coverage. The paper shows that 22.1% to 43.7% of Hit@10 successes still have at least one hard confounder ranked above the target, indicating that full-pool retrieval, hard-confounder discrimination, and evidence access measure complementary failure modes.

Core claim

The central claim is that conventional top-k metrics mask clinically meaningful failures in phenotype-driven rare-disease diagnosis: a model can achieve Hit@10 while ranking a plausible graph-defined alternative above the true disease. To expose this, the authors construct GraphRareBench, a provenance-preserving benchmark of 2,365 cases and 18,093 target–confounder pairs built from HPO, MONDO, disease-family, and causal-gene relations, and propose target-over-confounder (ToC) metrics that measure whether the target outranks its hard confounders. They also record which evidence a tool-using model requests, allowing an evidence-access audit separate from final ranking. The results show that tw

What carries the argument

The benchmark's load-bearing elements are (1) the fixed candidate pool, split into a full pool and a hard pool containing only the target plus graph-defined hard confounders; (2) the coarsened HPO query, where 97.7% of terms are one ontology step up from the source phenotype, reducing direct phenotype matching while preserving provenance; and (3) the ToC metrics, which compute the fraction of target–confounder pairs where the target outranks the confounder. Hard confounders are mined deterministically through seven graph mechanisms: high HPO overlap, SapBERT semantic neighbors, shared MONDO parents, disease-family membership, shared causal-gene families, shared causal-gene components, and ex

Load-bearing premise

The load-bearing premise is that the graph-defined hard confounders—mined from HPO overlap, semantic neighbors, shared ontology parents, disease-family membership, and causal-gene relations—represent the clinically plausible alternatives that a diagnostic system should rule out; if they do not correspond to real differential-diagnosis difficulty, the target-over-confounder metrics lose clinical meaning.

What would settle it

Take a random sample of 200 target–confounder pairs from GraphRareBench and ask clinicians to rate whether each pair is a realistic differential-diagnosis dilemma. If a large fraction of the pairs are judged implausible as competing diagnoses, or if pairs without any graph edge are rated equally plausible, then the graph-defined difficulty assumption fails and the ToC metrics measure artifacts rather than clinical competence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If top-k scores are not supplemented with ToC metrics, failures where a plausible alternative outranks the true disease will remain invisible.
  • Tool-using agents with similar final ranks can have very different evidence-seeking behavior; trace-based evidence coverage should be part of the evaluation.
  • The fixed candidate pools and evidence interfaces allow controlled interventions to isolate whether a failure is due to missing evidence, poor retrieval, or poor integration.
  • GraphRareBench can serve as a regression suite for updates to models, retrievers, or knowledge bases across confounder mechanisms.
  • The gene-component-disjoint split tests generalization beyond the causal genes seen during training, a specificity not offered by most existing benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If real clinical differentials involve features not captured by ontology/semantic/gene relations (e.g., overlapping imaging, treatments, or age-of-onset patterns), the ToC scores may overstate or understate true diagnostic difficulty; this could be tested by comparing graph-defined confounders with clinician-selected alternatives.
  • The large evidence-coverage gap between the two agents suggests a testable hypothesis: systems that access more target-specific evidence before ranking may be more robust to perturbations like candidate-order scrambling or restricted tool budgets.
  • The benchmark format could be extended to a phenotype-disjoint split (no shared HPO terms between train and test) to probe generalization across phenotype space rather than just gene components.
  • Because evidence is source-linked, GraphRareBench could be adapted to measure shortcut reliance: a model that ranks correctly without querying the evidence tool may be relying on memorized disease–phenotype associations, which the framework makes detectable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces GraphRareBench, a provenance-preserving benchmark for phenotype-driven rare-disease diagnosis. It constructs 2,365 ontology-derived cases with coarsened HPO queries, fixed candidate pools, graph-defined hard confounders, and source-linked evidence records. The benchmark is split by causal-gene components to reduce leakage, yielding 237 test cases. The authors evaluate negative controls, phenotype-driven tools (Exomiser, LIRICAL, HPO Similarity), a prompted LLM (DeepSeek-V4-Flash), supervised graph-evidence rankers (RAG2D, PPP, LPP) sharing a 21-feature interface, and tool-using agents. They report MRR, Hit@k, and target-over-confounder (ToC) metrics, and find that full-pool retrieval, hard-confounder discrimination, and evidence-access behavior capture complementary aspects of model performance. The headline results include MRRs of 0.640–0.740 for supervised rankers, ToCcase of 0.898–0.916, and an agent MRR difference of 0.029 that is not statistically significant while evidence coverage differs by 0.561.

Significance. If the construct validity of the 'hard confounder' labels is accepted, GraphRareBench would fill an important gap by adding differential-diagnosis-focused metrics and auditable evidence traces to phenotype-driven ranking benchmarks. The paper is careful in several respects: it performs a label-leakage screen, uses a gene-component-disjoint split, mines confounders deterministically, reports bootstrap confidence intervals for MRR, checks tail-ranking sensitivity, and provides deterministic compliance checks for agent tool use. These controls strengthen the internal validity of the comparisons. The main uncertainty is external: the graph-defined hard confounders are not shown to correspond to clinically plausible differential diagnoses, which is load-bearing for the interpretation of ToC and the complementarity claim. The omission of the 21-feature interface specification also limits reproducibility and prevents assessment of a possible circularity between feature construction and confounder-label generation.

major comments (4)
  1. [GraphRareBench Construction / Limitations] The label 'hard' is operationalized by seven graph/semantic relations (HPO overlap, SapBERT, MONDO parents, disease-family membership, causal-gene relations). The abstract and Q1 interpretation use ToC and the 22.1–43.7% Hit@10 failures to draw conclusions about 'clinically plausible alternatives.' However, no evidence shows that these graph-defined alternatives correspond to real differential-diagnosis difficulty; the Limitations only concede that the relations 'may not cover every distinction encountered in clinical practice.' This is a construct-validity problem, not merely a coverage gap. Please add a validation against curated differential diagnoses (e.g., OMIM/Orphanet differential fields or expert review of a sampled set of pairs). Without this, ToC should be described as 'graph-defined alternative ordering,' and the complementarity claim should be correspondingly qualified.
  2. [Compared methods / Table 2] The supervised rankers RAG2D/PPP/LPP are defined by their 'shared 21-feature interface,' and Q4's score-time feature-channel isolation (Table 2) is central to the evidence-audit claim. Yet the main text never defines the 21 features; Table 2 only names feature groups such as 'case-specific evidence' and 'aggregate graph scores.' Without a precise list (which HPO-overlap features, semantic similarity scores, graph-distance features, gene-context features; how they are normalized; which resources they consume), the supervised results are not reproducible and the circularity analysis below cannot be performed. Please provide the full feature specification in the main text or a named appendix.
  3. [Q3 / GraphRareBench Construction] ToC for the supervised rankers is at risk of being partially self-fulfilling. Hard confounders are mined using HPO overlap, SapBERT semantic similarity, shared MONDO parents, disease-family membership, and causal-gene relations; the supervised models are trained on graph-evidence features plausibly derived from the same relations. If so, the reported ToC values of 0.898–0.916 may reflect, in part, learning the label-generation rule rather than a general differential-diagnosis ability. The complementarity conclusion is partly supported by unsupervised HPO Similarity and agent evidence, so the issue is not fatal to all claims. Still, the paper should either ablate feature channels that correspond to confounder-defining relations or evaluate on a held-out confounder mechanism to show the ToC result is not an artifact of label-feature overlap.
  4. [Experimental Protocol / Table 3] The agent audit is based on one fixed protocol run per agent. The main text acknowledges this, but the Abstract reports 'their target-evidence coverage differed by 0.561' without the fixed-run caveat. Because the evidence-access component of the complementarity claim rests on this single run, the observed tool-use differences could reflect decoding variability, prompt sensitivity, or candidate order rather than stable agent properties. Please provide repeated-run variance estimates (e.g., multiple decoding seeds/temperatures) or explicitly present these results as an illustrative case study and soften the corresponding conclusion in the Abstract and Discussion.
minor comments (5)
  1. [Abstract] Typo: 'avaliable' should be 'available'.
  2. [Table 1] Bootstrap confidence intervals are reported only for MRR. Since ToCcase is a central metric, please report CIs for ToCcase as well.
  3. [Figure 3] The reason-sliced ToCpair values are presented without confidence intervals, and some slices are small (e.g., semantic slice has 100 pairs). Please show case-clustered CIs or at least give the pair counts in the figure itself.
  4. [References] A few citations appear malformed: 'AlDin, Z. E.' should likely be 'Al Din, Z. E.', and 'Ma, G.; NM, B.' appears to have a formatting artifact. Please check the reference list for consistency.
  5. [Q4 / Table 3] The tool-use differences (evidence calls, switches, coverage) are descriptive and no formal multiple-comparison correction is applied. Since the primary comparison of MRR is not significant, the coverage differences should be explicitly labeled as exploratory.

Circularity Check

0 steps flagged

No significant circularity: benchmark construction and evaluation are self-contained; feature-label coupling and clinical-validity caveats are limitations, not definitional reductions.

full rationale

GraphRareBench is a benchmark construction and evaluation paper rather than a derivation that predicts its own inputs. The central complementarity claim—that full-pool retrieval, hard-confounder discrimination, and evidence access are non-redundant—rests on contrastive empirical results: HPO Similarity shows low MRR (0.170) but high ToCcase (0.887); supervised rankers share a 21-feature interface yet differ in MRR and ToC; and the two agents have statistically tied MRR (difference 0.029, 95% CI includes zero) while differing greatly in target-evidence coverage (0.561). None of these contrasts is a fitted parameter renamed as a prediction. The hard confounders are defined by graph relations, and the supervised rankers use graph-evidence features, so their ToC values can partly reflect the same graph signal; however, the paper does not fit ToC itself, uses a gene-component-disjoint held-out split, and explicitly frames supervised results as interface-level comparisons. The interpretation that graph-defined confounders are 'clinically plausible alternatives' is an external-validity assumption acknowledged in Limitations ('these relations ... may not cover every distinction encountered in clinical practice'), which is a validity risk rather than circularity. Existing self-citations (e.g., Guo et al. 2026, KGRD) are descriptive and not load-bearing. No equation in the paper reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on the benchmark's construction choices: thresholds, candidate caps, feature design, and the validity of ontology-based similarity as a proxy for clinical difficulty. These are not derived from first principles; they are engineering decisions.

free parameters (3)
  • Hard-confounder mining thresholds
    Thresholds for 'high HPO overlap' and SapBERT semantic similarity are not given in the main text; they determine the difficulty and composition of hard confounders, directly affecting ToC scores.
  • Candidate pool caps = median 69 full-pool candidates per test case
    The cap on the number of additional non-hard candidates is a design choice; larger pools lower MRR, so the reported MRR values depend on this construction parameter.
  • Composition of the 21-feature interface
    The hand-selected graph-evidence features used by RAG2D, PPP, and LPP are not enumerated; they define the information available to supervised rankers and thus their achievable performance.
axioms (4)
  • domain assumption HPO annotations are accurate and complete for the 2,365 retained diseases
    The benchmark's ground truth and queries are built from HPO disease annotations; errors or sparsity here would propagate to all cases.
  • domain assumption Ontology structure (HPO/MONDO) and gene-component relations are valid proxies for clinical similarity
    Hard confounders are defined via these graph relations; the clinical meaningfulness of ToC metrics depends on this.
  • domain assumption Gene-component-disjoint splitting prevents information leakage relevant to ranking
    The partition assumes shared causal-gene components are the main leakage channel; other latent similarities could still transfer.
  • domain assumption Source-linked evidence records faithfully represent evidence available in practice
    The evidence tool returns constructed records; if these diverge from real clinical evidence, the trace audit conclusions may not transfer.

pith-pipeline@v1.3.0-alltime-deepseek · 11415 in / 13270 out tokens · 107194 ms · 2026-07-31T23:00:34.026097+00:00 · methodology

0 comments
read the original abstract

Phenotype-driven diagnostic benchmarks usually report the rank of the reference disease, but they rarely reveal which plausible alternatives are ranked above it or what evidence a tool-using model examines before making its decision. We introduce GraphRareBench, a provenance-preserving benchmark containing 2,365 ontology-derived cases and 18,093 target-confounder pairs. Each case includes a coarsened HPO query, a fixed candidate pool, graph-defined hard confounders, and source-linked evidence records. On the 237-case gene-component-disjoint test split, supervised rankers using a shared 21-feature interface achieved MRRs ranging from 0.640 to 0.740 and case-averaged target-over-confounder accuracies ranging from 0.898 to 0.916. Agents instantiated with Agents-A1 and DeepSeek-V4-Flash achieved MRRs of 0.746 and 0.718, respectively. Their paired MRR difference was not statistically significant, whereas their target-evidence coverage differed by 0.561. Together with the observation that 22.1% to 43.7% of selected Hit@10 successes still ranked at least one graph-defined hard confounder above the target, these results indicate that full-pool retrieval, hard-confounder discrimination, and observable evidence access capture complementary aspects of model behavior. GraphRareBench therefore provides a foundation for more transparent and evidence-aware evaluation of phenotype-driven diagnostic systems. Code and data are available at https://github.com/GUI0609/GraphRareBench.

Figures

Figures reproduced from arXiv: 2607.24878 by Guiling Guo, Jiahao Xu, Jia Yang, Qiyuan Li, Shuyuan Zheng, Zhonghai Sun.

Figure 1
Figure 1. Figure 1: GraphRareBench converts curated rare-disease resources into ontology-derived ranking cases with fixed candidate pools, graph [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Frozen-release scale and controlled expansion capacity. The current benchmark contains 2,365 fixed cases, of which [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reason-sliced target-over-confounder heatmap on the test partition. Entries are ToC [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 2 linked inside Pith

  1. [1]

    Nature protocols , volume=

    Next-generation diagnostics and disease-gene discovery with the Exomiser , author=. Nature protocols , volume=. 2015 , publisher=

    2015

  2. [2]

    Science translational medicine , volume=

    AMELIE speeds Mendelian diagnosis by matching patient phenotype and genotype to primary literature , author=. Science translational medicine , volume=. 2020 , publisher=

    2020

  3. [3]

    NAR genomics and Bioinformatics , volume=

    Phen2Gene: rapid phenotype-driven gene prioritization for rare diseases , author=. NAR genomics and Bioinformatics , volume=. 2020 , publisher=

    2020

  4. [4]

    NAR Genomics and Bioinformatics , volume=

    CADA: phenotype-driven gene prioritization based on a case-enriched knowledge graph , author=. NAR Genomics and Bioinformatics , volume=. 2021 , publisher=

    2021

  5. [5]

    BMC medical genomics , volume=

    Phenotype-driven gene prioritization for rare diseases using graph convolution on heterogeneous networks , author=. BMC medical genomics , volume=. 2018 , publisher=

    2018

  6. [6]

    NPJ digital medicine , volume=

    Few shot learning for phenotype-driven diagnosis of patients with rare genetic diseases , author=. NPJ digital medicine , volume=

  7. [7]

    npj Digital Medicine , year=

    KGRD: a knowledge-graph-augmented automated reasoning framework for diagnosis and counselling of paediatric rare genetic disorders , author=. npj Digital Medicine , year=

  8. [8]

    Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies , pages=

    Self-alignment pretraining for biomedical entity representations , author=. Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies , pages=

    2021

  9. [9]

    Genetics in Medicine , volume=

    Exome and genome sequencing for pediatric patients with congenital anomalies or intellectual disability: an evidence-based clinical guideline of the American College of Medical Genetics and Genomics (ACMG) , author=. Genetics in Medicine , volume=. 2021 , publisher=

    2021

  10. [10]

    Communications of the ACM , volume=

    Datasheets for datasets , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  11. [11]

    Nature Machine Intelligence , volume=

    Shortcut learning in deep neural networks , author=. Nature Machine Intelligence , volume=. 2020 , publisher=

    2020

  12. [12]

    Patterns , volume=

    Leakage and the reproducibility crisis in machine-learning-based science , author=. Patterns , volume=. 2023 , publisher=

    2023

  13. [13]

    Nucleic Acids Res [Internet] , volume=

    The Human Phenotype Ontology in 2024: phenotypes around the world , author=. Nucleic Acids Res [Internet] , volume=

    2024

  14. [14]

    Human mutation , volume=

    Representation of rare diseases in health information systems: the Orphanet approach to serve a wide range of end users , author=. Human mutation , volume=. 2012 , publisher=

    2012

  15. [15]

    Amberger, Joanna S and Bocchini, Carol A and Schiettecatte, Fran. OMIM. org: Online Mendelian Inheritance in Man (OMIM. Nucleic acids research , volume=. 2015 , publisher=

    2015

  16. [16]

    Nature genetics , volume=

    PanelApp crowdsources expert knowledge to establish consensus diagnostic gene panels , author=. Nature genetics , volume=. 2019 , publisher=

    2019

  17. [17]

    Genetics in Medicine , volume=

    The Gene Curation Coalition: A global effort to harmonize gene--disease evidence resources , author=. Genetics in Medicine , volume=. 2022 , publisher=

    2022

  18. [18]

    Genetics in Medicine , volume=

    The Clinical Genome Resource (ClinGen): advancing genomic knowledge through global curation , author=. Genetics in Medicine , volume=. 2025 , publisher=

    2025

  19. [19]

    Nucleic Acids Research , volume=

    ClinVar: updates to support classifications of both germline and somatic variants , author=. Nucleic Acids Research , volume=. 2025 , publisher=

    2025

  20. [20]

    Nucleic acids research , volume=

    The unified medical language system (UMLS): integrating biomedical terminology , author=. Nucleic acids research , volume=. 2004 , publisher=

    2004

  21. [21]

    Advances in neural information processing systems , volume=

    Translating embeddings for modeling multi-relational data , author=. Advances in neural information processing systems , volume=

  22. [22]

    International conference on machine learning , pages=

    Complex embeddings for simple link prediction , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  23. [23]

    European semantic web conference , pages=

    Modeling relational data with graph convolutional networks , author=. European semantic web conference , pages=. 2018 , organization=

    2018

  24. [24]

    Scientific data , volume=

    Building a knowledge graph to enable precision medicine , author=. Scientific data , volume=. 2023 , publisher=

    2023

  25. [25]

    Nature Medicine , volume=

    A foundation model for clinician-centered drug repurposing , author=. Nature Medicine , volume=. 2024 , publisher=

    2024

  26. [26]

    Rationale-guided retrieval augmented generation for medical question answering , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  27. [27]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Medical graph rag: Evidence-based medical large language model via graph retrieval-augmented generation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  28. [28]

    Nature , volume=

    An agentic system for rare disease diagnosis with traceable reasoning , author=. Nature , volume=. 2026 , publisher=

    2026

  29. [29]

    Nature , volume=

    Towards accurate differential diagnosis with large language models , author=. Nature , volume=. 2025 , publisher=

    2025

  30. [30]

    Advances in neural information processing systems , volume=

    Ddxplus: A new dataset for automatic medical diagnosis , author=. Advances in neural information processing systems , volume=

  31. [31]

    The Lancet Digital Health , volume=

    RareArena: a comprehensive benchmark dataset unveiling the potential of large language models in rare disease diagnosis , author=. The Lancet Digital Health , volume=. 2026 , publisher=

    2026

  32. [32]

    JAMA Network Open , volume=

    Large language model performance and clinical reasoning tasks , author=. JAMA Network Open , volume=. 2026 , publisher=

    2026

  33. [33]

    bioRxiv , pages=

    Benchmarking the Impact of Data Leakage on the Performance of Knowledge Graph Embedding Models for Biomedical Link Prediction , author=. bioRxiv , pages=. 2025 , publisher=

    2025

  34. [34]

    ACM Transactions on Information Systems (TOIS) , volume=

    Cumulated gain-based evaluation of IR techniques , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2002 , publisher=

    2002

  35. [35]

    Proceedings of the sixteenth ACM conference on Conference on information and knowledge management , pages=

    A comparison of statistical significance tests for information retrieval evaluation , author=. Proceedings of the sixteenth ACM conference on Conference on information and knowledge management , pages=

  36. [36]

    Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    Reliable confidence intervals for information retrieval evaluation using generative ai , author=. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

  37. [37]

    arXiv preprint arXiv:1806.08295 , year=

    How many random seeds? statistical power analysis in deep reinforcement learning experiments , author=. arXiv preprint arXiv:1806.08295 , year=

    Pith/arXiv arXiv

  38. [38]

    Nucleic acids research , volume=

    The human phenotype ontology in 2021 , author=. Nucleic acids research , volume=. 2021 , publisher=

    2021

  39. [39]

    The American Journal of Human Genetics , volume=

    Clinical diagnostics in human genetics with semantic similarity searches in ontologies , author=. The American Journal of Human Genetics , volume=. 2009 , publisher=

    2009

  40. [40]

    International journal of medical informatics , volume=

    FindZebra: a search engine for rare diseases , author=. International journal of medical informatics , volume=. 2013 , publisher=

    2013

  41. [41]

    Frontiers in genetics , volume=

    RDAD: a machine learning system to support phenotype-based rare disease diagnosis , author=. Frontiers in genetics , volume=. 2018 , publisher=

    2018

  42. [42]

    Genetics in Medicine , volume=

    Computational evaluation of exome sequence data using human and model organism phenotypes improves diagnostic efficiency , author=. Genetics in Medicine , volume=. 2016 , publisher=

    2016

  43. [43]

    Genes , volume=

    An improved phenotype-driven tool for rare mendelian variant prioritization: benchmarking exomiser on real patient whole-exome data , author=. Genes , volume=. 2020 , publisher=

    2020

  44. [44]

    The American Journal of Human Genetics , volume=

    Interpretable clinical genomics with a likelihood ratio paradigm , author=. The American Journal of Human Genetics , volume=. 2020 , publisher=

    2020

  45. [45]

    JMIR Medical Informatics , volume=

    An automatic and end-to-end system for rare disease knowledge graph construction based on ontology-enhanced large language models: development study , author=. JMIR Medical Informatics , volume=. 2024 , publisher=

    2024

  46. [46]

    Diagnostics , volume=

    ZebraMap: A Multimodal Rare Disease Knowledge Map with Automated Data Aggregation & LLM-Enriched Information Extraction Pipeline , author=. Diagnostics , volume=. 2025 , publisher=

    2025

  47. [47]

    Human Genetics and Genomics Advances , volume=

    A corpus of GA4GH Phenopackets: case-level phenotyping for genomic diagnostics and discovery , author=. Human Genetics and Genomics Advances , volume=. 2025 , publisher=

    2025

  48. [48]

    BMC bioinformatics , volume=

    Towards a standard benchmark for phenotype-driven variant and gene prioritisation algorithms: PhEval-Phenotypic inference Evaluation framework , author=. BMC bioinformatics , volume=. 2025 , publisher=

    2025

  49. [49]

    Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining , pages=

    RareBench: can LLMs serve as rare diseases specialists? , author=. Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining , pages=

  50. [50]

    Nature Communications , volume=

    Simulation of undiagnosed patients with novel genetic conditions , author=. Nature Communications , volume=. 2023 , publisher=

    2023

  51. [51]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    RareSyn: Health Record Synthesis for Rare Disease Diagnosis , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  52. [52]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Rareagents: Autonomous multi-disciplinary team for rare disease diagnosis and treatment , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  53. [53]

    Genetics , volume=

    Mondo: integrating disease terminology across communities , author=. Genetics , volume=. 2026 , publisher=

    2026

  54. [54]

    The American Journal of Human Genetics , volume=

    Assessing the utility of large language models for phenotype-driven gene prioritization in the diagnosis of rare genetic disease , author=. The American Journal of Human Genetics , volume=. 2024 , publisher=

    2024

  55. [55]

    European Journal of Human Genetics , volume=

    Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools , author=. European Journal of Human Genetics , volume=. 2026 , publisher=

    2026

  56. [56]

    arXiv preprint arXiv:2408.08422 , year=

    Assessing and enhancing large language models in rare disease question-answering , author=. arXiv preprint arXiv:2408.08422 , year=

    arXiv

  57. [57]

    JAMA network open , volume=

    Large language models for rare disease diagnosis at the undiagnosed diseases network , author=. JAMA network open , volume=

  58. [58]

    arXiv preprint arXiv:2601.11559 , year=

    MIMIC-RD: Can LLMs differentially diagnose rare diseases in real-world clinical settings? , author=. arXiv preprint arXiv:2601.11559 , year=

    arXiv

  59. [59]

    NPJ Digital Medicine , volume=

    A phenotype-based AI pipeline outperforms human experts in differentially diagnosing rare diseases using EHRs , author=. NPJ Digital Medicine , volume=. 2025 , publisher=

    2025

  60. [60]

    International Conference on Learning Representations , volume=

    Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume=

  61. [61]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  62. [62]

    2024 , eprint=

    t-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. 2024 , eprint=

    2024

  63. [63]

    2026 , howpublished =

    2026

  64. [64]

    arXiv preprint arXiv:2606.30616 , year=

    Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent , author=. arXiv preprint arXiv:2606.30616 , year=

    Pith/arXiv arXiv