Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models mostly generate untruthful biomedical hypotheses, but a claim-level groundedness score can pick out the truthful ones.

desk verdict Useful benchmark plus a groundedness filter whose main result is compromised by an unspecified knowledge-graph split; worth refereeing but not accepting as-is. read the letter →

arxiv 2505.14599 v2 pith:AGIYIAGP submitted 2025-05-20 cs.CL cs.AI

Guangzhi Xiong , Eric Xie , Corey Williams , Myles Kim , Amir Hassan Shariatmadari , Sikun Guo , Stefan Bekiranov , Aidong Zhang This is my paper
classification cs.CLcs.AI
keywords scientifichypothesisgenerationhallucinationdetectiongroundednessbiomedicalknowledgegraphTruthHypoKnowHDlargelanguagemodelsretrieval-augmented
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models, when asked to hypothesize relations between genes, chemicals, and diseases, mostly produce statements that do not match the relations recorded in later-published biomedical knowledge. To make this testable, the authors build TruthHypo, a benchmark that splits the PubTator 3.0 knowledge graph into relations known before 2023 and relations discovered after 2024, and ask LLMs to hypothesize the unseen relations. They then propose KnowHD, a detector that decomposes each hypothesis and its reasoning into atomic claims and scores groundedness as the fraction of claims entailed by retrieved literature and knowledge-graph context. The paper shows a positive correlation between groundedness score and hypothesis truthfulness, and that selecting the candidate with the highest groundedness score outperforms greedy search and majority voting when external knowledge is available.

What carries the argument

The central objects are TruthHypo, a benchmark built from the PubTator 3.0 biomedical knowledge graph by splitting relations into a 'seen' subset (papers with PMID up to 36,600,000, before 2023) and an 'unseen' subset (PMID at least 38,200,000, after 2024), with edges sharing head and tail entities removed and only multi-article relations retained; and KnowHD, a hallucination detector that parses a generated hypothesis and its rationale into atomic claims, retrieves context for each claim from a literature corpus via BM25 (with eight documents, restricted to PMID up to 36,600,000) and/or from a knowledge graph via contextG(p) = {(eh, r, et) in G | {eh, et} subset of V(p)}, then asks an LLM whether the context entails the claim. The groundedness of a hypothesis h is the average fraction 1/|C(h)| times the sum over claims p of the indicator 1[context(p) entails p]. This decomposition into atomic claims is what lets the detector separate supported reasoning steps from unsupported ones, and the scalar groundedness score is what drives the filtering result.

What would settle it

Take the TruthHypo test relations and recompute KnowHD groundedness scores twice: once with the knowledge graph restricted to edges whose source PMID is at most 36,600,000, and once with the unrestricted graph. If the accuracy-versus-groundedness correlation and the selection gains over greedy and majority baselines disappear in the restricted condition, the reported filtering effect is an artifact of temporal leakage rather than genuine hallucination detection.

Watch

Extended reading notes

Core claim

The paper's central claim is that KnowHD's groundedness score—the fraction of atomic claims in a hypothesis's rationale that are entailed by retrieved literature and knowledge-graph context—is an effective metric for filtering truthful hypotheses from the diverse outputs of LLMs. The authors operationalize this by prompting each model to generate five candidate hypotheses per entity pair and returning the one with the highest groundedness score, which improves relation-level accuracy over greedy decoding and self-consistency majority voting in the knowledge-augmented settings. The paper also establishes, via TruthHypo, that current LLMs generate truthful hypotheses at low rates: the best model, GPT-4o, exceeds 60% average accuracy only in some settings, and smaller models often fall to near chance. The correlation between groundedness and accuracy is shown by binning hypotheses by groundedness score and plotting mean accuracy per bin, with higher-groundedness groups achieving higher accuracy, for example 72.77% accuracy for Chemical & Gene hypotheses with groundedness above 80% versus 60.96% overall.

Load-bearing premise

The filtering result assumes that the knowledge sources used to compute groundedness contain only knowledge available before the hypothesis's ground-truth relation was discovered; the paper enforces this cutoff for literature but does not explicitly enforce it for the knowledge graph, so the correlation could partly come from a hypothesis being grounded by the very relation it is asked to predict.

Editorial extensions

If this is right

  • KnowHD can be used as a post-hoc filter on LLM-generated hypotheses, improving relation-level accuracy over greedy and majority-voting baselines when external knowledge is available.
  • Groundedness scores provide a quantitative signal that correlates with human expert and GPT-4o judgments of which hypothesis in a pair is more truthful.
  • Larger LLMs make better use of external knowledge: GPT-4o improves by 5.14 percentage points in accuracy when augmented with a knowledge graph and literature, while GPT-4o-mini improves by only 1.14 points.
  • LLMs currently identify plausible links between entities (high link-level F1) but fail at relation-level accuracy, so hallucination concentrates in the specific relation label rather than in the existence of an association.
  • Knowledge graphs and literature provide complementary grounding evidence, with the combined KG plus literature setting yielding the highest groundedness scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the KnowHD groundedness-truthfulness correlation holds under strict temporal separation of the knowledge graph, the method could be extended to rank hypotheses in other sciences where a temporally split knowledge base exists, such as materials science or astronomy.
  • The paper's grounding score is an estimate of entailment produced by an LLM judge; a natural stress test would replace the judge with a smaller or differently trained verifier to see whether the filtering benefit survives.
  • The authors explicitly restrict literature retrieval to pre-2023 papers but do not state the same restriction for the knowledge graph in KnowHD; this raises the possibility that some groundedness scores are inflated by the very edges the model is asked to predict, a concern that should be checked before adopting KnowHD as a general filter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces TruthHypo, a benchmark for evaluating whether LLM-generated biomedical hypotheses are truthful, built from the PubTator 3.0 knowledge graph with a temporal split into 'seen' (pre-2023) and 'unseen' (post-2024) relations. The authors also propose KnowHD, a hallucination detector that decomposes a hypothesis and its rationale into atomic claims and scores each claim's groundedness against a knowledge graph, retrieved literature, or both. On TruthHypo, the authors report that LLMs, especially smaller ones, struggle to generate truthful hypotheses, and that selecting hypotheses with the highest KnowHD groundedness scores improves accuracy over greedy and majority-voting baselines. A human study on open-ended hypothesis generation further suggests that experts and GPT-4o prefer highly-grounded hypotheses over lowly-grounded ones.

Significance. If the central claims hold, the paper makes a useful contribution: it provides a reusable benchmark with a principled temporal split, a concrete operationalization of hypothesis groundedness, and a systematic comparison of several LLMs across four knowledge settings. The release of data and code is a strength, as is the inclusion of a human study on an open-ended task. The significance is currently conditional on resolving a temporal-leak concern in the KnowHD graph grounding, because the main empirical claims—the positive groundedness-accuracy correlation and the filtering improvement—could be partly manufactured if the grounding graph includes the test edges themselves.

major comments (3)
  1. [Section 3, Eq. (2)] The knowledge graph G used in Eq. (2) is never explicitly restricted to the pre-2023 'seen' subset, unlike the literature context in Eq. (1), which is explicitly limited to PMID ≤ 36600000. Since the TruthHypo test set consists of post-2024 'unseen' edges, if G is the full PubTator 3.0 graph, then a correct hypothesis about an unseen relation will be grounded by the very edge it predicts, because contextG(p) includes all edges whose head and tail entities appear in the claim. This would manufacture the positive groundedness-truthfulness correlation in Figure 3 and the filtering improvements in Figure 4, and it would also contaminate the human-study grouping in Section 5. The authors must either state that G in Eq. (2) is restricted to the seen graph (and confirm this in the released code) or repeat the entire KnowHD analysis with a G that provably excludes the unseen test edges.
  2. [Section 2.1 and Section 2.3] The ground-truth labels in TruthHypo are defined by KG membership: positive test instances are unseen edges in PubTator 3.0, and negative 'no relation' instances are entity pairs that lack a direct relation in the KG. This equates absence from PubTator with the absence of a biological relation, which is a strong assumption given that PubTator is an automatically extracted, coverage-limited resource. The paper should explicitly discuss this limitation and, ideally, provide a small-scale human validation of a sample of positive and negative labels to demonstrate that the benchmark captures more than KG coverage.
  3. [Section 3, Eq. (3) and Figure 12 template] The entailment judgment in Eq. (3), 1[context(p) |= p], is implemented by prompting an LLM with no reported human validation or inter-annotator agreement for these per-claim judgments. The human study in Section 5 validates that experts prefer hypotheses with high KnowHD scores, but it does not validate that KnowHD's underlying claim-level entailment labels are accurate. Since the central claim is that KnowHD is an effective hallucination detector, the paper should include an evaluation of the detector's own reliability, for example by comparing KnowHD's claim-level groundedness judgments against human annotations on a sample of claims, or by reporting agreement statistics for the LLM judge.
minor comments (4)
  1. [Equation (3), Table 3, Table 4] The word 'groundedness' is misspelled as 'groudedness' in Eq. (3), Table 3, and Table 4; this should be corrected throughout.
  2. [Table 2] The 'Knowledge' column in Table 2 contains unexplained citations ([Wei et al., 2022], [Baek et al., 2024], [Lewis et al., 2020]) that appear to refer to the augmentation methods; Section 4.1 does not introduce these references, so the table should either cite the actual method names or remove the citations.
  3. [Figure 3] Figure 3 groups hypotheses by groundedness score but does not state the bin width, the range of sample sizes per bin, or the number of hypotheses excluded by the 'no less than 10' rule; reporting these details would help readers judge the stability of the trend.
  4. [Section 4.1] The claim that 'All LLMs were trained on the knowledge available before 2024, preventing recall of the exact knowledge' is imprecise: GPT-4o's training cutoff is not publicly documented in the paper, and 'preventing recall' is too strong for any model with broad web pretraining. A softer formulation would be more accurate.

Circularity Check

1 steps flagged · score 8.0 of 10

KnowHD's KG context (Eq. 2) is drawn from the full PubTator graph, whose unseen edges are the ground-truth labels, so groundedness predicts truthfulness by construction.

  1. self definitional [Section 2.1; Section 3, Eq. (2); Appendix A]
    "The 'unseen' subset, designed to represent new discoveries, comprises relations extracted from papers published after 2024, identified by PMIDs≥ 38200000. ... When using a knowledge graph G as the knowledge base, the context for a claim is derived from the graph structure: contextG(p)={(eh, r, et)∈ G | {eh, et} ⊆ V(p)}. ... each mention was matched to its unified representation in the PubTator 3.0 knowledge graph."

    TruthHypo's ground truth is the 'unseen' post-2024 subset of PubTator 3.0. Appendix A aligns KnowHD's entity matching to the 'PubTator 3.0 knowledge graph' without imposing the seen restriction that is explicit for the literature corpus (PMID≤36600000). Under Eq. (2), contextG(p) returns every edge in G whose endpoints occur in p. For a true hypothesis about an unseen edge (eh,r,et), that edge is in G and is therefore fed to the claim-verification LLM; for a false hypothesis, no such edge is present. The KnowHD groundedness score (Eq. 3) thus reduces to checking membership in the label set G. The positive groundedness-accuracy trend in Figure 3 and the 'groundedness-based selection' gains in Figure 4 are forced by this construction, as is the high/low grouping in Section 5.

full rationale

The paper's central claim is that KnowHD groundedness scores reliably filter truthful hypotheses. Section 2.1 defines the benchmark's ground truth as the 'unseen' portion of PubTator 3.0, i.e., relations from papers published after 2024. Section 3 defines KnowHD's graph context, Eq. (2), as all edges in a knowledge graph G whose endpoints appear in the claim, and Appendix A identifies that graph with PubTator 3.0 via entity matching. Nowhere does the KnowHD section restrict G to the pre-2023 'seen' subgraph, even though the same section explicitly restricts the literature corpus to PMID≤36600000. Consequently, a correct hypothesis about an unseen relation is grounded by the very target edge (eh,r,et) sitting in G, while an incorrect hypothesis about a relation absent from G is ungrounded. The groundedness-truthfulness correlation in Figure 3, the selection improvements in Figure 4, and the high/low groundedness pairing in the Section 5 human study therefore reduce, on this construction, to membership in the label set rather than to an independent hallucination signal. The literature-based component alone would not have this defect, but the reported KnowHD scores combine KG and literature, and the KG component is not temporally constrained in the text. No load-bearing self-citation or imported-uniqueness step was found; the authors' prior work is cited but is not what forces the result. Because the manuscript's operationalized metric contains the answer it is claimed to predict, the central filtering result is circular as written, though it could be repaired by explicitly restricting G to the seen subset.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the assumption that PubTator graph edges approximate truth and that their absence means no relation; that the LLM judge for entailment is reliable; that LLM training did not memorize future edges; and, critically, that the KG used in KnowHD is temporally restricted to pre-2023 content, which the paper never states. These are domain assumptions or unstated conveniences, not derived results.

free parameters (4)
  • k (number of retrieved documents) = 8 for claim verification, 32 for hypothesis generation
    Hand-chosen in Appendix A; controls how much context the grounding judge sees and affects groundedness scores.
  • tau (BM25 relevance threshold) = 0.0
    Set in Appendix A; with rank cutoff k it defines which documents form contextD(p).
  • Groundedness gap threshold for human study pairs = 30%
    Section 5 keeps only pairs where the higher groundedness score exceeds the lower by at least 30%, yielding 54 pairs for annotation.
  • PMID cutoffs for temporal split = 36600000 and 38200000
    Section 2.1 uses these to define the seen (pre-2023) and unseen (post-2024) subsets, which determine all ground-truth labels.
assumptions (4)
  • domain assumption PubTator 3.0 edges from post-2024 literature constitute ground truth for hypothesis truthfulness, and absence of an edge implies 'no relation'.
    Section 2.1 derives all positive and negative labels from KG edge presence or absence, equating literature extraction with scientific truth.
  • domain assumption The LLM used for claim verification correctly decides whether retrieved context entails an atomic claim.
    Section 3 and prompt template Figure 12 rely on LLM judgments; no human validation or agreement measure is reported for these entailment labels.
  • domain assumption The evaluated LLMs were trained only on pre-2024 data and cannot memorize the unseen relations.
    Section 4.1 assumes all models 'were trained on the knowledge available before 2024', without per-entity verification.
  • ad hoc to paper The knowledge graph G used in KnowHD grounding is temporally restricted to the seen subset, even though the paper never states this.
    Section 3 Eq. (2) defines contextG(p) over G without a temporal filter, unlike the literature retrieval in the same section; the central correlation requires this restriction to avoid leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models." pith.science (2026). https://pith.science/paper/AGIYIAGP

@misc{pith2026250514599,
  author       = {Pith},
  title        = {Pith review of: Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGIYIAGP}},
  note         = {Machine review of arXiv:2505.14599}
}
read the original abstract

Large language models (LLMs) have shown significant potential in scientific disciplines such as biomedicine, particularly in hypothesis generation, where they can analyze vast literature, identify patterns, and suggest research directions. However, a key challenge lies in evaluating the truthfulness of generated hypotheses, as verifying their accuracy often requires substantial time and resources. Additionally, the hallucination problem in LLMs can lead to the generation of hypotheses that appear plausible but are ultimately incorrect, undermining their reliability. To facilitate the systematic study of these challenges, we introduce TruthHypo, a benchmark for assessing the capabilities of LLMs in generating truthful scientific hypotheses, and KnowHD, a knowledge-based hallucination detector to evaluate how well hypotheses are grounded in existing knowledge. Our results show that LLMs struggle to generate truthful hypotheses. By analyzing hallucinations in reasoning steps, we demonstrate that the groundedness scores provided by KnowHD serve as an effective metric for filtering truthful hypotheses from the diverse outputs of LLMs. Human evaluations further validate the utility of KnowHD in identifying truthful hypotheses and accelerating scientific discovery. Our data and source code are available at https://github.com/Teddy-XiongGZ/TruthHypo.

Figures

Figures reproduced from arXiv: 2505.14599 by the authors.

Figure 1
Figure 1. Overview of the TruthHypo benchmark, including dataset construction, task formulation, and truthfulness evaluation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the KnowHD hallucination detection frame [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Mean accuracy corresponding to different levels of groundedness. Hypotheses are grouped based on their groundedness scores [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Accuracy improvements of GPT-4o-mini using KnowHD [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Prompt template for constructing user input with given entities. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Prompt template for hypothesis generation with parametric knowledge only. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for hypothesis generation with knowledge from parameters and knowledge graphs (KGs). [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompt template for hypothesis generation with knowledge from parameters and literature. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for hypothesis generation with knowledge from parameters, KG, and literature. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Prompt template for claim identification. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for entity recognition [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for claim verification [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interestingness First Classifiers

    cs.LG 2025-08 conditional novelty 6.0 of 10

    EUREKA uses LLM pairwise comparisons to rank features by interestingness and trains logistic regression on the top-ranked features, producing non-obvious yet above-chance classifiers on six tabular datasets.

Reference graph

Works this paper leans on

61 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Scientific Hypothesis Generation by a Large Language Model: Laboratory Validation in Breast Cancer Treatment

    Abbi Abdel-Rehim, Hector Zenil, Oghenejokpeme Orhobor, Marie Fisher, Ross J Collins, Elizabeth Bourne, Gareth W Fearnley, Emma Tate, Holly X Smith, Larisa N Soldatova, et al. Scientific hypothesis generation by a large language model: Laboratory validation in breast cancer treatment. arXiv preprint arXiv:2405.12258 , 2024

    work page Pith review arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  3. [3]

    Researchagent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738 , 2024

    arXiv 2024

  4. [4]

    Harnessing the power of adversarial prompting and large language models for robust hypothesis generation in astronomy

    Ioana Ciuc a , Yuan-Sen Ting, Sandor Kruk, and Kartheik Iyer. Harnessing the power of adversarial prompting and large language models for robust hypothesis generation in astronomy. arXiv preprint arXiv:2306.11648 , 2023

    arXiv 2023

  5. [5]

    Marg: Multi-agent review generation for scientific papers

    Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259 , 2024

    arXiv 2024

  6. [6]

    Towards a rigorous science of interpretable machine learning

    Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 , 2017

    arXiv 2017

  7. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  8. [8]

    On the creativity of large language models

    Giorgio Franceschelli and Mirco Musolesi. On the creativity of large language models. AI & SOCIETY , pages 1--11, 2024

    work page 2024

Show all 61 references
  1. [9]

    Forecasting high-impact research topics via machine learning on evolving knowledge graphs

    Xuemei Gu and Mario Krenn. Forecasting high-impact research topics via machine learning on evolving knowledge graphs. arXiv preprint arXiv:2402.08640 , 2024

    2024 arXiv

  2. [10]

    Embracing foundation models for advancing scientific discovery

    Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, and Aidong Zhang. Embracing foundation models for advancing scientific discovery. In 2024 IEEE International Conference on Big Data (BigData) , pages 1746--1755. IEEE, 2024

    2024

  3. [11]

    Williams, Stefan Bekiranov, and Aidong Zhang

    Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Myles Kim, Corey M. Williams, Stefan Bekiranov, and Aidong Zhang. Ideabench: Benchmarking large language models for research idea generation. In 31st SIGKDD Conference on Knowledge Discovery and Data Mining - ...

    2025

  4. [12]

    Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas

    Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas. arXiv preprint arXiv:2410.14255 , 2024

    arXiv

  5. [13]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232 , 2023

    2023 arXiv

  6. [14]

    Autonomous llm-driven research—from data to human-verifiable research papers

    Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay, and Roy Kishony. Autonomous llm-driven research—from data to human-verifiable research papers. NEJM AI , 2(1):AIoa2400555, 2025

    2025

  7. [15]

    A survey on knowledge graphs: Representation, acquisition, and applications

    Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems , 33(2):494--514, 2021

    2021

  8. [16]

    Demystifying large language models for medicine: A primer

    Qiao Jin, Nicholas Wan, Robert Leaman, Shubo Tian, Zhizheng Wang, Yifan Yang, Zifeng Wang, Guangzhi Xiong, Po-Ting Lai, Qingqing Zhu, et al. Demystifying large language models for medicine: A primer. arXiv preprint arXiv:2410.18856 , 2024

    arXiv

  9. [17]

    Large language models versus natural language understanding and generation

    Nikitas Karanikolas, Eirini Manga, Nikoletta Samaridi, Eleni Tousidou, and Michael Vassilakopoulos. Large language models versus natural language understanding and generation. In Proceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics , pages 2...

    2023

  10. [18]

    Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network

    Mario Krenn, Lorenzo Buffoni, Bruno Coutinho, Sagi Eppel, Jacob Gates Foster, Andrew Gritsevskiy, Harlin Lee, Yichao Lu, Jo \ a o P Moutinho, Nima Sanjabi, et al. Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially ...

    2023

  11. [19]

    Mycrunchgpt: A chatgpt assisted framework for scientific machine learning

    Varun Kumar, Leonard Gleyzer, Adar Kahana, Khemraj Shukla, and George Em Karniadakis. Mycrunchgpt: A chatgpt assisted framework for scientific machine learning. arXiv preprint arXiv:2306.15551 , 2023

    2023 arXiv

  12. [20]

    Paperqa: Retrieval-augmented generative agent for scientific research

    Jakub L \'a la, Odhran O'Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. Paperqa: Retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559 , 2023

    2023 arXiv

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  14. [22]

    Chain of ideas: Revolutionizing research via novel idea development with llm agents

    Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, et al. Chain of ideas: Revolutionizing research via novel idea development with llm agents. arXiv preprint arXiv:2410.13185 , 2024

    arXiv

  15. [23]

    Learning entity and relation embeddings for knowledge graph completion

    Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence , volume 29, 2015

    2015

  16. [24]

    A survey on graph classification and link prediction based on gnn

    Xingyu Liu, Juan Chen, and Quan Wen. A survey on graph classification and link prediction based on gnn. arXiv preprint arXiv:2307.00865 , 2023

    2023 arXiv

  17. [25]

    Conversational drug editing using retrieval and domain feedback

    Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Conversational drug editing using retrieval and domain feedback. In The Twelfth International Conference on Learning Representations , 2024

    2024

  18. [26]

    Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011--2022)

    Hui Wen Loh, Chui Ping Ooi, Silvia Seoni, Prabal Datta Barua, Filippo Molinari, and U Rajendra Acharya. Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011--2022). Computer methods and programs in biomedicine , 226:10...

    2011

  19. [27]

    Improving biomedical information retrieval with neural retrievers

    Man Luo, Arindam Mitra, Tejas Gokhale, and Chitta Baral. Improving biomedical information retrieval with neural retrievers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 11038--11046, 2022

    2022

  20. [28]

    Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence , pages 1--11, 2024

    2024

  21. [29]

    Think-on-graph 2.0: Deep and interpretable large language model reasoning with knowledge graph-guided retrieval

    Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, and Jian Guo. Think-on-graph 2.0: Deep and interpretable large language model reasoning with knowledge graph-guided retrieval. arXiv e-prints , pages arXiv--2407, 2024

    2024

  22. [30]

    Explainable ai is dead, long live explainable ai! hypothesis-driven decision support using evaluative ai

    Tim Miller. Explainable ai is dead, long live explainable ai! hypothesis-driven decision support using evaluative ai. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency , pages 333--342, 2023

    2023

  23. [31]

    Evaluating the effectiveness of retrieval-augmented large language models in scientific document reasoning

    Sai Munikoti, Anurag Acharya, Sridevi Wagle, and Sameera Horawalavithana. Evaluating the effectiveness of retrieval-augmented large language models in scientific document reasoning. arXiv preprint arXiv:2311.04348 , 2023

    2023 arXiv

  24. [32]

    A review of relational machine learning for knowledge graphs

    Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE , 104(1):11--33, 2015

    2015

  25. [33]

    Can chatgpt be used to generate scientific hypotheses? Journal of Materiomics , 10(3):578--584, 2024

    Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. Can chatgpt be used to generate scientific hypotheses? Journal of Materiomics , 10(3):578--584, 2024

    2024

  26. [34]

    Graph retrieval-augmented generation: A survey

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921 , 2024

    2024 arXiv

  27. [35]

    Large language models are zero shot hypothesis proposers

    Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers. arXiv preprint arXiv:2311.05965 , 2023

    2023 arXiv

  28. [36]

    Large language models as biomedical hypothesis generators: A comprehensive evaluation

    Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxiang Li, Zhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jinfang, and Bowen Zhou. Large language models as biomedical hypothesis generators: A comprehensive evaluation. In First Conference on Language Modeling , 2024

    2024

  29. [37]

    Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination

    Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S Weld. Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination. arXiv preprint arXiv:2409.14634 , 2024

    2024 arXiv

  30. [38]

    A review on large language models: Architectures, applications, taxonomies, open issues and challenges

    Mohaimenul Azam Khan Raiaan, Md Saddam Hossain Mukta, Kaniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eunus Ali, and Sami Azam. A review on large language models: Architectures, applications, taxonomies, open issues and challen...

    2024

  31. [39]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389, 2009

    2009

  32. [40]

    Knowledge graph large language model (kg-llm) for link prediction

    Dong Shu, Tianle Chen, Mingyu Jin, Chong Zhang, Mengnan Du, and Yongfeng Zhang. Knowledge graph large language model (kg-llm) for link prediction. arXiv preprint arXiv:2403.07311 , 2024

    2024 arXiv

  33. [41]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109 , 2024

    2024 arXiv

  34. [42]

    Colidr: Concept learning using aggregated disentangled representations

    Sanchit Sinha, Guangzhi Xiong, and Aidong Zhang. Colidr: Concept learning using aggregated disentangled representations. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2699--2710, 2024

    2024

  35. [43]

    A self-explaining neural architecture for generalizable concept learning

    Sanchit Sinha, Guangzhi Xiong, and Aidong Zhang. A self-explaining neural architecture for generalizable concept learning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages 503--511, 2024

    2024

  36. [44]

    Language agents achieve superhuman synthesis of scientific knowledge

    Michael D Skarlinski, Sam Cox, Jon M Laurent, James D Braza, Michaela Hinks, Michael J Hammerling, Manvitha Ponnapati, Samuel G Rodriques, and Andrew D White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409.13740 , 2024

    2024 arXiv

  37. [45]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  38. [46]

    Scimon: Scientific inspiration machines optimized for novelty

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259 , 2023

    2023 arXiv

  39. [47]

    Knowledge graph retrieval-augmented generation for llm-based recommendation

    Shijie Wang, Wenqi Fan, Yue Feng, Xinyu Ma, Shuaiqiang Wang, and Dawei Yin. Knowledge graph retrieval-augmented generation for llm-based recommendation. arXiv preprint arXiv:2501.02226 , 2025

    2025 arXiv

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824--24837, 2022

    2022

  41. [49]

    Pubtator 3.0: an ai-powered literature resource for unlocking biomedical knowledge

    Chih-Hsuan Wei, Alexis Allot, Po-Ting Lai, Robert Leaman, Shubo Tian, Ling Luo, Qiao Jin, Zhizheng Wang, Qingyu Chen, and Zhiyong Lu. Pubtator 3.0: an ai-powered literature resource for unlocking biomedical knowledge. Nucleic Acids Research , page gkae235, 2024

    2024

  42. [50]

    Generating scientific claims for zero-shot scientific fact checking

    Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. Generating scientific claims for zero-shot scientific fact checking. arXiv preprint arXiv:2203.12990 , 2022

    2022 arXiv

  43. [51]

    Dynamic link prediction using graph representation learning with enhanced structure and temporal information

    Chaokai Wu, Yansong Wang, and Tao Jia. Dynamic link prediction using graph representation learning with enhanced structure and temporal information. In 2023 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD) , pages 279--284. IEEE, 2023

    2023

  44. [52]

    Benchmarking retrieval-augmented generation for medicine

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics: ACL 2024 , 2024

    2024

  45. [53]

    Improving retrieval-augmented generation in medicine with iterative follow-up questions

    Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, and Aidong Zhang. Improving retrieval-augmented generation in medicine with iterative follow-up questions. In Biocomputing 2025: Proceedings of the Pacific Symposium , pages 199--214. World Scientific, 2024

    2025

  46. [54]

    Improving scientific hypothesis generation with knowledge grounded large language models

    Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Improving scientific hypothesis generation with knowledge grounded large language models. arXiv preprint arXiv:2411.02382 , 2024

    2024 arXiv

  47. [55]

    Large language models for automated open-domain scientific hypotheses discovery

    Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.02726 , 2023

    2023 arXiv

  48. [56]

    Large language models for rediscovering unseen chemistry scientific hypotheses

    Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. Large language models for rediscovering unseen chemistry scientific hypotheses. In 2nd AI4Research Workshop: Towards a Knowledge-grounded Scientific Research...

    2025

  49. [57]

    Meta-review generation with checklist-guided iterative introspection

    Qi Zeng, Mankeerat Sidhu, Hou Pong Chan, Lu Wang, and Heng Ji. Meta-review generation with checklist-guided iterative introspection. arXiv preprint arXiv:2305.14647 , 2023

    2023 arXiv

  50. [58]

    Link prediction based on graph neural networks

    Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems , 31, 2018

    2018

  51. [59]

    Goal driven discovery of distributional differences via language descriptions

    Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. Advances in Neural Information Processing Systems , 36:40204--40237, 2023

    2023

  52. [60]

    Hypothesis generation with large language models

    Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326 , 2024

    2024 arXiv

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.