Pith. sign in

REVIEW 4 major objections 6 minor 189 references

BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper presents BioMol-MQA, a question-answering dataset on polypharmacy in which large language models score 0.22 lexical exact-match zero-shot but 0.62 when given the gold multimodal context, indicating that reliable answers require…

desk verdict Useful new multimodal QA benchmark for polypharmacy, but the LLM-generated molecular-interaction labels need external validation before the molecular half of the dataset is trustworthy. read the letter →

arxiv 2506.05766 v1 pith:3IMQOCBT submitted 2025-06-06 cs.CL

Saptarshi Sengupta , Shuhua Yang , Paul Kwong Yu , Fali Wang , Suhang Wang This is my paper
classification cs.CL
keywords multimodalquestionansweringretrieval-augmentedgenerationpolypharmacydrug-druginteractionknowledgegraphSMILESLLMreasoningbiomedical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BioMol-MQA asks whether large language models can answer realistic polypharmacy questions that require combining three kinds of information: a graph of drug-drug and drug-protein interactions, short background texts on each entity, and molecular structure strings (SMILES). The dataset contains 1,683 questions built from these modalities plus the multimodal knowledge graph itself, split into training, validation, and test sets. Across seven LLMs, average lexical exact-match is 0.22 zero-shot but rises to 0.62 when the models are given the exact gold context, which the authors read as evidence that grounding on domain knowledge, not model scale alone, is required. A simple hybrid retriever (BM25 plus a graph query) improves performance but leaves a clear gap to the gold-context upper bound.

What carries the argument

The load-bearing artifact is a multimodal knowledge graph with 494 drug nodes, 198 protein nodes, 18,585 drug-drug edges, 314 labeled drug-protein edges, and 499 LLM-added molecular interaction edges that turn it into a multi-graph. Each drug carries a Wikipedia-derived background text (rephrased by GPT-4o into denser clinical prose) and a canonical SMILES string from PubChem; each protein carries background text only. Questions are generated by GPT-4.1 from graph triples combined with either both entities' texts or their SMILES, under two constraints: no entity name appears verbatim and the question must test the relationship, not isolated facts. Verification uses Claude-3.7 Sonnet as an automatic judge plus a human expert on a 100-question sample. The task is defined as mapping (graph, text, SMILES, question) to a single answer node, with answers always on the directed end of an interaction.

What would settle it

Draw a random sample of the 699 molecular DDI questions and have a panel of medicinal chemists, or an independent structure-based interaction predictor, judge whether the recorded interaction and mechanism actually follow from the two SMILES strings; if agreement with the LLM-generated labels is far below the ~0.9 EM that models reach with gold context, the labels themselves are the failing component. A second check is whether human domain experts given exactly the gold text and SMILES reproduce the gold answers at high agreement, which would confirm the questions are answerable rather than merely consistent.

Watch

Extended reading notes

Core claim

The central claim is that BioMol-MQA is a valid and difficult testbed for multimodal retrieval-augmented reasoning in drug safety, and that current models fail it without retrieval. The benchmark's questions never name their answer entities; a correct response requires resolving the drug or protein from its textual description, connecting it to a knowledge-graph edge, and, for molecular drug-drug interactions, interpreting SMILES structure. Benchmarks on the test split show seven frontier and open LLMs averaging 0.22 lexical EM and 0.28 F1 zero-shot, versus 0.62 EM and 0.67 F1 when the gold triples, background text, and SMILES are supplied. The two best zero-shot models, o4-mini and DeepSeek-R1, reach 0.88 and 0.90 EM in this upper-bound setting, indicating the questions are answerable, while a basic BM25-plus-Neo4j retriever lifts o4-mini only to 0.54 EM, showing that existing retrievers are the current bottleneck.

Load-bearing premise

The load-bearing premise is that GPT-4o's automatically generated molecular interactions are chemically valid enough to serve as the gold answers for the 699 molecular drug-drug questions, a premise the paper does not check against any external interaction database.

Editorial extensions

If this is right

  • Grounded retrieval, not scale alone, is what moves LLM performance on BioMol-MQA: average exact-match more than doubles from 0.22 to 0.62 when gold multimodal context is provided, so models that cannot retrieve this context will likely fail similar drug-safety questions.
  • Even a deliberately simple hybrid retriever (BM25 for text plus Neo4j over triples) improves o4-mini from 0.32 to 0.54 EM, establishing a baseline that more sophisticated multimodal RAG systems must beat.
  • The dataset's stratified 80-10-10 split supports training and evaluating joint question-modality representations, and the paper points to graph link prediction and molecular property prediction as downstream beneficiaries.
  • Because only drugs have both text and SMILES while proteins have text only, the benchmark covers the graph-plus-text and graph-plus-structure combinations the task definition requires, but not a protein-structure modality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the strongest test of the dataset's gold answers would be to validate the 699 GPT-4o-generated molecular interactions against a cheminformatics interaction database or expert chemistry annotation; if agreement is low, the molecular-question subset needs relabeling, not better retrievers.
  • Editorial extension: the result that BM25 beats dense retrievers on this corpus hints that the hard part of retrieval here is lexical and entity-level matching rather than semantic similarity, so future retrievers may need to treat SMILES as molecular graphs instead of embedding them as text.
  • Editorial extension: if the zero-shot-to-gold gap replicates outside this testbed, a practical consequence for health applications is that LLM answers should carry a confidence flag indicating whether molecular context was actually retrieved and used.
  • Editorial extension: an interpretable side-output of the pipeline is the evidence field GPT-4o emits for each molecular interaction; scoring that evidence against functional-group rules would turn the dataset into a benchmark for chemically explainable generation, not just QA.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BioMol-MQA, a question-answering benchmark for polypharmacy built on a multimodal knowledge graph that combines drug-protein/drug-drug interaction triples, Wikipedia-style background text, and SMILES molecular structures. The dataset construction pipeline has five stages: base graph acquisition, text post-processing, GPT-4o-based molecular interaction extraction, GPT-4.1-based question generation, and a human/LLM quality audit. The authors report that frontier and open-weight LLMs score low in zero-shot exact match (claimed average 0.22) and much higher when given gold context (claimed average 0.62), and that a simple hybrid BM25+Neo4j retriever only partially closes the gap. They position the dataset as a testbed for multimodal retrieval-augmented generation in healthcare.

Significance. If the dataset is validated, it fills a genuine gap: most RAG benchmarks are unimodal or general-domain, while BioMol-MQA integrates graph, text, and molecular-structure modalities and includes multi-hop questions. The paper provides a transparent pipeline description, releases the dataset, and benchmarks seven LLMs and several retrievers. Credit is due for the multi-hop question design, the inclusion of molecular and protein interactions, and the attempt to separate zero-shot, RAG, and gold-context performance. However, the benchmark's reliability rests on the chemical validity of GPT-4o-generated molecular interaction labels and on the adequacy of a 100-question quality audit performed largely by a co-author; neither is currently established. The headline empirical numbers also appear inconsistent with the reported tables. These issues must be resolved before the central claim about the necessity of multimodal RAG can be accepted.

major comments (4)
  1. [Abstract; §4.1; Table 6] The abstract and Section 4.1 claim average zero-shot EM/F1/BERTScore of 0.22/0.28/0.77 and gold-context (upper-bound) scores of 0.62/0.67/0.83, but Table 6 does not support these averages. Averaging the seven per-model tuples in Table 6 gives zero-shot EM ≈ 0.22, F1 ≈ 0.30, BERTScore ≈ 0.78 and upper-bound EM ≈ 0.81, F1 ≈ 0.81, BERTScore ≈ 0.94. The central claim that gold context raises EM from 0.22 to 0.62 is therefore not reproducible from the table as printed. Please correct the numbers, define exactly which subset or split the 0.62 figure refers to, and ensure the abstract matches the corrected values.
  2. [§3.3; Table 3; Table 6] The molecular interaction labels that underpin 699 of 1,683 questions (41.5%) are generated by GPT-4o from SMILES strings with no external verification: there is no check against an interaction database, no rule-based or SMARTS validation of the reported evidence, and no expert annotation of chemical validity. Because the questions and gold answers are generated from these triples, any hallucinated interaction propagates directly into the benchmark's gold labels. The headline zero-shot versus gold-context gap may therefore partly measure how well an LLM reads an LLM-generated triple rather than how well it reasons over real molecular interactions. Please validate a sample of the Stage III labels against an external source (e.g., STITCH, PubChem, ChEMBL, or domain-expert review) and report the benchmark results separately for questions that do and do not depend on the GPT-4o molecular labels.
  3. [§3.5; Table 5; §4.1] The quality audit covers only 100 randomly sampled QA pairs, is scored by a single human annotator who is also a co-author plus an LLM judge, and the rubric (clarity, coverage, assumptions, inferability) does not verify the correctness of the underlying KG triples or the chemical validity of the molecular interactions. This audit is then used to claim that the questions are answerable and fair for the whole dataset. The sample is too small and too narrowly scored to certify the remaining 1,583 questions, and the absence of any inter-annotator agreement measure limits the reliability of the human scores. Please either expand the audit with multiple annotators and report agreement, or restrict the fairness/answerability claim to the audited subset and validate gold-label correctness separately.
  4. [§4.2; Table 7; footnote] The Neo4j graph retriever and the hybrid BM25+Neo4j retriever are evaluated on the subset of triples used for question generation, while the text retrievers search the full text corpus. This is not an apples-to-apples comparison: the graph side is given a much smaller, question-aligned retrieval set, which can only improve its measured performance relative to the text retrievers. The conclusion that 'simple hybrid retrieval only partially closes the gap' is therefore not supported by the current experimental setup. Please evaluate all retrievers on the same full graph/corpus, or report both subset and full-graph results clearly labeled, so that the retrieval comparison is meaningful.
minor comments (6)
  1. [§3.4] The paper states that 15% of DPI questions mention the protein name, which violates the stated requirement that entity names not appear verbatim in the question. Since DPI questions are included in the test split, this leakage likely inflates zero-shot performance for that subset; please report per-question-type results or re-generate the affected questions.
  2. [§3.2] The 200-token threshold for text post-processing is described as 'set empirically' but no sensitivity analysis is provided. Please report how this threshold was chosen and whether the downstream question complexity or LLM performance is sensitive to it.
  3. [§4.1; Table 6] The stated average zero-shot F1 of 0.28 does not match the value computed from Table 6 (≈0.30); please recompute all reported averages and verify them against the table.
  4. [Table 7] The table header reports 'Hits@k' but the values are given as hard/soft pairs; please clarify the hard/soft definition directly in the table caption rather than only in the text.
  5. [Abstract; §F] The abstract contains a placeholder '/githubCode' rather than a working dataset/code URL. The appendix includes a GitHub link, but the main text should state explicitly where the dataset, prompts, and evaluation scripts can be obtained for reproducibility.
  6. [§3.5] The scoring direction of the rubric metrics is inconsistent in the text: for Clarity higher scores indicate more difficulty, while for Assumptions and Inference higher scores are better. Please clarify this in the table and surrounding description to avoid misinterpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RAG gap is an empirical measurement on a held-out split, and the LLM-generated molecular labels are a validity risk rather than a circular derivation.

full rationale

The paper's central claim is the measured performance gap on a held-out test split (Table 6: mean zero-shot lexical EM 0.22 vs. gold-context 0.62). This is an empirical result, not a derived quantity: the upper-bound condition supplies the exact triples and texts from which each question was generated, which is the standard open-book control for a QA dataset, but the model outputs are not fitted to those labels, and the gold-context score is not forced by construction (the same condition yields only 0.62 EM, leaving substantial headroom). The self-referential flavor of the pipeline (GPT-4o generates molecular edges, GPT-4.1 writes questions, and Claude-3.7 judges a 100-question sample) is a data-provenance and external-validity concern: the 699 molecular questions rest on GPT-4o-derived triples that are not checked against an external interaction database. That undermines the biomedical trustworthiness of those labels, but it does not make the RAG comparison circular, because the benchmark evaluates whether a model can read and reason over whatever context is supplied. The verification rubric (Assumptions/Inferable) certifies internal answerability; it does not certify chemical truth, and the paper is transparent that Stage III is a synthetic augmentation. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. The paper's limitations (Appendix O) acknowledge modality imbalance and evaluation cost but do not conceal a derivation that reduces to its inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest mainly on dataset construction choices rather than fitted parameters. The key unverified elements are the LLM-generated molecular interaction labels and the LLM-generated questions, treated as ground truth without comprehensive external validation. The hand-chosen thresholds control the composition of the benchmark but do not determine the benchmark conclusions by themselves.

free parameters (3)
  • Text length threshold for post-processing = 200 tokens
    Documents longer than 200 tokens are rewritten by GPT-4o to increase density; the threshold is described as set empirically in section 3.2.
  • STITCH confidence threshold = score > 900
    Only drug-protein interactions with a STITCH score above 900 are retained; this arbitrary cut affects the graph size and is described in section 3.1.1 and Appendix C.
  • Random multi-hop edge sample size = 100 per hop length
    One hundred 2-hop and 3-hop DDI/DPI edges are randomly sampled for multi-hop questions, stated in section 3.4.
assumptions (5)
  • domain assumption The base knowledge graph from Zitnik et al. [142] accurately represents polypharmacy drug-drug and drug-protein interactions.
    The entire dataset is built on this external graph in section 3.1.1.
  • ad hoc to paper GPT-4o's molecular interaction extraction is chemically valid enough to serve as ground truth for molecular DDI questions.
    Stage III (section 3.3) uses GPT-4o to add molecular interaction labels without external validation against existing databases.
  • ad hoc to paper GPT-4.1 generated questions are answerable and grounded in the provided data.
    Questions are created by prompting GPT-4.1 in section 3.4, and quality is checked on only 100 samples by Claude and one human co-author in section 3.5.
  • domain assumption Wikipedia summaries plus LLM rewriting provide sufficient and accurate entity background.
    The text modality is built from Wikipedia pages rewritten by GPT-4o, as described in sections 3.1.2 and 3.2.
  • domain assumption LLM-as-a-judge is a reliable proxy for human evaluation.
    The paper uses Claude-3.7 Sonnet for automatic evaluation and cites prior work on LLM judges in section 3.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions." pith.science (2026). https://pith.science/paper/3IMQOCBT

@misc{pith2026250605766,
  author       = {Pith},
  title        = {Pith review of: BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IMQOCBT}},
  note         = {Machine review of arXiv:2506.05766}
}
read the original abstract

Retrieval augmented generation (RAG) has shown great power in improving Large Language Models (LLMs). However, most existing RAG-based LLMs are dedicated to retrieving single modality information, mainly text; while for many real-world problems, such as healthcare, information relevant to queries can manifest in various modalities such as knowledge graph, text (clinical notes), and complex molecular structure. Thus, being able to retrieve relevant multi-modality domain-specific information, and reason and synthesize diverse knowledge to generate an accurate response is important. To address the gap, we present BioMol-MQA, a new question-answering (QA) dataset on polypharmacy, which is composed of two parts (i) a multimodal knowledge graph (KG) with text and molecular structure for information retrieval; and (ii) challenging questions that designed to test LLM capabilities in retrieving and reasoning over multimodal KG to answer questions. Our benchmarks indicate that existing LLMs struggle to answer these questions and do well only when given the necessary background data, signaling the necessity for strong RAG frameworks.

Figures

Figures reproduced from arXiv: 2506.05766 by the authors.

Figure 1
Figure 1. Example from BioMol-MQA. Colors highlight elements from each information source (Blue = Graph, pink|orange = Text) that a model must connect to answer questions. In this example, a model needs to first identify the correct NSAID (Oxaprozin), determine its outgoing edges in the graph that have a gastric inflammation label, and figure out which drug on the other end is used to manage nociceptive pain. More examples ar… view at source ↗
Figure 2
Figure 2. Our dataset development pipeline. The overall pipeline of developing our dataset is summarized in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Simple phrases as liver problems are replaced by technical jargon as hepatotoxicity. Before PP After PP Tokens 300 233 Readability 15 28 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The final knowledge-graph depicting each modality, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Impact of RAG on o4-mini and DeepSeek R1 (DS). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

189 extracted references · 39 canonical work pages

  1. [1]

    Text-based question answering from information retrieval and deep neural network perspectives: A survey

    Zahra Abbasiantaeb and Saeedeh Momtazi. Text-based question answering from information retrieval and deep neural network perspectives: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(6):e1412, 2021

    2021

  2. [2]

    Synthetic dialogue dataset generation using LLM agents

    Yelaman Abdullin, Diego Molla, Bahadorreza Ofoghi, John Yearwood, and Qingyang Li. Synthetic dialogue dataset generation using LLM agents. In Sebastian Gehrmann, Alex Wang, João Sedoc, Elizabeth Clark, Kaustubh Dhole, Khyathi Raghavi Chandu, Enrico Santus, and Hooman Sedghamiz, editors, Proceedings of the Third Workshop on Natural Language Generation, Eva...

  3. [3]

    Ask in any modality: A comprehensive survey on multimodal retrieval- augmented generation

    Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mo- hammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. Ask in any modality: A comprehensive survey on multimodal retrieval- augmented generation. arXiv preprint arXiv:2502.08826, 2025

    arXiv 2025

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Together AI – The AI Acceleration Cloud - Fast Inference, Fine-Tuning & Training — together.ai

    Together AI. Together AI – The AI Acceleration Cloud - Fast Inference, Fine-Tuning & Training — together.ai. https://www.together.ai/, 2025. [Accessed 13-05-2025]

    2025

  6. [6]

    FLAIR: An easy-to-use framework for state-of-the-art NLP

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland V ollgraf. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Waleed Ammar, Annie Louis, and Nasrin Mostafazadeh, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 5...

    doi:10.18653/v1/n19-4010 2019

  7. [7]

    Multimodal large language models in health care: appli- cations, challenges, and future outlook

    Rawan AlSaad, Alaa Abd-Alrazaq, Sabri Boughorbel, Arfan Ahmed, Max-Antoine Renault, Rafat Damseh, and Javaid Sheikh. Multimodal large language models in health care: appli- cations, challenges, and future outlook. Journal of medical Internet research , 26:e59505, 2024

    2024

  8. [8]

    Claude 3.7 system card - Anthropic — docs.anthropic.com, 2025

    Anthropic. Claude 3.7 system card - Anthropic — docs.anthropic.com, 2025. URL https://assets.anthropic.com/m/785e231869ea8b3b/original/ claude-3-7-sonnet-system-card.pdf . [Accessed 29-04-2025]

    2025

Show all 189 references
  1. [9]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  2. [10]

    The human proteome in druggable - The Human Protein Atlas — proteinatlas.org

    The Human Protein Atlas. The human proteome in druggable - The Human Protein Atlas — proteinatlas.org. https://www.proteinatlas.org/humanproteome/tissue/ druggable, 2025. [Accessed 11-05-2025]

    2025

  3. [11]

    Nishant Balepur, Feng Gu, Abhilasha Ravichander, Shi Feng, Jordan Lee Boyd-Graber, and Rachel Rudinger. Reverse question answering: Can an LLM write a question so hard (or bad) that it can‘t answer? In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 C...

    2025

  4. [12]

    Interactive question answering systems: Literature review

    Giovanni Maria Biancofiore, Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, and Fedelucio Narducci. Interactive question answering systems: Literature review. ACM Comput- ing Surveys, 56(9):1–38, 2024. 10

    2024

  5. [14]

    Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, ...

    2022 doi

  6. [15]

    Dated data: Tracing knowledge cutoffs in large language models

    Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. Dated data: Tracing knowledge cutoffs in large language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=wS7PxDjy6m

    2024

  7. [17]

    From lo- cal to global: A graph rag approach to query-focused summarization

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From lo- cal to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024

    2024 arXiv

  8. [18]

    A survey on rag meeting llms: Towards retrieval-augmented large language models

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages...

    2024

  9. [19]

    Ragbench: Explainable benchmark for retrieval-augmented generation systems

    Robert Friel, Masha Belyi, and Atindriyo Sanyal. Ragbench: Explainable benchmark for retrieval-augmented generation systems. arXiv preprint arXiv:2407.11005, 2024

    2024 arXiv

  10. [20]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1, 2023

    2023 arXiv

  11. [21]

    Introducing Gemini 2.0: our new AI model for the agentic era — blog.google

    Google. Introducing Gemini 2.0: our new AI model for the agentic era — blog.google. https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/ , 2024. [Accessed 14-05-2025]

    2024

  12. [22]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  13. [23]

    On the effectiveness of large language models in domain-specific code generation

    Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, and Juhong Wang. On the effectiveness of large language models in domain-specific code generation. ACM Transactions on Software Engineering and Methodology, 34(3):1–22, 2025

    2025

  14. [24]

    Domain-specific language model pretraining for biomedical natural language processing

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021. 11

    2021

  15. [25]

    Towards generalist prompting for large language models by mental models

    Haoxiang Guan, Jiyan He, Shuxin Zheng, En-Hong Chen, Weiming Zhang, and Nenghai Yu. Towards generalist prompting for large language models by mental models. arXiv preprint arXiv:2402.18252, 2024

    2024 arXiv

  16. [26]

    The fog index after twenty years

    Robert Gunning. The fog index after twenty years. Journal of Business Communication, 6 (2):3–13, 1969. doi: 10.1177/002194366900600202. URL https://doi.org/10.1177/ 002194366900600202

    1969 doi

  17. [27]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  18. [28]

    What can large language models do in chemistry? a comprehensive benchmark on eight tasks

    Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662– 59688, 2023

    2023

  19. [29]

    Polypharmacy: evaluating risks and deprescribing

    Anne D Halli-Tierney, Catherine Scarbrough, and Dana Carroll. Polypharmacy: evaluating risks and deprescribing. American family physician, 100(1):32–38, 2019

    2019

  20. [30]

    Inductive representation learning on large graphs

    Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017

    2017

  21. [31]

    Promqa: Question answering dataset for multimodal procedural activity understanding

    Kimihiro Hasegawa, Wiradee Imrattanatrai, Zhi-Qi Cheng, Masaki Asada, Susan Holm, Yuran Wang, Ken Fukuda, and Teruko Mitamura. Promqa: Question answering dataset for multimodal procedural activity understanding. arXiv preprint arXiv:2410.22211, 2024

  22. [32]

    The ability of chatgpt in paraphrasing texts and reducing plagiarism: a descriptive analysis

    Soheil Hassanipour, Sandeep Nayak, Ali Bozorgi, Mohammad-Hossein Keivanlou, Tirth Dave, Abdulhadi Alotaibi, Farahnaz Joukar, Parinaz Mellatdoust, Arash Bakhshi, Dona Kuriyakose, et al. The ability of chatgpt in paraphrasing texts and reducing plagiarism: a descriptive analysis...

    2024

  23. [33]

    Prompt engineering of gpt-4 for chemical research: what can/cannot be done? Science and Technology of Advanced Materials: Methods, 3(1):2260300, 2023

    Kan Hatakeyama-Sato, Naoki Yamane, Yasuhiko Igarashi, Yuta Nabae, and Teruaki Hayakawa. Prompt engineering of gpt-4 for chemical research: what can/cannot be done? Science and Technology of Advanced Materials: Methods, 3(1):2260300, 2023

    2023

  24. [34]

    G-retriever: Retrieval-augmented generation for textual graph understanding and question answering

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876–132907, 2024

    2024

  25. [35]

    Consistency training by synthetic question generation for conversational question answering

    Hamed Hematian Hemati and Hamid Beigy. Consistency training by synthetic question generation for conversational question answering. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024 doi

  26. [36]

    Graphalign: Pretrain- ing one graph neural network on multiple graphs via feature alignment

    Zhenyu Hou, Haozhan Li, Yukuo Cen, Jie Tang, and Yuxiao Dong. Graphalign: Pretrain- ing one graph neural network on multiple graphs via feature alignment. arXiv preprint arXiv:2406.02953, 2024

    2024 arXiv

  27. [37]

    Grag: Graph retrieval-augmented generation

    Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. Grag: Graph retrieval-augmented generation. arXiv preprint arXiv:2405.16506, 2024

    2024 arXiv

  28. [38]

    Om- nimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Om- nimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024

    2024

  29. [39]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst....

    2025 doi

  30. [40]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  31. [41]

    Atlas: few-shot learning with retrieval augmented language models

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435

    2023

  32. [42]

    A survey on knowledge graphs: Representation, acquisition, and applications

    Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S Yu. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494–514, 2021

    2021

  33. [43]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  34. [44]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  35. [45]

    Flashrag: A modular toolkit for efficient retrieval-augmented generation research.CoRR, abs/2405.13576, 2024

    Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmented generation research.CoRR, abs/2405.13576, 2024. doi: 10.48550/ARXIV .2405.13576. URLhttps://doi.org/10.48550/arXiv.2405.13576

    2024 doi arXiv

  36. [46]

    PubMedQA: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Languag...

    2019

  37. [47]

    Biomedical question answering: A survey of approaches and challenges

    Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. Biomedical question answering: A survey of approaches and challenges. ACM Comput. Surv., 55(2), January 2022. ISSN 0360-0300. doi: 10.1145/349...

    2022 doi

  38. [48]

    Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval

    Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11):btad651, 2023

    2023

  39. [49]

    Wikimedia data for AI: a review of wikimedia datasets for NLP tasks and AI-assisted editing

    Isaac Johnson, Lucie-Aimée Kaffee, and Miriam Redi. Wikimedia data for AI: a review of wikimedia datasets for NLP tasks and AI-assisted editing. In Lucie Lucie-Aimée, Angela Fan, Tajuddeen Gwadabe, Isaac Johnson, Fabio Petroni, and Daniel van Strien, editors,Proceedings of the...

    2024

  40. [50]

    SemEval-2024 task 2: Safe biomedical natural language inference for clinical trials

    Mael Jullien, Marco Valentino, and André Freitas. SemEval-2024 task 2: Safe biomedical natural language inference for clinical trials. In Atul Kr. Ojha, A. Seza Do ˘gruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, and Aiala Rosá, editors,Pro- ceedings o...

    2024 doi

  41. [51]

    Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Prentice Hall PTR, 3rd edition, 2025. URL https://web.stanford. edu/~jurafsky/slp3/. Onl...

    2025

  42. [52]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Em...

    2020 doi

  43. [53]

    Pubchem 2025 update

    Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2025 update. Nucleic Acids Research, 53(D1):D1516–D1525, 2025

    2025

  44. [54]

    Benchmarking cognitive biases in large language models as evaluators

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Compu- tational Linguistics: AC...

    2024 doi

  45. [55]

    From data to commonsense reasoning: the use of large language models for explainable ai

    Stefanie Krause and Frieder Stolzenburg. From data to commonsense reasoning: the use of large language models for explainable ai. arXiv preprint arXiv:2407.03778, 2024

    2024 arXiv

  46. [56]

    Stitch: interaction networks of chemicals and proteins

    Michael Kuhn, Christian von Mering, Monica Campillos, Lars Juhl Jensen, and Peer Bork. Stitch: interaction networks of chemicals and proteins. Nucleic acids research, 36(suppl_1): D684–D688, 2007

    2007

  47. [57]

    SNAP Datasets: Stanford large network dataset collection

    Jure Leskovec and Andrej Krevl. SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, June 2014

    2014

  48. [58]

    Retrieval- augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing syst...

    2020

  49. [59]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594, 2024

    2024

  50. [60]

    Llms-as-judges: a comprehensive survey on llm-based evaluation methods

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024

    2024 arXiv

  51. [61]

    Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking

    Juanhui Li, Harry Shomer, Haitao Mao, Shenglai Zeng, Yao Ma, Neil Shah, Jiliang Tang, and Dawei Yin. Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking. Advances in Neural Information Processing Systems, 36:3853–3866, 2023

    2023

  52. [62]

    Bigsmiles: a structurally-based line notation for describing macromolecules

    Tzyy-Shyang Lin, Connor W Coley, Hidenobu Mochigase, Haley K Beech, Wencong Wang, Zi Wang, Eliot Woods, Stephen L Craig, Jeremiah A Johnson, Julia A Kalow, et al. Bigsmiles: a structurally-based line notation for describing macromolecules. ACS central science, 5(9): 1523–1531, 2019

    2019

  53. [63]

    Synthetic context generation for question generation

    Naiming Liu, Zichao Wang, and Richard Baraniuk. Synthetic context generation for question generation. arXiv preprint arXiv:2406.13188, 2024

    2024 arXiv

  54. [64]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. 14

    2024

  55. [65]

    Chatqa: Surpassing gpt-4 on conversational qa and rag

    Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. Advances in Neural Information Processing Systems, 37:15416–15459, 2024

    2024

  56. [66]

    Biochemistry, essential amino acids

    Michael J Lopez and Shamim S Mohiuddin. Biochemistry, essential amino acids. InStatPearls [Internet]. StatPearls Publishing, 2024

    2024

  57. [67]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7

    2019

  58. [68]

    Alejandro Lozano, Scott L Fleming, Chia-Chun Chiang, and Nigam Shah. Clinfo. ai: An open- source retrieval-augmented large language model system for answering medical questions using scientific literature. In PACIFIC SYMPOSIUM ON BIOCOMPUTING 2024, pages 8–23. World Scientific, 2023

    2024

  59. [69]

    ReACC: A retrieval-augmented code completion framework

    Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. ReACC: A retrieval-augmented code completion framework. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Comput...

    2022 doi

  60. [70]

    MoleculeQA: A dataset to evaluate factual accuracy in molecular comprehension

    Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, and Yu Li. MoleculeQA: A dataset to evaluate factual accuracy in molecular comprehension. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computati...

    2024

  61. [71]

    A study on the efficiency and gen- eralization of light hybrid retrievers

    Man Luo, Shashank Jain, Anchit Gupta, Arash Einolghozati, Barlas Oguz, Debojeet Chat- terjee, Xilun Chen, Chitta Baral, and Peyman Heidari. A study on the efficiency and gen- eralization of light hybrid retrievers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editor...

  62. [72]

    Biomedgpt: An open multimodal large language model for biomedicine

    Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Massimo Hong, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: An open multimodal large language model for biomedicine. IEEE Journal of Biomedical and Health Informatics, 2024

    2024

  63. [73]

    Realrag: Retrieval-augmented realistic image generation via self-reflective contrastive learning

    Yuanhuiyi Lyu, Xu Zheng, Lutao Jiang, Yibo Yan, Xin Zou, Huiyu Zhou, Linfeng Zhang, and Xuming Hu. Realrag: Retrieval-augmented realistic image generation via self-reflective contrastive learning. arXiv preprint arXiv:2502.00848, 2025

    2025

  64. [74]

    Smith Marsh

    Daphne E. Smith Marsh. Overview of generic drugs and drug naming, 2023/04 2023. URL https://www.merckmanuals.com/home/drugs/brand-name-and-generic-drugs/ overview-of-generic-drugs-and-drug-naming

    2023

  65. [75]

    doi: 10.18653/v1/2023.acl-short.139

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-short.139. URL https://aclanthology.org/2023.acl-short.139/

    2023 doi

  66. [76]

    Gnn-rag: Graph neural retrieval for large language model reasoning

    Costas Mavromatis and George Karypis. Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139, 2024

    2024 arXiv

  67. [77]

    A survey of multimodal retrieval- augmented generation

    Lang Mei, Siyu Mo, Zhihan Yang, and Chong Chen. A survey of multimodal retrieval- augmented generation. arXiv preprint arXiv:2504.08748, 2025

    2025 arXiv

  68. [78]

    Laypeople’s use of and attitudes toward large language models and search engines for health queries: Survey study

    Tamir Mendel, Nina Singh, Devin M Mann, Batia Wiesenfeld, and Oded Nov. Laypeople’s use of and attitudes toward large language models and search engines for health queries: Survey study. Journal of Medical Internet Research, 27:e64290, 2025. 15

    2025

  69. [79]

    What is polypharmacy? a systematic review of definitions

    Nashwa Masnoon, Sepehr Shakib, Lisa Kalisch-Ellett, and Gillian E Caughey. What is polypharmacy? a systematic review of definitions. BMC geriatrics, 17:1–10, 2017

    2017

  70. [80]

    Agentinstruct: Toward generative teaching with agentic flows

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502, 2024

    2024 arXiv

  71. [81]

    Extractive clinical question-answering with multianswer and multifocus questions: data set development and evaluation study

    Sungrim Moon, Huan He, Heling Jia, Hongfang Liu, Jungwei Wilfred Fan, et al. Extractive clinical question-answering with multianswer and multifocus questions: data set development and evaluation study. JMIR AI, 2(1):e41818, 2023

    2023

  72. [82]

    Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe

    Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: higher-order graph neural networks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirt...

    2019

  73. [83]

    Llama 3.3 | Model Cards and Prompt formats — llama.com

    Meta. Llama 3.3 | Model Cards and Prompt formats — llama.com. https://www.llama. com/docs/model-cards-and-prompt-formats/llama3_3/ , 2024. URL https://www. llama.com/docs/model-cards-and-prompt-formats/llama3_3/ . [Accessed 01-05- 2025]

    2024

  74. [84]

    Home - Gene - NCBI — ncbi.nlm.nih.gov

    National Library of Medicine. Home - Gene - NCBI — ncbi.nlm.nih.gov. https://www. ncbi.nlm.nih.gov/gene, 2025. [Accessed 09-05-2025]

    2025

  75. [85]

    OpenAI Platform — platform.openai.com

    OpenAI. OpenAI Platform — platform.openai.com. https://platform.openai.com/ docs/models/text-embedding-3-large , 2024. [Accessed 13-05-2025]

    2024

  76. [86]

    Introducing GPT-4.1 in the API — openai.com

    OpenAI. Introducing GPT-4.1 in the API — openai.com. https://openai.com/index/ gpt-4-1/, 2025. [Accessed 11-05-2025]

    2025

  77. [87]

    Neo4j Graph Database & Analytics – The Leader in Graph Databases — neo4j.com

    Neo4j. Neo4j Graph Database & Analytics – The Leader in Graph Databases — neo4j.com. https://neo4j.com/, 2025. [Accessed 13-05-2025]

    2025

  78. [88]

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Marine Carpuat, Marie-Catherine de Marneffe, and Iv...

    2022

  79. [89]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recog- nize and favor their own generations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Infor- mation Processing Systems , volume 37, pages ...

  80. [90]

    Knowledge graph-based question answering with electronic health records

    Junwoo Park, Youngwoo Cho, Haneol Lee, Jaegul Choo, and Edward Choi. Knowledge graph-based question answering with electronic health records. In Machine Learning for Healthcare Conference, pages 36–53. PMLR, 2021

    2021

  81. [91]

    OpenAI o3 and o4-mini System Card — openai.com

    OpenAI. OpenAI o3 and o4-mini System Card — openai.com. https://openai.com/ index/o3-o4-mini-system-card/ , 2025. URL https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf . [Accessed 01-05-2025]

    2025

  82. [92]

    Spiqa: A dataset for multimodal question answering on scientific papers

    Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for multimodal question answering on scientific papers. NeurIPS, 2024. 16

    2024

  83. [93]

    Line Notation (SMILES and InChI), aug 11 2020

    LibreTexts project. Line Notation (SMILES and InChI), aug 11 2020. URL https://chem.libretexts.org/Courses/Fordham_University/Chem1102%3A_ Drug_Discovery_-_From_the_Laboratory_to_the_Clinic/05%3A_Organic_ Molecules/5.08%3A_Line_Notation_(SMILES_and_InChI). [Online; accessed 20...

    2020

  84. [94]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 7f1f0218e45f5414c79c0679633e47bc-Paper-Conference.pdf

    2024

  85. [95]

    Coquad: a covid-19 question answering dataset system, facilitating research, benchmarking, and practice

    Shaina Raza, Brian Schwartz, and Laura C Rosella. Coquad: a covid-19 question answering dataset system, facilitating research, benchmarking, and practice. BMC bioinformatics, 23(1): 210, 2022

    2022

  86. [96]

    Genetics: what is a gene? Nature, 441(7092), 2006

    Helen Pearson. Genetics: what is a gene? Nature, 441(7092), 2006

    2006

  87. [97]

    Modeling relational data with graph convolutional networks

    Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, proceedings...

    2018

  88. [98]

    Introduction to informa- tion retrieval, volume 39

    Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to informa- tion retrieval, volume 39. Cambridge University Press Cambridge, 2008

    2008

  89. [99]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 23...

    2016 doi

  90. [100]

    sentence-transformers/all-MiniLM-L6-v2 · Hugging Face — hugging- face.co

    sentence transformers. sentence-transformers/all-MiniLM-L6-v2 · Hugging Face — hugging- face.co. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 ,

  91. [101]

    User prompts vs

    Rocco Savage. User prompts vs. system prompts: What’s the difference? — regie.ai.https:// www.regie.ai/blog/user-prompts-vs-system-prompts , 2025. URL https://www. regie.ai/blog/user-prompts-vs-system-prompts . [Accessed 05-05-2025]

    2025

  92. [102]

    LexicalRichness: A small module to compute textual lexical richness, 2022

    Lucas Shen. LexicalRichness: A small module to compute textual lexical richness, 2022. URL https://github.com/LSYS/lexicalrichness

    2022

  93. [103]

    REPLUG: Retrieval-augmented black-box language models

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmented black-box language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North...

    2024

  94. [104]

    Simple entity-centric questions challenge dense retrievers

    Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Empirical Methods in Natur...

    2021 doi

  95. [105]

    Toward expert-level medical question answering with large language models

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8, 2025. 17

    2025

  96. [106]

    Beyond the imitation game: quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Transactions on ...

    2023

  97. [107]

    Imagerag: Dynamic image retrieval for reference-guided image generation

    Rotem Shalev-Arkushin, Rinon Gal, Amit H Bermano, and Ohad Fried. Imagerag: Dynamic image retrieval for reference-guided image generation. arXiv preprint arXiv:2502.09411, 2025

    2025

  98. [108]

    90 https://www.asbmb.org/asbmb-today/opinions/031222/ 90-of-drugs-fail-clinical-trials , 2022

    Duxin Sun. 90 https://www.asbmb.org/asbmb-today/opinions/031222/ 90-of-drugs-fail-clinical-trials , 2022. [Accessed 13-05-2025]

    2022

  99. [109]

    Genetic analysis of a female patient with x-linked lymphoproliferative disease type 2: a case report

    Yalin Sun, Shu Teng, Wen Li, Huaping Wang, and Zhenghong Qi. Genetic analysis of a female patient with x-linked lymphoproliferative disease type 2: a case report. Journal of Medical Case Reports, 19(1):193, Apr 2025. ISSN 1752-1947. doi: 10.1186/s13256-025-05237-8. URL https:/...

    2025 doi

  100. [110]

    Ever Wonder How Drugs Are Named? Read On | Pfizer — pfizer.com

    Kate Silver. Ever Wonder How Drugs Are Named? Read On | Pfizer — pfizer.com. https://www.pfizer.com/news/articles/ever_wonder_how_drugs_ are_named_read_on, 2020. [Accessed 22-04-2025]

    2020

  101. [111]

    Multimodal{qa}: complex question answering over text, tables and images

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodal{qa}: complex question answering over text, tables and images. In International Conference on Learning Representations, 2021. URL http...

    2021

  102. [112]

    MolLM: a unified language model for integrating biomedical text with 2D and 3D molecular representations

    Xiangru Tang, Andrew Tran, Jeffrey Tan, and Mark B Gerstein. MolLM: a unified language model for integrating biomedical text with 2D and 3D molecular representations. Bioinformat- ics, 40(Supplement_1):i357–i368, 06 2024. ISSN 1367-4811. doi: 10.1093/bioinformatics/ btae260. U...

    2024 doi

  103. [113]

    From show to tell: A survey on deep learning-based image captioning

    Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From show to tell: A survey on deep learning-based image captioning. IEEE transactions on pattern analysis and machine intelligence, 45(1):539–559, 2022

    2022

  104. [114]

    Relational graph convolutional networks: a closer look

    Thiviyan Thanapalasingam, Lucas van Berkel, Peter Bloem, and Paul Groth. Relational graph convolutional networks: a closer look. PeerJ Computer Science, 8:e1073, 2022

    2022

  105. [115]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meet...

    2023 doi

  106. [116]

    The string database in 2023: protein–protein association networks and functional enrichment analyses for any sequenced genome of interest

    Damian Szklarczyk, Rebecca Kirsch, Mikaela Koutrouli, Katerina Nastou, Farrokh Mehryary, Radja Hachilif, Annika L Gable, Tao Fang, Nadezhda T Doncheva, Sampo Pyysalo, et al. The string database in 2023: protein–protein association networks and functional enrichment analyses fo...

    2023

  107. [117]

    Understanding shannon’s entropy metric for information, 2014

    Sriram Vajapeyam. Understanding shannon’s entropy metric for information, 2014. URL https://arxiv.org/abs/1405.2061

    2014 arXiv

  108. [118]

    Graph Attention Networks

    Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Represen- tations, 2018. URL https://openreview.net/forum?id=rJXMpikCZ

    2018

  109. [119]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025. URL https://qwenlm.github.io/blog/qwen3/

    2025

  110. [120]

    Txgemma: Efficient and agentic llms for therapeutics

    Eric Wang, Samuel Schmidgall, Paul F Jaeger, Fan Zhang, Rory Pilgrim, Yossi Matias, Joelle Barral, David Fleet, and Shekoofeh Azizi. Txgemma: Efficient and agentic llms for therapeutics. arXiv preprint arXiv:2504.06196, 2025. 18

    2025 arXiv

  111. [121]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of...

    2024 doi

  112. [122]

    Home | United States Medical Licensing Examination — usmle.org

    USMLE. Home | United States Medical Licensing Examination — usmle.org. https: //www.usmle.org/, 2025. [Accessed 13-05-2025]

    2025

  113. [123]

    Beyond the limits: a survey of techniques to extend the context length in large language models

    Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Ar- maghan Eshaghi. Beyond the limits: a survey of techniques to extend the context length in large language models. In Proceedings of the Thirty-Third International Joint Conference on Artifi- cial...

    2024 doi

  114. [124]

    Retrieval-based controllable molecule generation

    Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anand- kumar. Retrieval-based controllable molecule generation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=vDFA1tpuLvk

    2023

  115. [125]

    Protein-targeting drug discovery, 2023

    Caterina Vicidomini and Giovanni N Roviello. Protein-targeting drug discovery, 2023

    2023

  116. [126]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  117. [127]

    Smiles, a chemical language and information system

    David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28 (1):31–36, 1988

    1988

  118. [128]

    Retrieval-augmented perception: High-resolution image perception meets visual rag

    Wenbin Wang, Yongcheng Jing, Liang Ding, Yingjie Wang, Li Shen, Yong Luo, Bo Du, and Dacheng Tao. Retrieval-augmented perception: High-resolution image perception meets visual rag. arXiv preprint arXiv:2503.01222, 2025

    2025 arXiv

  119. [129]

    Synthetic multimodal question generation

    Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Khoshfetrat Pakazad, Tongshuang Wu, and Graham Neubig. Synthetic multimodal question generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Com- putational Lin...

    2024

  120. [130]

    Stark: Benchmarking llm retrieval on textual and relational knowledge bases.Advances in Neural Information Processing Systems, 37:127129–127153, 2024

    Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis Ioannidis, Karthik Subbian, James Y Zou, and Jure Leskovec. Stark: Benchmarking llm retrieval on textual and relational knowledge bases.Advances in Neural Information Processing Systems, 3...

    2024

  121. [131]

    Ioannidis, Huzefa Rangwala, and RISHITA ANUBHAI

    Zifeng Wang, Zichen Wang, Balasubramaniam Srinivasan, Vassilis N. Ioannidis, Huzefa Rangwala, and RISHITA ANUBHAI. Biobridge: Bridging biomedical foundation models via knowledge graphs. In The Twelfth International Conference on Learning Representations,

  122. [132]

    URL https://openreview.net/forum?id=jJCeMiwHdH

  123. [133]

    Crag-comprehensive rag benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470–10490, 2024

    2024

  124. [134]

    Question-analysis prompting improves LLM performance in reasoning tasks

    Dharunish Yugeswardeenoo, Kevin Zhu, and Sean O’Brien. Question-analysis prompting improves LLM performance in reasoning tasks. In Xiyan Fu and Eve Fleisig, editors, Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Re...

    2024 doi

  125. [135]

    Pubmed — Wikipedia, the free encyclopedia, 2025

    Wikipedia contributors. Pubmed — Wikipedia, the free encyclopedia, 2025. URL https: //en.wikipedia.org/w/index.php?title=PubMed&oldid=1289629349. [Online; ac- cessed 13-May-2025]

    2025

  126. [136]

    Scientific large language models: A survey on biological & chemical domains

    Qiang Zhang, Keyan Ding, Tianwen Lv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Mengyao Zhang, Jinlu Zhang, Jiyu Cui, Renjun Xu, Hongyang Chen, Xiaohui Fan, Huabin Xing, and Huajun Chen. Scientific ...

    2025 doi

  127. [137]

    doi: 10.18653/v1/2024.findings-emnlp.759

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.759. URL https://aclanthology.org/2024.findings-emnlp.759/

    2024 doi

  128. [138]

    Benchmarking large language models for news summarization

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57, 2024

    2024

  129. [139]

    Bridging the gap: Generating a comprehensive biomedical knowledge graph question answering dataset

    Xi Yan, Patrick Westphal, Jan Seliger, and Ricardo Usbeck. Bridging the gap: Generating a comprehensive biomedical knowledge graph question answering dataset. In ECAI 2024, pages 1198–1205. IOS Press, 2024

    2024

  130. [140]

    Empower large language model to perform better on industrial domain-specific question answering

    Fangkai Yang, Pu Zhao, Zezhong Wang, Lu Wang, Bo Qiao, Jue Zhang, Mohit Garg, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Empower large language model to perform better on industrial domain-specific question answering. In Mingxuan Wang and Imed Zitouni, editors, Proceedi...

    2023 doi

  131. [141]

    Realm: Rag-driven enhancement of multimodal electronic health records analysis via large language models

    Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, et al. Realm: Rag-driven enhancement of multimodal electronic health records analysis via large language models. arXiv preprint arXiv:2402.07016, 2024

    2024 arXiv

  132. [142]

    Modeling polypharmacy side effects with graph convolutional networks

    Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13):i457–i466, 2018. 20 A Knowledge Base Sampling

    2018

  133. [143]

    Online health information–seeking in the era of large language models: Cross-sectional web-based survey study

    Hye Sun Yun and Timothy Bickmore. Online health information–seeking in the era of large language models: Cross-sectional web-based survey study. Journal of Medical Internet Research, 27:e68560, 2025

    2025

  134. [145]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations,

  135. [148]

    Development of a large-scale medical visual question-answering dataset.Communications Medicine, 4(1):277, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Development of a large-scale medical visual question-answering dataset.Communications Medicine, 4(1):277, 2024

    2024

  136. [149]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023

    2023 arXiv

  137. [152]

    §3.1.1) contains ~4M drug-drug interactions

    The base knowledge graph (c.f. §3.1.1) contains ~4M drug-drug interactions. However, as we explain in Appendix D.1, most drugs do not have common/generic names. We require our drug entities to have common names for acquiring background-text data to generate questions (c.f. §3....

  138. [153]

    However, most of these edges are absent in the STITCH database (Appendix C)

    The total number of proteins interacting with each of the 494 drugs is 198, yielding ~15K DPI edges. However, most of these edges are absent in the STITCH database (Appendix C). This ultimately leaves a total of 314 DPI edges

  139. [154]

    relation

    We explain why there are no protein-protein (PPI) edges in Appendix H. Our resulting graph thus contains ~18.5K DDIs and 314 DPIs, yielding a total of ~18.9 edges, 494 drug and 198 protein nodes. B SMILES When querying PubChem for drug data, it returns two kinds ofSMILES, cano...

  140. [155]

    Do your best to make the material as dense as possible by using complex domain-specific jargon

  141. [156]

    DO NOT ADD any external information, i.e., strictly utilise the provided context only

  142. [157]

    Ignore historical details in the text, such as year of discovery, etc., focusing only on the core biological details

  143. [158]

    Here are the key points

    Avoid including unnecessary text such as "Here are the key points" or other filler language. DRUG NAME: {} DRUG INFORMATION: {} Figure A2: Text Post-Processing Prompt abilities to reason across domains. By targeting the limitations of the above studies, we aim to provide a val...

  144. [159]

    Standard RAG still shows promise against IRCoT [ 45] often outperforming it and other variants on different datasets

  145. [160]

    This increases the overall number of API calls, making it expensive to implement

    IRCoT is a sequential framework, meaning that each step of reasoning depends on the prior retrieval and reasoning step. This increases the overall number of API calls, making it expensive to implement. For context, all of our generations were done in batch processing mode to l...

  146. [161]

    Finally, we do attempt to use Google’s Gemini 2.0 Flash [21]

    As a result of the sequential nature of IRCoT, batch processing is not possible, thereby limiting scalability. Finally, we do attempt to use Google’s Gemini 2.0 Flash [21]. However, the results were quite poor, and we noticed that it struggled to follow our prompts. As such, w...

  147. [162]

    Focus only on interactions relevant under physiological conditions (e.g., hydrogen bonding, steric clashes, electrostatic interactions)

  148. [163]

    Do not infer interactions from external knowledge or assumptions about the drug identities

  149. [164]

    Only report interactions supported by structural features in the SMILES

  150. [165]

    stringId_A

    If no interaction exists, respond with ’NONE’. For any interaction you do identify, format your response as: • INTERACTION: [Specific name of the interaction] • MECHANISM: [Brief explanation of how/why this interaction occurs] • EVIDENCE: [Direct structural features or groups ...

  151. [166]

    muscle-pain

    CLARITY: How difficult is the question’s language? SCORES: (a) 0 - Easy; Straightforward and uses common phrases like "muscle-pain". (b) 1 - Medium; Overall comprehensible but uses some domain-specific jargon like "hepatotoxicity". (c) 2 - Hard; Quite difficult and requires go...

  152. [167]

    SCORES: (a) 0 - Low; Completely ignores the given information

    COVERAGE: Does the question make use of the provided modalities (text/triple)? Note, the question does *NOT* need to utilise *ALL* of the given text. SCORES: (a) 0 - Low; Completely ignores the given information. (b) 1 - Medium; Uses only information for one entity or one moda...

  153. [168]

    (b) 1 - Okay; Most of the question relies on the provided data, but some assumptions are made

    ASSUMPTIONS: Does the question include information beyond what is provided? SCORES: (a) 0 - Bad; A lot of the question’s data is absent from the provided data. (b) 1 - Okay; Most of the question relies on the provided data, but some assumptions are made. (c) 2 - Best; Relies s...

  154. [169]

    (b) 1 - Maybe; Can be potentially derived but requires additional data to infer

    INFERABLE: Can the answer be derived from the provided information? SCORES: (a) 0 - No; The answer is irrelevant in regard to the question. (b) 1 - Maybe; Can be potentially derived but requires additional data to infer. (c) 2 - Yes; There exists entailment between the answer ...

  155. [170]

    Here is my evaluation

    Do not write filler text such as "Here is my evaluation", etc

  156. [171]

    No need to repeat the descriptions for each score.>""" BACKGROUND INFORMATION:{} QUESTION: {} ANSWER: {} Figure A4: Prompt for LLM-based evaluation

    Provide your output as, - METRIC REASONING: <Brief explanation of thought process for the metric.> - METRIC SCORE: <0, 1, 2 based on the guidelines. No need to repeat the descriptions for each score.>""" BACKGROUND INFORMATION:{} QUESTION: {} ANSWER: {} Figure A4: Prompt for L...

  157. [172]

    The answer must be either Drug 1 or Drug 2

    Write exactly one question integrating the background knowledge of both drugs and their relationship. The answer must be either Drug 1 or Drug 2

  158. [173]

    The question may be as complex as desired, but it must be answerable

  159. [174]

    Do NOT mention the drugs by name in the question; use only their background descriptions

  160. [175]

    The question should **specifically test knowledge of the triple-described relationship or interaction, not just isolated facts about either drug.**

  161. [176]

    The answer should be only the name of the correct drug

  162. [177]

    Output in the following format: Question: Answer: DRUG 1 NAME: {} DRUG 1 BACKGROUND INFORMATION: {} DRUG 2 NAME: {} DRUG 2 BACKGROUND INFORMATION: {} DRUG-DRUG INTERACTION TRIPLE (subject-predicate-object): {} Figure A5: Question Generation Prompt For Bio-based 1-hop DDIs. Mod...

  163. [178]

    • MolLM is an LLM-based retriever that fine-tunes a BERT checkpoint on text, SMILES, and molecular graphs

    checkpoint for query-article retrieval. • MolLM is an LLM-based retriever that fine-tunes a BERT checkpoint on text, SMILES, and molecular graphs. As such, its inclusion provides insights into retrieval for SMILES data. • DPR is a simple BERT-based retriever trained for questi...

  164. [179]

    as our query encoder. Model Hits@5 Hits@10 Hits@15 Recall@5 MRR Neo4j 0.04/0.04 0.07/0.08 0.07/0.09 0.01 0.03 Graph Convolution Network 0.01/0.02 0.02/0.04 0.03/0.06 0.01 0.01 Graph SAGE Network 0.01/0.02 0.01/0.04 0.01/0.06 0 0.01 Graph Attention Network 0.02/0.05 0.04/0.08 0...

  165. [180]

    It first builds indexes, i.e., builds a graph database using the provided set of triples (entity- relation-entity)

  166. [181]

    When a query arrives,

    Next, it creates embeddings (using all-MiniLM-L6-v2 [100]) for each relation in the database to locate relationships semantically similar to the one described by the question. When a query arrives,

  167. [182]

    It first does a best attempt to find entities in the question via Named Entity Recognition (implemented with Flair models [6])

  168. [183]

    Next, it looks for relations closest to the query using the same sentence encoder (all-MiniLM-L6-v2 [100]) used to embed the relations

  169. [184]

    All of these steps can be modified

    Finally, it returns the top-k triples that have these entities and relationships. All of these steps can be modified. We tried different settings (such as using a simpler regex-based entity recogniser, different embedding models such as PubMedBERT [24], etc.) and landed on thi...

  170. [185]

    The overall distribution of question lengths is shown in Figure A7

    Question Length: The average question length (in tokens7) is much higher than existing biomedical QA datasets such as PubMedQA [46] and even datasets designed specifically for long-context QA such as QuALITY [88] (66 v/s 14 and 12.5 respectively) indicating a need for deep rea...

  171. [186]

    It is used to gauge linguistic diversity

    Type-to-Token (TTR) Ratio : TTR [ 102] is defined as the (total number of unique words/total words) in a text piece. It is used to gauge linguistic diversity. Lower val- ues imply repetitive text, i.e., a small number of unique words and vice versa. With an average TTR of 0.84...

  172. [187]

    In other words, this measure will tell us if the questions are repetitive or diverse in content

    Shannon Entropy: We measure Shannon Entropy [117] to gauge semantic coverage. In other words, this measure will tell us if the questions are repetitive or diverse in content. Entropy (in bits) lies between 0 andlog2(k) where k is the number of categories. In our case, we use q...

  173. [188]

    We consider the average dependency tree depth for all questions (10.86) as a measure of linguistic difficulty

    Dependency Tree Depth: A dependency tree8 [51] provides an overview of the grammatical roles (determiner, noun-phrase, etc.) of each word and the relationships (subject, object, etc.) that exist between them. We consider the average dependency tree depth for all questions (10....

  174. [189]

    Proteins only have background text

    Although multi-modal, only drugs in our dataset benefit from two modalities, i.e., SMILES and text. Proteins only have background text. This limits the type of questions we can craft for it

  175. [190]

    There are more drugs with SMILES than text in the knowledge graph, which creates an imbalance between the SMILES set and the text corpus in terms of size

  176. [191]

    This potentially limits large-scale evaluation due to associated costs

    As our questions are very specific to the healthcare domain, they can only be qualitatively evaluated by domain practitioners. This potentially limits large-scale evaluation due to associated costs. 31

  177. [2019]

    doi: 10.18653/v1/D19-1259

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1259. URL https: //aclanthology.org/D19-1259/

    doi

  178. [2020]

    URL https://openreview.net/forum?id=SkeHuCVFDr

  179. [2021]

    [Accessed 13-05-2025]

    2025

  180. [2023]

    URL https://aclanthology.org/2023

    Association for Computational Linguistics. URL https://aclanthology.org/2023. gem-1.16/

    2023

  181. [2024]

    doi: 10.18653/v1/2024.findings-emnlp.216

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.216. URL https://aclanthology.org/2024.findings-emnlp.216/

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.