Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A chemistry benchmark shows that even with the right context, LLMs struggle to chain several facts.

desk verdict A useful new chemistry multi-hop QA benchmark whose headline difficulty numbers are softened by an outcome-dependent exclusion filter; worth refereeing with revisions. read the letter →

arxiv 2504.16414 v2 pith:S6IOOZVM submitted 2025-04-23 cs.CL

Mohammad Khodadad , Ali Shiraee Kasmaee , Mahdi Astaraki , Nicholas Sherck , Hamidreza Mahyar , Soheila Samiee This is my paper · ORCID
classification cs.CL
keywords multi-hopreasoningchemistryquestionansweringknowledgegraphconstructionnamedentityrecognitionretrieval-augmentedgenerationlargelanguagemodelevaluationcompositionalbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a new chemistry dataset of 971 multi-hop questions built from recent chemistry preprints and a knowledge graph, then evaluates 13 large language models in two settings: with the full source text supplied and with no context at all. The authors report that even the best model answers only about 84% correctly when given perfect context, and the best no-context model falls below 48%. Their central claim is that compositional reasoning—joining facts from different sources into one answer—remains a genuine bottleneck for current models, and that retrieval context roughly doubles accuracy but does not eliminate errors. The paper also presents the data-generation pipeline as a reusable recipe for building hard multi-hop reasoning benchmarks in other scientific domains.

What carries the argument

The central mechanism is a knowledge graph built from the introductions of chemistry preprints. Chemical entities are extracted and verified through a named entity recognition model refined by a language model, relations between co-occurring entities are extracted as triplets, and nodes are enriched with external database information. A randomized breadth-first search samples paths whose edges come from different source documents, guaranteeing that no single passage can answer the question. One-hop questions are then generated per triplet and combined into a single multi-hop question by reverse chaining, so the final answer is the first entity in the path and the question cannot be solved without composing evidence across documents.

What would settle it

Ask expert chemists to judge a random sample of the questions that the paper excluded because all evaluated models answered them incorrectly. If a substantial share of those excluded questions are judged unambiguous, factual, and answerable from the provided context, then the exclusion rule biases the benchmark toward easier questions and the reported error rates need to be recalculated.

Watch

Extended reading notes

Core claim

The paper's claim, stated on its own terms, is that even state-of-the-art large language models, including reasoning-specialised models that scale test-time computation, face significant challenges in multi-hop compositional reasoning within chemistry. The strongest evidence is that with full relevant context supplied, the top correctness rate is 84.35%, while without context the top rate is 47.58%; context nearly doubles average performance across models. Because even perfect retrieval still leaves a substantial error rate, the paper concludes that the limiting factor is compositional reasoning itself rather than access to the necessary facts. The authors further claim that their automated pipeline—named entity recognition, relation extraction, knowledge graph construction, and reverse-chained question aggregation—produces questions requiring one to four hops from distinct source documents, and that an expert-filtered subset supports the benchmark's validity.

Load-bearing premise

The benchmark's scores are only meaningful if questions that every tested model got wrong were genuinely ambiguous or flawed; if some were merely hard, the reported correctness rates are inflated.

Editorial extensions

If this is right

  • Providing full relevant context roughly doubles correctness for both reasoning and non-reasoning models, so retrieval augmentation remains a high-leverage intervention for multi-hop scientific question answering.
  • Because even perfect context leaves a measurable error floor, multi-hop QA systems should not assume that retrieving the right documents is sufficient for a correct answer.
  • Reasoning-specialised models use fewer thinking tokens when context is supplied, suggesting that external evidence reduces the burden of internal knowledge search during reasoning.
  • Accuracy is higher for single-hop than for multi-hop questions in the context-provided setting, but remains fairly flat across two to four hops, indicating that hop count alone does not fully capture difficulty.
  • The pipeline can generate large domain-specific reasoning datasets with only a sample of expert verification, lowering the cost of building new benchmarks in specialised fields.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exclusion of questions that every evaluated model answered incorrectly could inflate measured capability if some of those questions were valid but simply hard; re-testing the excluded pool with stronger future models, or with expert chemists, would calibrate the benchmark's difficulty.
  • The reverse-chaining construction always produces a single entity as the final answer, so an extension to relational, numerical, or procedural answers would test a different and potentially harder form of composition.
  • A natural next experiment is iterative retrieval, where the model receives one source at a time and decides what to fetch next, matching the paper's stated limitation of using a single full-context block.
  • If the error floor persists when context is guaranteed correct, a useful diagnostic is to probe each hop independently before asking the model to combine the answers, thereby separating failure to retrieve a fact from failure to compose facts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ChemKGMultiHopQA, a chemistry-domain multi-hop question-answering benchmark of 971 questions generated from ChemRxiv articles through named entity recognition, knowledge graph construction, and OpenAI o3-mini/gpt-4o-based question generation and verification. The authors evaluate 13 large language models in two settings: with full supporting context and without context. They report that providing context nearly doubles correctness rates, that the best model with context (Claude Sonnet 3.7 with extended thinking) reaches 84.35%, and that the best no-context model (o3-mini) reaches 47.58%. The central claim is that even state-of-the-art models struggle with multi-hop compositional reasoning in chemistry, and that even perfect retrieval with full context does not eliminate reasoning errors.

Significance. If the benchmark is valid, it fills a real gap in domain-specific multi-hop reasoning evaluation: existing resources such as ChemLitQA-multi are smaller, mostly single-entity chains, or not chemistry-specific. The automated pipeline is a plausible contribution, and the design choices—short answers, distinct source documents per hop, and context/no-context comparison—are sensible. The evaluation of 13 models and the public release of the dataset and code are strengths. However, the validity evidence is currently too thin to support the strongest claim that residual errors are attributable to compositional reasoning failures rather than to ambiguous, invalid, or incorrectly grounded benchmark items. The paper is honest in disclosing the outcome-dependent filter (S7.3) and the generation-model overlap (S7.5), but those disclosures directly weaken the headline inference.

major comments (4)
  1. [S7.3] The filtering rule described in S7.3—'To minimize ambiguity, we excluded questions that were answered incorrectly by all evaluated models from the benchmark'—is outcome-dependent and affects the central difficulty and validity claims. The paper acknowledges that only a subset of the excluded questions was manually assessed and that 'most' were categorized as having multiple valid answers, leaving open the possibility that valid hard questions were removed. Because the reported correctness rates are computed on the post-filter set, they overstate model capability relative to the original generated pool, and the observation that models make residual errors on the remaining questions cannot be interpreted as evidence about compositional reasoning unless the authors show that the filter does not preferentially remove valid difficult items. Please report statistics on the full generated pool, or re-run the evaluation on a random sample not subjected to this filter, and quantify how the headline numbers change.
  2. [5.1 and S7.4] The expert validation is too small to support the inference that the residual errors are reasoning failures. Out of 52 pilot questions, 12 (23%) were dropped due to low evaluator confidence, and of the remaining 40, 5 (12.5%) were rated Poor. If a similar rate holds in the full benchmark, roughly 120 of the 971 items could be defective, a number comparable to or larger than the best model's 15.65% error rate with context. Please report the residual error rates restricted to the expert-validated Good and Ok items, release the per-question expert annotations, and conduct a substantially larger validation sample before attributing residual errors to compositional reasoning.
  3. [3.2 and S7.5] There is a circularity concern in the evaluation design: the questions are generated with o3-mini and verified with gpt-4o, and the same models are then evaluated on those questions. Section S7.5 acknowledges that 'o3-mini was primarily used to generate the questions, which may have introduced a slight bias, resulting in its minor improvement in correctness.' This undermines the no-context model comparison, particularly the claim that OpenAI models perform better because of richer pretraining data. Please quantify the effect by evaluating on a held-out set generated with different models, or by reporting performance separately on items that the evaluated OpenAI models did and did not generate, and disclose the generation overlap per question.
  4. [4, evaluation metric] The correctness metric uses exact match followed by a GPT-4o binary assessment for non-matching answers. Since gpt-4o is also one of the evaluated models and is part of the generation and verification pipeline, the judge may have systematic preferences for outputs from the same model family, potentially inflating the correctness rates of OpenAI models. Please provide a sample of judge decisions with human agreement rates, and clarify how near-miss or partially correct answers are treated.
minor comments (6)
  1. [4.2 and Table S5] Mistral Large's performance on the HotpotQA chemistry subset (4.59% without context and 0.92% with context) is dramatically lower than all other models and is inconsistent with its ChemKGMultiHopQA results; this anomaly is never discussed and could distort the dataset comparison in Figure 4.
  2. [S7.9] The text contains an unresolved 'Figure ??' placeholder in the sentence referencing Figure S13; this should be fixed.
  3. [Table 2] The HotpotQA-Chemistry row lists 'no chemical entities' under entity types, which is confusing because the subset is defined by chemistry categories; please clarify whether the intended meaning is that the questions are not grounded in chemistry-specific knowledge bases.
  4. [4.1] Model names are used inconsistently, e.g., 'Qwen QWEN 32B' and 'Deepseek R1 Distil QWEN 32B'; please standardize the model names and match them exactly to the provider names in Table 3.
  5. [S7.4, Table S4] The 'Avg. Corr.' columns in Table S4 are not directly interpretable because the denominator (number of models considered) is not stated; please report the number of models and standard deviations or confidence intervals.
  6. [4] The statement 'The full Q&A dataset, along with the evaluation code, is accessible here' does not include a URL or repository identifier in the manuscript text; please provide a working link.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: model scores are empirical measurements, and the benchmark construction does not encode the measured outcomes.

full rationale

I find no circular derivation in this paper. The central claims are empirical evaluations of LLMs on a newly constructed chemistry multi-hop QA benchmark; no reported correctness rate is obtained by fitting a parameter to the outcome it is then said to predict. The construction pipeline (knowledge-graph extraction, path sampling, one-hop question formulation, multi-hop aggregation, and verification) runs upstream of evaluation and does not encode the measured accuracy values. The outcome-dependent exclusion of questions answered incorrectly by all evaluated models (S7.3) is a genuine benchmark-validity concern because it can make the remaining set easier and can bias difficulty comparisons, but it does not make any model's score equivalent to an input by construction, and the paper explicitly discloses the generation/evaluation overlap: 'o3-mini was primarily used to generate the questions, which may have introduced a slight bias, resulting in its minor improvement in correctness' (S7.5). The HotpotQA chemistry subset provides an external benchmark anchored to a different, human-curated source, and the expert review, though limited to 40 high-confidence questions, is independent of the automated pipeline. No load-bearing self-citations or imported uniqueness theorems appear. For these reasons, the observed residual errors and the claim that perfect retrieval does not eliminate reasoning errors are empirical findings rather than artifacts of definitional circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on benchmark validity: chemical facts extracted from ChemRxiv, LLM-based verification, and model-based filtering. No free parameters are fitted; the main uncharged premises are about data quality and representativeness. No new physical or theoretical entities are introduced; all nodes are chemical entities drawn from literature and external databases.

assumptions (5)
  • domain assumption ChemRxiv introduction text is a sufficient source of factual, objective chemical statements for constructing valid ground-truth QA pairs.
    Invoked in Section 3.1 and S7.1.1, where the first 500 words of each introduction are the only textual basis for entities, relations, and questions.
  • domain assumption LLM-based verification with gpt-4o and o3-mini reliably determines factuality, answerability, and uniqueness of answers.
    Section S7.2.4 uses LLM prompts as the primary verifier for all questions; only 52 questions receive expert review.
  • ad hoc to paper Questions answered incorrectly by all evaluated models are ambiguous or flawed, rather than simply difficult.
    S7.3 and S7.4 state these questions were excluded 'to minimize ambiguity'; this unproven filter reshapes the benchmark and inflates correctness rates.
  • domain assumption The 40 high-confidence expert-reviewed questions are representative of the full 971-question dataset.
    Section 5.1 reports expert review of 52 randomly selected questions, with 12 dropped, then generalizes the 87.5% approval to the whole benchmark.
  • domain assumption Graph paths with edges from distinct source texts require genuine multi-hop reasoning rather than lexical shortcutting.
    Section 3.2 and S7.2.1 enforce distinct source texts per edge; shortcut edges appear in 9.9% of questions but are treated as acceptable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study." pith.science (2026). https://pith.science/paper/S6IOOZVM

@misc{pith2026250416414,
  author       = {Pith},
  title        = {Pith review of: Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6IOOZVM}},
  note         = {Machine review of arXiv:2504.16414}
}
read the original abstract

In this study, we introduced a new benchmark consisting of a curated dataset and a defined evaluation process to assess the compositional reasoning capabilities of large language models within the chemistry domain. We designed and validated a fully automated pipeline, verified by subject matter experts, to facilitate this task. Our approach integrates OpenAI reasoning models with named entity recognition (NER) systems to extract chemical entities from recent literature, which are then augmented with external knowledge bases to form a comprehensive knowledge graph. By generating multi-hop questions across these graphs, we assess LLM performance in both context-augmented and non-context augmented settings. Our experiments reveal that even state-of-the-art models face significant challenges in multi-hop compositional reasoning. The results reflect the importance of augmenting LLMs with document retrieval, which can have a substantial impact on improving their performance. However, even perfect retrieval accuracy with full context does not eliminate reasoning errors, underscoring the complexity of compositional reasoning. This work not only benchmarks and highlights the limitations of current LLMs but also presents a novel data generation pipeline capable of producing challenging reasoning datasets across various domains. Overall, this research advances our understanding of reasoning in computational linguistics.

Figures

Figures reproduced from arXiv: 2504.16414 by the authors.

Figure 1
Figure 1. An Overview of the knowledge graph generation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the QA generation Pipeline. from these sources, as well as edges representing the relationships extracted from the textual data [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of selected models based on correctness rate, cost, and latency. The cost axis [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of LLMs’ performance on the chemical subset of HotPotQA with the curated [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Impact of reasoning and context on models’ Correctness Rate (left panel) and latency (right [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Analysis of the impact of the number of hops on models’ performance in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering

    cs.CL 2026-01 conditional novelty 7.0 of 10

    On ChemKGMultiHopQA, iterative retrieval-reasoning outperformed oracle gold-context static RAG for all 11 LLMs tested, with gains up to 25.6 percentage points.

Reference graph

Works this paper leans on

53 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  2. [2]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    arXiv 2022

  3. [3]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023

    2023

  4. [4]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024. 10

    work page 2024

  5. [5]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though

    Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.04682, 2025

    arXiv 2025

  6. [6]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020

    2020

  7. [7]

    Neurosymbolic ai: the 3rd wave

    Artur d’Avila Garcez and Luis C Lamb. Neurosymbolic ai: the 3rd wave. arXiv e-prints, pages arXiv–2012, 2020

    work page 2012

  8. [8]

    A simple neural network module for relational reasoning

    Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017

    2017

Show all 53 references
  1. [9]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. Accessed: 2025-03-20

    2024

  2. [10]

    Openai o3 mini system card, 2024

    OpenAI. Openai o3 mini system card, 2024. Accessed: 2025-03-20

    2024

  3. [11]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Advancing reasoning in large language models: Promising methods and approaches

    Avinash Patil. Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2502.03671, 2025

    2025 arXiv

  6. [14]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [15]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  8. [16]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  9. [17]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  10. [18]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  11. [19]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  12. [20]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    2021

  13. [21]

    Chemlit-qa: A human evaluated dataset for chemistry rag tasks

    Geemi Wellawatte, Huixuan Guo, Magdalena Lederbauer, Anna Borisova, Matthew Hart, Marta Brucka, and Philippe Schwaller. Chemlit-qa: A human evaluated dataset for chemistry rag tasks. In AI for Accelerated Materials Design-NeurIPS 2024

    2024

  14. [22]

    Olympicarena medal ranks: Who is the most intelligent ai so far? arXiv preprint arXiv:2406.16772, 2024

    Zhen Huang, Zengzhi Wang, Shijie Xia, and Pengfei Liu. Olympicarena medal ranks: Who is the most intelligent ai so far? arXiv preprint arXiv:2406.16772, 2024. 11

    2024 arXiv

  15. [23]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  16. [24]

    Large language models for reticular chemistry

    Zhiling Zheng, Nakul Rampal, Theo Jaffrelot Inizan, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. Large language models for reticular chemistry. Nature Reviews Materials, pages 1–13, 2025

    2025

  17. [25]

    Multi-hop question answering.Foundations and Trends® in Information Retrieval, 17(5):457–586, 2024

    Vaibhav Mavi, Anubhav Jangra, Adam Jatowt, et al. Multi-hop question answering.Foundations and Trends® in Information Retrieval, 17(5):457–586, 2024

    2024

  18. [26]

    Constructing datasets for multi-hop reading comprehension across documents

    Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302, 2018

    2018

  19. [27]

    Musique: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  20. [28]

    Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries

    Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv preprint arXiv:2401.15391, 2024

    2024 arXiv

  21. [29]

    A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

    Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, et al. A framework for evaluating the chemical knowledge and reasoning abilities of large langu...

    2025

  22. [30]

    Knowledge graph generation from text

    Igor Melnyk, Pierre Dognin, and Payel Das. Knowledge graph generation from text. arXiv preprint arXiv:2211.10511, 2022

    2022 arXiv

  23. [31]

    Extract, define, canonicalize: An llm-based framework for knowledge graph construction

    Bowen Zhang and Harold Soh. Extract, define, canonicalize: An llm-based framework for knowledge graph construction. arXiv preprint arXiv:2404.03868, 2024

    2024 arXiv

  24. [32]

    Building dynamic knowledge graphs from text using machine reading comprehension

    Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. Building dynamic knowledge graphs from text using machine reading comprehension. arXiv preprint arXiv:1810.05682, 2018

    2018 arXiv

  25. [33]

    Cear: Automatic construction of a knowledge graph of chemical entities and roles from scientific literature

    Stefan Langer, Fabian Neuhaus, and Andreas Nürnberger. Cear: Automatic construction of a knowledge graph of chemical entities and roles from scientific literature. arXiv preprint arXiv:2407.21708, 2024

    2024 arXiv

  26. [34]

    Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training

    Wenxiong Liao, Zhengliang Liu, Yiyang Zhang, Xiaoke Huang, Fei Qi, Siqi Ding, Hui Ren, Zihao Wu, Haixing Dai, Sheng Li, et al. Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training. In 2023 IEEE International Conference on Bioinforma...

    2023

  27. [35]

    Nilinker: attention-based approach to nil entity linking

    Pedro Ruas and Francisco M Couto. Nilinker: attention-based approach to nil entity linking. Journal of Biomedical Informatics, 132:104137, 2022

    2022

  28. [36]

    Domain-specific language model pretraining for biomedical natural language processing, 2020

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing, 2020

    2020

  29. [37]

    Pubchem in 2021: new data content and improved web interfaces

    Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem in 2021: new data content and improved web interfaces. Nucleic acids research, 49(D1):D1388–D1395, 2021

    2021

  30. [38]

    Generate-then-ground in retrieval-augmented generation for multi-hop question answering

    Zhengliang Shi, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. Generate-then-ground in retrieval-augmented generation for multi-hop question answering. arXiv preprint arXiv:2406.14891, 2024

    2024 arXiv

  31. [39]

    Hierarchical retrieval-augmented generation model with rethink for multi-hop question answering

    Xiaoming Zhang, Ming Wang, Xiaocui Yang, Daling Wang, Shi Feng, and Yifei Zhang. Hierarchical retrieval-augmented generation model with rethink for multi-hop question answering. arXiv preprint arXiv:2408.11875, 2024. 12

    2024 arXiv

  32. [40]

    MeOH" to

    Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. Hoprag: Multi-hop reasoning for logic-aware retrieval-augmented generation. arXiv preprint arXiv:2502.12442, 2025. 13 7 Appendix S7.1 Detailed Knowledge Graph Generation In this section, we ex...

    2025 arXiv

  33. [41]

    If an entity appears in the text but has no meaningful chemical relationship with another entity in the set, ignore it

    Entity Matching: Consider only the entities provided in the given set. If an entity appears in the text but has no meaningful chemical relationship with another entity in the set, ignore it

  34. [42]

    reacts with,

    Chemically Significant Relations Only: Extract relations that describe actual chemical interactions, transformations, or properties (e.g., "reacts with," "catalyzes," "dissolves in," "produces")

  35. [43]

    Avoid observations, opinions, and findings

    Factual Relations: Only extract factual relations. Avoid observations, opinions, and findings

  36. [44]

    Tuple Format: Output extracted facts in the form of (entity1, relation, entity2)

  37. [45]

    is," "are,

    Avoid Generic Relations: Exclude weak relations like "is," "are," "exists," "relates to." Focus on specific interactions. Valid Relation Types (Examples): ✓ "reacts with" ✓ "catalyzes" ✓ "binds to" ✓ "dissolves in" ✓ "oxidizes" ✓ "inhibits" ✓ "precipitates with" ✓ "acts as a s...

  38. [46]

    formic acid can be produced from carbon dioxide

  39. [47]

    formic acid is the simplest carboxylic acid with antibacterial and preservative properties

  40. [48]

    formic acid can act as a non-gaseous CO surrogate

  41. [49]

    Path (multi-hop chain of reasoning): carbon dioxide → formic acid → carbonylation reactions * Source 1 and source 2 are coming from different documents

    carbonylation reactions safely produce carboxylic acids under mild conditions using formic acid as a CO surrogate. Path (multi-hop chain of reasoning): carbon dioxide → formic acid → carbonylation reactions * Source 1 and source 2 are coming from different documents. Figure S1...

  42. [50]

    Solution -phase chemistry is the standard route to functionalise graphene

  43. [51]

    Graphene can form membranes that provide radiolytic shielding

  44. [52]

    Membranes used for DAC selectively excludenitrogen

  45. [53]

    Path (multi-hop chain of reasoning): solution → graphene → membranes→ nitrogen→ Cr3(Cr4Cl)3(BTT)82 *Sources 1–4 are extracted from four different documents

    Cr3(Cr4Cl)3(BTT)82 MOF shows very high O2 selectivity over nitrogen. Path (multi-hop chain of reasoning): solution → graphene → membranes→ nitrogen→ Cr3(Cr4Cl)3(BTT)82 *Sources 1–4 are extracted from four different documents. Figure S11: A 4-hop multi-document question–answer ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.