Pith. sign in

REVIEW 3 major objections 5 minor 43 references

SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SciToolAgent, an LLM agent that plans against a hand-built knowledge graph of scientific tools, reports 94% accuracy on a 531-question multi-tool benchmark, beating strong baselines by about 10 percentage points.

desk verdict A solid engineering paper with real contributions in tool knowledge graphs and safety checking, but the 94% accuracy claim rests on a benchmark generated and graded by the same GPT-4o family that powers the agent. read the letter →

arxiv 2507.20280 v1 pith:O7DNMRHZ submitted 2025-07-27 cs.AI cs.CL

Keyan Ding , Jing Yu , Junjie Huang , Yuchen Yang , Qiang Zhang , Huajun Chen This is my paper
classification cs.AIcs.CL
keywords scientifictoolagentknowledgegraphretrieval-augmentedgenerationmulti-toolorchestrationLLMworkflowssafetycheckingSciEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an LLM-driven agent can take over multi-step scientific tool use, not by memorizing tool prompts but by planning against an explicit graph of tool relationships. It introduces SciToolAgent, whose Planner retrieves relevant tools from a hand-built scientific tool knowledge graph (SciToolKG), assembles them into an ordered chain-of-tools, and executes them with error retries and a safety filter. On a curated 531-question benchmark spanning biology, chemistry, and materials science, the agent reports 94% final-answer accuracy, about 10 percentage points above state-of-the-art agent baselines. The claim matters because it points to a way for researchers without tool-specific expertise to run complex computational workflows that currently require specialist knowledge.

What carries the argument

SciToolKG is a directed graph $G=(V,E)$ whose nodes are tools and their attributes (function, input/output formats, category, source, safety level) and whose edges encode dependencies and compatibility. The Planner's retrieval-augmented generation works in four steps: full-graph top-$k$ retrieval by semantic similarity $S(q,T_i)$; sub-graph exploration of $d$-hop neighborhoods to score complementary tools $T_j$ by $S(q,T_i \oplus T_j)$; combination and ranking by the product $S(q,T_i) \times S(q,T_i \oplus T_j)$; and LLM generation of an ordered chain $T_1 \rightarrow T_2 \rightarrow \cdots \rightarrow T_m$. The Executor then runs the chain, and the safety module flags any molecule or protein whose maximal similarity to a hazardous database entry exceeds a threshold $\delta = 0.95$, using the average of Tanimoto, Dice, and cosine coefficients for molecules and the Smith-Waterman alignment score for proteins. This machinery replaces naive in-context tool selection with a global, order-aware plan, which is the mechanism credited for the accuracy gains.

What would settle it

Construct a held-out set of multi-tool scientific questions written and verified by domain experts who did not use GPT-4o or ReAct, run SciToolAgent, ReAct, and Reflexion on it, and grade final answers by human exact-match; if the 10-point margin disappears, the central claim of graph-driven advantage is falsified.

Watch

Extended reading notes

Core claim

The central claim is that explicit, structured knowledge about tool dependencies is what an LLM agent needs to automate scientific workflows at scale. SciToolAgent builds that structure by hand into SciToolKG, then uses retrieval-augmented generation so the Planner can find the right tools, order them correctly, and form a chain-of-tools for each query. The Executor runs the chain and retries on errors, the safety module blocks outputs that resemble known hazardous molecules or toxic proteins, and the Summarizer synthesizes results and triggers plan refinement when needed. On SciToolEval, this design reaches 94% overall answer accuracy, with the largest advantage on multi-tool Level-2 tasks, and it successfully completes four case studies in protein design, chemical reactivity prediction, chemical synthesis, and MOF screening.

Load-bearing premise

The evaluation pipeline — questions and reference plans generated by GPT-4o running ReAct, then answers scored by GPT-4o similarity — may be biased toward the same model family and agent pattern that SciToolAgent uses, so the measured gains might not persist on independently created scientific tasks.

Editorial extensions

If this is right

  • Multi-tool scientific workflows can be automated end-to-end: given a natural-language goal, the agent selects, sequences, runs, and summarizes tools across biology, chemistry, and materials science without human intervention.
  • The largest gains appear on multi-tool problems, where the graph-based planner beats ReAct by about 20 points and Reflexion by about 10 points in final answer accuracy.
  • The safety module can block hazardous outputs before they reach the user, as in the 4-chlorophenol case where ReAct and Reflexion did not flag toxicity.
  • Open-source LLMs can be made competitive by fine-tuning on SciToolKG-generated instructions, with Qwen2.5-7B-FT showing a 10-point improvement that brings it closer to Qwen2.5-72B.
  • GPT-4o is reported as the cost-effective default foundation model, giving near-best accuracy at lower API cost than o1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If SciToolEval's GPT-4o-generated questions and GPT-4o similarity scoring favor the same model family that powers SciToolAgent, the reported 94% and the 10-point margin are upper bounds; an independently authored benchmark with human-exact scoring would test that.
  • The retrieval formula's product of similarity scores is a hand-set design; replacing it with a learned ranker or graph neural network over SciToolKG could reveal whether sub-graph exploration, rather than the ranking rule, drives the gains.
  • The safety check's 0.95 fingerprint-similarity threshold could miss novel hazardous compounds that are dissimilar to entries in the safeguard databases; testing with recently discovered toxic substances would probe its robustness.
  • Because SciToolKG is manually curated, scaling to new tools and domains will likely require automated knowledge extraction from tool documentation and literature, a step the paper describes as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SciToolAgent is an LLM-driven agent for scientific tool orchestration. It contributes a manually curated knowledge graph (SciToolKG) of over 500 tools, a graph-based retrieval planner that outputs a chain of tools, an executor with error handling and a similarity-based safety checker, and a summarizer with iterative replanning. On the authors' new SciToolEval benchmark (531 questions), the paper reports roughly 94% overall accuracy and 10-20% gains over ReAct, Reflexion, ChemCrow, and CACTUS, plus four case studies in protein design, reactivity prediction, synthesis, and MOF screening.

Significance. If the results were established by an independent evaluation, the contribution would be significant: it is one of the few agent frameworks that integrate hundreds of scientific tools, and the safety module addresses a real gap. The manuscript's concrete strengths are the released code, the manually constructed KG, and the worked case studies that show end-to-end multi-tool workflows. However, the quantitative claim is not yet independently grounded: the benchmark questions, reference plans, and reference answers are generated by GPT-4o with ReAct, and the same model family is used as the automatic judge.

major comments (3)
  1. [§2.2 and §4.3] The headline accuracy numbers are measured with a partially circular pipeline. Section 4.3 states that SciToolEval questions are generated by GPT-4o, reference plans and answers are produced by GPT-4o using ReAct (step 2), and only a manual review (step 3) filters them. Section 2.2 states that the two main metrics are computed by prompting GPT-4o to perform a similarity evaluation against these references. Because SciToolAgent's Planner, Executor, and Summarizer default to GPT-4o, the evaluation rewards outputs that resemble GPT-4o-generated references; ReAct/Reflexion outputs may be stylistically further from those references, inflating both the absolute 94% accuracy and the reported 10% margin. The manual review can catch factual errors but does not remove the grading bias. Please add a human-scored held-out set (or an externally authored benchmark with independently verified answers) and report the same comparisons on it.
  2. [§4.4.2, Eqs. (5)–(6)] The safety module is a core claimed contribution, but the paper provides no quantitative validation of it. The safeguard database and the threshold δ=0.95 are introduced without an evaluation of false positives or false negatives on known hazardous versus safe molecules/proteins. The only demonstrated successful alert is the 4-chlorophenol case in §2.3.3. To support the claim of 'responsible and ethical tool usage,' please report a small validation set (e.g., hazardous compounds from PubChem plus benign controls) with hit rates and false-alarm rates, and state how δ was chosen.
  3. [§4.4.1, Eqs. (1)–(3)] The paper attributes the gains to SciToolKG-based planning, but the experiments do not isolate that mechanism. The default retrieval hyperparameters k=5, d=3, n=10 are fixed in §4.4.1 without a sensitivity analysis, and no ablation removes the subgraph exploration or replaces graph-based retrieval with a flat retrieval baseline. Consequently, the comparison in Fig. 2 conflates the KG planner with the overall agent design. Please add ablations (no KG, flat top-k, other k/d/n values) to show that the graph-based subgraph exploration is responsible for the reported planning and answer accuracy.
minor comments (5)
  1. [Data Availability] Data Availability lists only the toxic-compound and toxic-protein sources; SciToolEval itself is not linked. Releasing the benchmark with reference plans and tool annotations would allow reproduction of the headline 94% figure.
  2. [Fig. 2] The caption of Fig. 2 lists the methods and models, but the small unlabeled bars make it hard to map colors to names; please add a clear legend or explicit panel annotations.
  3. [§4.3, step 2] In §4.3 step 2, ReAct receives only the tools relevant to each question, whereas SciToolAgent retrieves over the full graph; state this asymmetry explicitly because it affects the interpretation of the baseline comparison.
  4. [§4.4.1(3)] The phrase 'a total of k2 tools' should read 'k^2 tools'; likewise, define the constraint n ≤ k^2 consistently with the notation in Eqs. (1)–(3).
  5. [§2.2, Pass Rate] The exact definition of Pass Rate is missing a stopping criterion: specify how a query is judged 'successfully completed' when the Summarizer requests replanning, and whether multiple attempts are counted as a pass.

Circularity Check

1 steps flagged · score 4.0 of 10

The 94% accuracy claim rests on a partially self-referential evaluation: SciToolEval references are generated by GPT-4o/ReAct and scored by GPT-4o similarity, while SciToolAgent's default model is also GPT-4o. The core KG and safety methods are not circular, but the headline result is not fully independently secured.

  1. other [Section 2.2 (Evaluation metrics) and Section 4.3 (Tool execution and answer generation)]
    "For both tool planning accuracy and final answer accuracy, we prompt GPT-4o to perform a similarity evaluation between the generated response and the ground truth. ... Once the questions are generated, we proceed to tool execution and answer generation by utilizing the ReACT framework in conjunction with GPT-4o."

    The ground-truth plans and answers in SciToolEval are produced by GPT-4o executing ReAct, and the same GPT-4o is used as the similarity judge for both Tool Planning Accuracy and Final Answer Accuracy. SciToolAgent's default Planner, Executor, and Summarizer are also GPT-4o. The reported 94% accuracy and the 10% margin over baselines therefore partly measure how well GPT-4o outputs match GPT-4o-generated references, an evaluation self-consistency effect rather than a fully independent measure of scientific tool-use competence. Manual expert review checks for gross errors, but it does not remove the bias introduced by generating references and grading with the same model family that powers the proposed agent.

full rationale

I found no definitional circularity in the agent's derivation itself: SciToolKG is manually constructed from tool documentation, retrieval and chain-of-tools generation are explicit algorithmic steps, and the safety module uses external databases and explicit similarity equations. The central concern is confined to the evaluation loop. SciToolEval's ground truth is generated by GPT-4o/ReAct, and the accuracy metrics use GPT-4o as judge, while SciToolAgent defaults to GPT-4o. Because the standard answers are also grounded in actual tool executions and were manually reviewed, this is not a full equivalence-by-construction, but the headline quantitative claim is not externally secured against same-model self-consistency bias. The Qwen2.5-7B-FT result additionally relies on instructions generated from the same SciToolKG/GPT-4o pipeline as SciToolEval, which further limits generalization claims without being a formal fitted-prediction circularity. Overall, this is partial evaluation circularity rather than circularity of the method's derivation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central quantitative result depends on hand-set retrieval hyperparameters and on a benchmark that is generated and graded by the same model family used inside the agent. The safety module adds a threshold without independent calibration evidence.

free parameters (4)
  • retrieval top-k k = 5
    Section 4.4.1 sets k = 5 for full-graph retrieval; no sensitivity analysis is reported.
  • subgraph depth d = 3
    Section 4.4.1 sets d = 3 for sub-graph exploration; the effect on planning accuracy is not studied.
  • combined tool selection top-n n = 10
    Section 4.4.1 sets n = 10 for final tool combination; no ablation is presented.
  • safety similarity threshold delta = 0.95
    Section 4.4.2 flags outputs above delta = 0.95 as dangerous; the threshold is chosen without reported calibration or false-positive analysis.
assumptions (4)
  • domain assumption Semantic similarity between query text and tool metadata is a reliable signal of tool relevance.
    Used in full-graph retrieval and sub-graph scoring in Section 4.4.1, Equations (1)-(3).
  • ad hoc to paper SciToolEval ground truth answers and reference plans are correct and unbiased.
    The benchmark is generated by GPT-4o + ReAct and manually reviewed; the central accuracy comparison treats these as ground truth.
  • ad hoc to paper The safeguard database and the threshold delta correctly separate hazardous from safe outputs.
    Safety checking in Section 4.4.2 relies on this without reporting false positive/negative rates.
  • domain assumption Baselines were adapted to the full toolset without hurting their performance.
    Section 2.2 states baselines were given the full toolset for fairness, but no details are given on how the adaptation affected each baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration." pith.science (2026). https://pith.science/paper/O7DNMRHZ

@misc{pith2026250720280,
  author       = {Pith},
  title        = {Pith review of: SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O7DNMRHZ}},
  note         = {Machine review of arXiv:2507.20280}
}
read the original abstract

Scientific research increasingly relies on specialized computational tools, yet effectively utilizing these tools demands substantial domain expertise. While Large Language Models (LLMs) show promise in tool automation, they struggle to seamlessly integrate and orchestrate multiple tools for complex scientific workflows. Here, we present SciToolAgent, an LLM-powered agent that automates hundreds of scientific tools across biology, chemistry, and materials science. At its core, SciToolAgent leverages a scientific tool knowledge graph that enables intelligent tool selection and execution through graph-based retrieval-augmented generation. The agent also incorporates a comprehensive safety-checking module to ensure responsible and ethical tool usage. Extensive evaluations on a curated benchmark demonstrate that SciToolAgent significantly outperforms existing approaches. Case studies in protein engineering, chemical reactivity prediction, chemical synthesis, and metal-organic framework screening further demonstrate SciToolAgent's capability to automate complex scientific workflows, making advanced research tools accessible to both experts and non-experts.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 32 canonical work pages

  1. [1]

    & Wachter, S

    Birhane, A., Kasirzadeh, A., Leslie, D. & Wachter, S. Science in the age of large language models. Nature Reviews Physics 5, 277–280 (2023)

    2023

  2. [2]

    Schick, T. et al. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  3. [3]

    Yang, R. et al. GPT4Tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  4. [4]

    Guo, T. et al. What can large language models do in chemistry? A comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems 36, 59662–59688 (2023)

    work page 2023

  5. [5]

    Zhao, W. X. et al. A survey of large language models. arXiv:2303.18223 (2023)

    arXiv 2023

  6. [6]

    Min, B. et al. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys 56, 1–40 (2023)

    work page 2023

  7. [7]

    Wang, L. et al. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 186345 (2024)

    work page 2024

  8. [8]

    C., Collison, C

    Ramos, M. C., Collison, C. J. & White, A. D. A review of large language models and autonomous agents in chemistry. arXiv:2407.01603 (2024)

    arXiv 2024

Show all 43 references
  1. [9]

    A., MacKnight, R., Kline, B

    Boiko, D. A., MacKnight, R., Kline, B. & Gomes, G. Autonomous chemical research with large language models. Nature 624, 570–578 (2023)

    2023

  2. [10]

    & Born, J

    Janakarajan, N., Erdmann, T., Swaminathan, S., Laino, T. & Born, J. Language models in molecular discovery. arXiv:2309.16235 (2023)

    2023 arXiv

  3. [11]

    Nature Machine Intelligence 6, 525–535 (2024)

    Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari & Andrew D White Augmenting large language models with chemistry tools. Nature Machine Intelligence 6, 525–535 (2024)

    2024

  4. [12]

    McNaughton, A. D. et al. CACTUS: Chemistry agent connecting tool usage to science. ACS Omega 9, 46563–46573 (2024)

    2024

  5. [13]

    Jin, Q., Yang, Y., Chen, Q. & Lu, Z. GeneGPT: Augmenting large lan- guage models with domain tools for improved access to biomedical information. Bioinformatics 40, btae075 (2024)

    2024

  6. [14]

    Huang, K. et al. CRISPR-GPT: An LLM agent for automated design of gene- editing experiments. arXiv:2404.18021 (2024). 19

    2024 arXiv

  7. [15]

    & Wang, H

    Liu, H. & Wang, H. GenoTEX: A benchmark for evaluating LLM-based exploration of gene expression data in alignment with bioinformaticians. arXiv:2406.15341 (2024)

    2024 arXiv

  8. [16]

    & Buehler, M

    Ghafarollahi, A. & Buehler, M. J. ProtAgents: Protein discovery via large lan- guage model multi-agent collaborations combining physics and machine learning. arXiv:2402.04268 (2024)

    2024 arXiv

  9. [17]

    & Fung, V

    Jia, S., Zhang, C. & Fung, V. LLMatDesign: Autonomous materials discovery with large language models. arXiv:2406.13163 (2024)

    2024 arXiv

  10. [18]

    & Kim, J

    Kang, Y. & Kim, J. ChatMOF: an artificial intelligence system for predicting and generating metal-organic frameworks using large language models. Nature Communications 15, 4705 (2024)

    2024

  11. [19]

    Wu, H. et al. ChatEDA: A large language model powered autonomous agent for EDA. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 43, 3184–3197 (2024)

    2024

  12. [20]

    & Buehler, M

    Ni, B. & Buehler, M. J. MechAgents: Large language model multi-agent collabora- tions can solve mechanics problems, generate new data, and integrate knowledge. Extreme Mechanics Letters 67, 102131 (2024)

    2024

  13. [21]

    Yao, S. et al. ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations (2023)

    2023

  14. [22]

    He, J. et al. Control risk for potential misuse of artificial intelligence in science. arXiv:2312.06632 (2023)

    2023 arXiv

  15. [23]

    Liu, X. et al. ToolNet: Connecting large language models with massive tools via tool graph. arXiv:2403.00839 (2024)

    2024 arXiv

  16. [24]

    Hao, S., Liu, T., Wang, Z. & Hu, Z. ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems 36 (2024)

    2024

  17. [25]

    & Yao, S

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. & Yao, S. Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  18. [26]

    Ingraham, J. B. et al. Illuminating protein space with a programmable generative model. Nature 623, 1070–1078 (2023)

    2023

  19. [27]

    Lin, Z. et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379, 1123–1130 (2023)

    2023

  20. [28]

    Atilgan, A. R. et al. Anisotropy of fluctuation dynamics of proteins with an elastic network model. Biophysical journal 80, 505–515 (2001)

    2001

  21. [29]

    Bakan, A., Meireles, L. M. & Bahar, I. Prody: protein dynamics inferred from theory and experiments. Bioinformatics 27, 1575–1577 (2011)

    2011

  22. [30]

    Cock, P. J. et al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics 25, 1422 (2009)

    2009

  23. [31]

    Schwaller, P. et al. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. ACS central science 5, 1572–1583 (2019)

    2019

  24. [32]

    Pei, Q. et al. BioT5+: Towards generalized biological understanding with IUPAC integration and multi-task tuning. Findings of the Association for Computational Linguistics 1216–1240 (2024)

    2024

  25. [33]

    Papadatos, G. et al. SureChEMBL: a large-scale, chemically annotated patent 20 document database. Nucleic Acids Research 44, D1220–D1228 (2016)

    2016

  26. [34]

    Kim, S. et al. Pubchem substance and compound databases. Nucleic acids research 44, D1202–D1213 (2016)

    2016

  27. [35]

    Bobbitt, N. S. et al. MOFX-DB: An online database of computational adsorption data for nanoporous materials. Journal of Chemical & Engineering Data 68, 483–498 (2023)

    2023

  28. [36]

    Nandy, A. et al. Mofsimplify, machine learning models with extracted stability data of three thousand metal–organic frameworks. Scientific Data 9, 74 (2022)

    2022

  29. [37]

    Dubbeldam, D., Calero, S., Ellis, D. E. & Snurr, R. Q. RASPA: molecular sim- ulation software for adsorption and diffusion in flexible nanoporous materials. Molecular Simulation 42, 81–101 (2016)

    2016

  30. [38]

    URL https://blast.ncbi.nlm.nih.gov

    BLAST: Basic local alignment search tool. URL https://blast.ncbi.nlm.nih.gov. Accessed 2 Oct. 2024

    2024

  31. [39]

    URL http://www.rdkit.org

    RDKit: Open-source cheminformatics software. URL http://www.rdkit.org. Accessed 6 Oct. 2024

    2024

  32. [40]

    & H´ eberger, K

    Bajusz, D., R´ acz, A. & H´ eberger, K. Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics 7, 1–13 (2015)

    2015

  33. [41]

    F., Waterman, M

    Smith, T. F., Waterman, M. S. et al. Identification of common molecular subsequences. Journal of molecular biology 147, 195–197 (1981)

    1981

  34. [42]

    Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (2022)

    2022

  35. [43]

    Vaswani, A. et al. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017). 21

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.