Pith. sign in

REVIEW 3 major objections 5 minor 10 references

Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models should be deeply integrated into every step of the scientific process, from hypothesis to discovery, to become creative engines rather than mere assistants.

desk verdict A balanced, broad perspective on LLMs in science whose normative thesis is honest but unproven; worth publishing as a perspective after revision. read the letter →

arxiv 2505.16477 v1 pith:Q3URT7LL submitted 2025-05-22 cs.AI

Yanbo Zhang , Sumeer A. Khan , Adnan Mahmud , Huck Yang , Alexander Lavin , Michael Levin , Jeremy Frey , Jared Dunnmon
show 5 more authors
James Evans Alan Bundy Saso Dzeroski Jesper Tegner Hector Zenil
This is my paper
classification cs.AI
keywords largelanguagemodelsscientificdiscoveryhypothesisgenerationLLMagentsmethodhallucinationformalverificationfoundation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large language models are currently useful scientific assistants—they annotate data, plan and run experiments, and propose candidate hypotheses—but have not yet become the "creative engines" that discover new scientific principles. Its central claim is that to get there, LLMs should be deeply integrated into every step of the scientific process, in collaboration with human scientists and measured by clear evaluation metrics. The paper treats scientific discovery as a reward-search loop of observation, hypothesis, and experiment, and reviews where LLM agents can enter that loop and what still blocks them: hallucinations, weak reasoning, planning failures, and opaque self-explanations. A sympathetic reader would take away that the path from AI-assisted science to AI-driven discovery depends on building trustworthy, verifiable LLM-agent systems rather than on scaling raw text generation.

What carries the argument

The load-bearing mechanism is the hypothesis-experiment-observation loop, treated as a reward-search in knowledge space, with LLM agents inserted at every stage. Around that loop the paper assembles a toolbox of methods—retrieval-augmented generation for grounding, chain-of-thought and agent loops for planning and correction, sampling-and-voting self-consistency for reliability, translation into machine-checked proof languages or executable code for validation, and human oversight for safety—and argues that these components, combined into agent systems, carry LLMs from text assistants to discovery engines.

What would settle it

A controlled study where LLM-agent systems design and run real experiments in several fields, with formal verification and human checks, would settle the claim: if the verified hypotheses fail no better than randomly generated ones, or if the verification pipeline accepts plausible-but-wrong hypotheses systematically, the deep-integration thesis is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the scientific method itself—not just individual lab tasks—can and should be augmented end-to-end by LLMs. The authors frame discovery as a reward-search process with a hypothesis-experiment-observation loop and place LLM agents at its three hinges: expanding observation through annotation and qualitative description of black-box systems, proposing hypotheses by recombining literature and even treating hallucinations as a stream of candidate conjectures, and automating experimentation through tool use, code execution, and planning. They conclude that deep integration into all steps, aligned with human goals and evaluated by clear metrics, is what would turn LLMs into creative engines rather than copilots. The paper is equally explicit about current limits: it catalogs the reversal curse, order-sensitive reasoning, planning failures, and unreliable self-correction, and it proposes replacing binary trust with a continuous "algorithmic confidence" for LLM-agent systems, with all LLM-assisted research either human-verified or confidence-tested.

Load-bearing premise

The paper's recommendation assumes that current fixes—retrieval grounding, repeated sampling, formal proof checking, and human oversight—will scale to open-ended scientific tasks, and that translating hypotheses into formal languages will not introduce errors that erode trust.

Editorial extensions

If this is right

  • LLM agents can take over repetitive experimental loops, scaling studies that would otherwise require years of human effort—exemplified in the paper by autonomous chemical reactions and large-scale neuron interpretation.
  • The same model can play two roles: a strict, verifiable executor in experiment and validation phases, and a deliberately speculative "hallucinator" at the hypothesis stage, with the two roles coupled by filtering and testing.
  • Trust in LLM outputs would be reported as a continuous algorithmic confidence value rather than a pass/fail, and all LLM-assisted research would carry either human verification or such a confidence score.
  • Scientific communication and community learning would change: negative results and failed experiments would be published and mined systematically, since LLMs make it cheap to learn from them.
  • Domain-specific foundation models trained on scientific data as a new modality would provide reusable scaffolds that can be inserted into many downstream tasks, accelerating prediction and design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A directly testable extension: hybrid human-LLM teams should beat either alone on open-ended discovery benchmarks if the paper's thesis is right, but only when confidence is measured stage-by-stage; if they do not, the deep-integration recommendation loses its force.
  • The paper's hint that an "answer" should generate the next "question" points toward self-play-style scientific agents, where each verified hypothesis yields a new discrepancy to explain—an architecture closer to game self-play than to current reasoning benchmarks.
  • If "algorithmic confidence" becomes a rigorous, standardized measurement, it could be reported alongside every LLM-assisted result—analogous to a p-value—turning trust from a human judgment into a scientific quantity.
  • The hallucination-as-creativity claim could be settled by comparing the yield of valid, novel experiments from unfiltered versus filtered LLM hypotheses in real laboratories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This perspective paper reviews the current and potential roles of large language models (LLMs) across the scientific process, from literature review and observation to hypothesis generation, experiment design and execution, and automation. It surveys LLM agents, retrieval-augmented generation, foundation models for science, and human-in-the-loop approaches, and it discusses challenges such as hallucinations, reasoning limitations, interpretability, and community-level effects. The paper's central conclusion is a normative recommendation: deep integration of LLMs into all steps of the scientific process should be pursued, but in collaboration and alignment with human goals and with clear evaluation metrics. It also advocates for publishing negative results and for an 'algorithmic confidence' framework to quantify trust in LLM-assisted research.

Significance. As a synthesis, the paper is useful and generally accurate: it covers a broad literature, distinguishes copilots from agents and foundation models, and is explicit about known weaknesses (reversal curse, planning failures, self-correction failures, and autoformalization errors). Its taxonomy (observation, hypothesis, experiment, automation) and summary tables provide a compact map of the field. The paper's principal weakness is that its central recommendation is a normative thesis rather than an empirically supported claim; no experiments or quantitative evidence are offered to show that the proposed deep integration would be net beneficial or that the cited mitigations scale to open-ended discovery. Still, for a perspective, the paper is honest about the gap, and it proposes falsifiable directions (e.g., algorithmic confidence testing, benchmarks for autoformalization) that could be pursued. The paper therefore has programmatic value even though it does not itself deliver a proof of concept.

major comments (3)
  1. [Experimentation and Automation (Validation paragraph)] The paper's strongest cited safeguard for separating valid hypotheses from hallucinations is formal verification via LEAN or Python (refs 131-133). However, the same paragraph immediately concedes 'the reliance on LLMs to translate hypotheses into formal languages, which may introduce errors in the process.' This is load-bearing: a verifier certifies the formalized statement, not the originally intended hypothesis, so an autoformalization error can produce a formally 'verified' but scientifically wrong claim. The paper recommends deep integration into all steps of the scientific process (Abstract) but does not quantify end-to-end error rates or propose a concrete method to bound them. I recommend the authors either temper the conclusion to an explicitly conditional recommendation or add a research agenda item for measuring autoformalization fidelity on open-ended scientific hypotheses.
  2. [Conclusions (algorithmic confidence proposal)] The paper recommends that 'all LLM-assisted research should either be verified by humans or undergo algorithmic confidence testing,' but 'algorithmic confidence' is never operationally defined. No input features, uncertainty quantification method, or validation protocol are given, and no existing framework is cited as a concrete instantiation. Since the abstract's 'clear evaluation metrics' is part of the central recommendation, this concept needs at least a formal definition or a reference to a concrete, testable approach; otherwise the core recommendation is vacuous at the point where it is most needed.
  3. [Introduction and Conclusions] The title promises 'From Hypothesis to Discovery,' but the evidence reviewed for end-to-end discovery is thin: most cited successes are hypothesis suggestions or narrow experimental automations (drug combinations, CRISPR-GPT, nanobody design), while the only validated discoveries come from domain-specific foundation models like Evo, not from an LLM completing the full hypothesis-experiment-observation loop. The authors openly state the premise that AI has been limited in fundamental science (Introduction), but the forward-looking conclusion ('may come to represent a synthesis of human and artificial intelligence') goes beyond the presented evidence. For a perspective this is acceptable, but the manuscript should explicitly separate current evidence from speculation, e.g., by marking which claimed capabilities are demonstrated and which are aspirational.
minor comments (5)
  1. [Conclusions] The phrase 'failed research failed research' is duplicated and should be corrected.
  2. [References] Reference 2 ('The Atlanctic') contains a typo; it should be 'The Atlantic.'
  3. [Experimentation and Automation] The paper refers to 'Reflexion' in one place and 'Reflection' in another; the spelling should be made consistent.
  4. [Figure 2 caption] The caption contains an extra parenthesis in 'human-machine discovered))'; this should be fixed.
  5. [Table 1] The row labeled 'Human-in-the-loop' appears as a bare entry under 'Automation'; it would be clearer if the table included a short description of how human oversight is applied in each stage.

Circularity Check

0 steps flagged · score 1.0 of 10

Perspective/review paper whose conclusion is a normative recommendation; self-citations are contextual and not load-bearing, so no circularity is established.

full rationale

This paper is a perspective and literature review, not a derivation. It does not fit parameters to data and then predict related quantities; it contains no equations that close a loop; and its central recommendation—deep integration of LLMs into scientific practice with human oversight and evaluation metrics—is a normative claim assembled from external evidence. The self-citations (refs 41-43 on literature search and peer-review bottlenecks, ref 98 on search as key to discovery, ref 212 on AI frontiers) are contextual and would not change the conclusion if removed. The formal-verification discussion in the Validation paragraph cites external work (LEAN, Prover9, ARC/Python experiments) and explicitly concedes the autoformalization limitation: 'A limitation, however, is the reliance on LLMs to translate hypotheses into formal languages, which may introduce errors in the process.' That concession identifies a reliability risk, not a circular step: the paper does not invoke its own conclusion as evidence for the safeguard. No uniqueness theorem is imported from the authors' prior work, and no known result is renamed as a new derivation. Accordingly the honest finding is no significant circularity (score 1).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or new physical entities. Its argument rests on domain assumptions about the nature of science and about LLM reliability that are not empirically established.

assumptions (4)
  • domain assumption Scientific discovery can be described as a reward-searching process where hypotheses are proposed and falsified by experiments.
    The paper's whole framing (Section 'Augmenting the Scientific Method') borrows Popper's falsificationism and treats discovery as search. This is a philosophical assumption, not a proven theorem, and it shapes the conclusion about what LLMs need to do.
  • domain assumption LLMs are auto-regressive conditional generative models whose outputs are sampled token distributions.
    The paper describes LLMs in this way in Section 'Prompting LLMs' and builds its analysis on this standard architecture, assuming it will remain the basis for scientific AI tools.
  • domain assumption Formal verification and human oversight can reduce LLM errors to acceptable levels for scientific use.
    The conclusion that deep integration 'should be pursued' depends on the untested premise that RAG, self-consistency, LEAN verification, and human-in-the-loop will be sufficient. The paper itself notes residual error risk in the Validation section.
  • ad hoc to paper Hallucinations can sometimes be a source of novel hypotheses rather than only errors.
    This is a speculative claim in Section 'Hallucinations as Putative Sources of Novel Hypotheses'. The authors explicitly say 'this remains unexplored territory', so it is an axiom of their argument rather than an established result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery." pith.science (2026). https://pith.science/paper/Q3URT7LL

@misc{pith2026250516477,
  author       = {Pith},
  title        = {Pith review of: Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3URT7LL}},
  note         = {Machine review of arXiv:2505.16477}
}
read the original abstract

With recent Nobel Prizes recognising AI contributions to science, Large Language Models (LLMs) are transforming scientific research by enhancing productivity and reshaping the scientific method. LLMs are now involved in experimental design, data analysis, and workflows, particularly in chemistry and biology. However, challenges such as hallucinations and reliability persist. In this contribution, we review how Large Language Models (LLMs) are redefining the scientific method and explore their potential applications across different stages of the scientific cycle, from hypothesis testing to discovery. We conclude that, for LLMs to serve as relevant and effective creative engines and productivity enhancers, their deep integration into all steps of the scientific process should be pursued in collaboration and alignment with human scientific goals, with clear evaluation metrics. The transition to AI-driven science raises ethical questions about creativity, oversight, and responsibility. With careful guidance, LLMs could evolve into creative engines, driving transformative breakthroughs across scientific disciplines responsibly and effectively. However, the scientific community must also decide how much it leaves to LLMs to drive science, even when associations with 'reasoning', mostly currently undeserved, are made in exchange for the potential to explore hypothesis and solution regions that might otherwise remain unexplored by human exploration alone.

Figures

Figures reproduced from arXiv: 2505.16477 by the authors.

Figure 1
Figure 1. (A) LLMs generate sentences in an auto-regressive manner, sampling tokens from a predicted distribution at each step. (B) A typical prompt for LLMs consists of a system prompt and a user prompt. The LLM will then respond as an assistant. A multi￾round dialogue will repeat the user and assistant contents. (C) LLM agents are systems that use a large language model as its core reasoning and decision-making engine, enab… view at source ↗
Figure 2
Figure 2. Illustration of the scientific discovery process: Scientific research can be [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 2 canonical work pages

  1. [10]

    & Zhang, Y

    Li, Z., Ji, J. & Zhang, Y. From Kepler to Newton: Explainable AI for Science Discovery. (2022). 11. Baker, N. et al. Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence. (2019) doi:10.2172/1478744. 12. Manta, C. D., Hu, E. & Bengio, Y. GFlowNets for Causal Discovery: an Overview. OpenRevie...

    doi:10.2172/1478744 2022

  2. [31]

    E., Ghattas, O

    Willcox, K. E., Ghattas, O. & Heimbach, P. The imperative of physics-based modeling and inverse theory in computational science. Nat Comput Sci 1, 166–168 (2021). 32. Webster, P. Six ways large language models are changing healthcare. Nat Med 29, 2969–2971 (2023). 33. Eriksen, A. V, Möller, S. & Ryg, J. Use of GPT-4 to Diagnose Complex Clinical Cases. NEJ...

    arXiv 2021

  3. [51]

    yoheinakajima/babyagi

    Nakajima, Y. yoheinakajima/babyagi. Preprint at https://github.com/yoheinakajima/babyagi (2024). 52. Chase, H. LangChain. https://github.com/langchain-ai/langchain (2022). 53. Liu, J. LlamaIndex. URL: ‘https://github.com/jerryjliu/llama_index’ (2022). 54. Khattab, O. et al. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. 12...

    arXiv 2024

  4. [71]

    Learning to Reason with LLMs

    OpenAI. Learning to Reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/ (2024). 72. Kaplan, J. et al. Scaling laws for neural language models. arXiv preprint arXiv:2001. 08361 (2020). 73. Wei, J. et al. Emergent abilities of large language models. arXiv preprint arXiv:2206. 07682 (2022). 74. Nguyen, E. et al. Sequence modeling and des...

    arXiv 2024

  5. [92]

    Fan, W. et al. A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 6491–6501 (2024) doi:10.1145/3637528.3671470. 93. Li, Y., Xu, M., Miao, X., Zhou, S. & Qian, T. Prompting Large Language Models for Counterfactual Generation: An Empir...

    arXiv 2024

  6. [109]

    Wang, G. et al. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291 (2023). 110. Darvish, K. et al. ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization. (2024). 111. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. & Yao, S. Reflexion: Language Agents with Verbal Reinf...

    arXiv 2023

  7. [130]

    Lavin, A. et al. Technology readiness levels for machine learning systems. Nat Commun 13, (2020). 131. Zhou, J. P. et al. Don’t Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalization. 12th International Conference on Learning Representations, ICLR 2024 (2024). 132. Olausson, T. X. et al. LINC: A Neurosymbolic Approach for Logical Rea...

    arXiv 2020

  8. [167]

    Berglund, L. et al. The Reversal Curse: LLMs trained on ‘A is B’ fail to learn ‘B is A’. arXiv preprint arXiv:2309. 12288 (2023). 168. Chen, X., Chi, R. A., Wang, X. & Zhou, D. Premise Order Matters in Reasoning with Large Language Models. Proc Mach Learn Res 235, 6596–6620 (2024). 169. Allen-Zhu, Z. & Labs Yuanzhi Li YuanzhiLi, F. Physics of Language Mod...

    work page 2023

Show all 10 references
  1. [185]

    Negative

    Wiegreffe, S. & Pinter, Y. Attention is not not Explanation. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference 11–20 (2019) doi:10.18653/V...

    2019 doi

  2. [205]

    Awkward wording. Rephrase

    Hyland, K. Academic publishing and the myth of linguistic injustice. J Second Lang Writ 31, 58–69 (2016). 206. Clavero, M. ‘“Awkward wording. Rephrase”’: linguistic injustice in ecological journals. (2010). 207. Strauss, P. Shakespeare and the English poets: The influence of n...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.