Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

AI Scientists Fail Without Strong Implementation Capability

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The fundamental bottleneck for AI Scientists is implementing and verifying experiments, not generating ideas.

desk verdict A clear, honest position paper on the AI Scientist implementation gap, worth citing and refereeing, though its 'fundamental bottleneck' claim leans on an unvalidated same-group LLM judge. read the letter →

arxiv 2506.01372 v2 pith:FO6N5HIC submitted 2025-06-02 cs.AI cs.CLcs.LG

Minjun Zhu , Qiujie Xie , Yixuan Weng , Jian Wu , Zhen Lin , Linyi Yang , Yue Zhang This is my paper
classification cs.AIcs.CLcs.LG
keywords AIScientistimplementationgapLLMagentsscientificdiscoveryverificationexperimentexecutionLLM-as-a-judgeAI-generatedpapers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the fundamental bottleneck for AI Scientist systems is not their ability to generate novel research ideas, but their capacity to execute and verify the experiments needed to test those ideas. It supports this claim with benchmark results showing that state-of-the-art LLMs score very low on tasks requiring them to run and reproduce real research code, and with a simulated peer review of 28 AI-generated papers in which every paper was flagged for experimental weakness. The paper defines an AI Scientist as an end-to-end system that both formulates ideas and carries out verification, and contends that current systems lack the latter capability. If correct, progress toward human-level AI scientists depends on solving implementation and verification rather than on scaling idea generation.

What carries the argument

The load-bearing concept is the 'implementation gap,' framed by a formal definition of an AI Scientist as a system $\mathcal{S}_{AI}$ that takes a research question and domain knowledge as input and outputs new knowledge plus verifiable artifacts, requiring both idea generation and experimental execution. The paper's empirical support rests on two instruments: a table of LLM performance on complex engineering benchmarks, and the DeepReviewer-14B review model used to score generated papers on soundness, presentation, contribution, and decision. The defect-category analysis, in which Experimental Weakness appears in 100% of the 28 evaluated papers, ties the benchmark failures to end-to-end research output.

What would settle it

Have expert human reviewers with machine learning backgrounds independently score the same 28 AI-generated papers using the same review criteria as DeepReviewer-14B, without knowing the model's scores; if human reviewers largely rate the papers as acceptable or find the experiments reproducible, the implementation-bottleneck claim loses its main empirical support. A second test: demonstrate that one AI Scientist system can produce code and experiments that an independent team reproduces without human fixes.

Watch

Extended reading notes

Core claim

The central claim is that the 'implementation gap' — the shortfall between generating ideas and reliably executing the verification procedures those ideas require — is the fundamental bottleneck for current AI Scientist systems. Quantitative evidence from five benchmarks (MLE-Bench, PaperBench, SciReplicate-Bench, CORE-Bench, and ML-Dev-Bench) shows LLM agents achieve low accuracy on code execution, reproduction, and debugging tasks. A simulated peer-review evaluation using DeepReviewer-14B finds that 28 publicly available research papers produced by five AI Scientist systems receive low average ratings and that all 28 exhibit 'Experimental Weakness.' The paper concludes that current systems lack the execution capabilities needed to run rigorous experiments and produce high-quality scientific papers.

Load-bearing premise

The paper's case depends on DeepReviewer-14B's scores being a valid measure of paper quality, since no human peer-review comparison is offered for the 28 AI-generated papers.

Editorial extensions

If this is right

  • Improvements in execution and verification capabilities would matter more than further advances in idea generation for making AI Scientists useful.
  • Evaluations of AI Scientist systems should center on whether their experiments are reproducible and their results verifiable, not just on the novelty of the proposed ideas.
  • Acceptance of AI-generated papers at workshops or conferences may overstate readiness, since the underlying experiments may not be reproducible or rigorous.
  • The community needs a comprehensive benchmark covering the whole scientific workflow, from idea to implementation, to fairly compare AI Scientist systems.
  • Human-AI collaboration as a 'co-scientist' may be the pragmatic route until implementation capabilities mature.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the implementation gap is indeed the bottleneck, then scaling idea-generation models alone will not produce usable AI scientists; research investment should shift toward execution scaffolds, verification tools, and feedback loops.
  • The paper's evidence would be strengthened by a human baseline: having expert reviewers score the same 28 papers would test whether DeepReviewer-14B's harsh ratings reflect real quality or model-specific bias.
  • A testable extension is a benchmark that measures the full end-to-end workflow, from question to reproducible result, since existing benchmarks mostly measure code reproduction rather than open-ended discovery.
  • If verification is the bottleneck, then hybrid systems where humans run experiments and AI proposes ideas may be the realistic near-term path, as the paper's own alternative-views section suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This position paper argues that the fundamental bottleneck for AI Scientists is implementation/verification capability rather than idea generation. The authors define an AI Scientist as an end-to-end system that independently formulates ideas and executes verification, then cite external benchmarks (PaperBench, MLE-Bench, SciReplicate-Bench, CORE-Bench, ML-Dev-Bench) showing low LLM performance on complex engineering tasks. They present a trend analysis of arXiv papers suggesting that implementation-focused work receives more citations, and they introduce a new evaluation of 28 papers from five AI Scientist systems using DeepReviewer-14B, which finds 100% experimental weakness across all papers. The paper discusses root causes (reasoning limits, planning deficiencies, collaboration issues, evaluation gaps) and proposes future directions including workflow structuring, RL acceleration, and multi-agent coordination.

Significance. If the central claim holds, it productively refocuses AI Scientist research toward implementation and verification, a timely and actionable message for the community. The external benchmark evidence in Section 3.2 is real, well-cited, and directly relevant, and the paper is transparent about the acknowledged selection bias in its own 28-paper sample. The position is plausible and the discussion of fundamental limitations is thoughtful. However, the new 28-paper evaluation relies on a same-author LLM judge with no calibration against human review, and the 'fundamental bottleneck' claim requires a stronger comparison between idea-generation and implementation capabilities than the paper currently provides.

major comments (4)
  1. [Section 3.3, Tables 2 and 3] The evaluation that directly supports the paper's central claim uses DeepReviewer-14B (Zhu et al., 2025) as the sole judge of the 28 AI-generated papers. This model comes from the same authorship group, and the paper reports no validation of DeepReviewer-14B against human peer review for AI-generated papers, no human-authored baseline scored under the same rubric, and no inter-annotator agreement. Consequently, the 100% 'Experimental Weakness' rate and the low average scores in Table 2 could reflect judge bias or a general property of AI-written papers rather than an implementation-specific gap. Because this is the only direct evidence about the quality of AI Scientist outputs, the measurement assumption is load-bearing for the paper's conclusion, and it needs to be addressed by calibration, a baseline, or a substantially more cautious interpretation.
  2. [Section 3.3, Table 2] The quantitative claims about system comparison rest on very small samples: Zochi has only 2 papers and AI Scientist v2 has only 3, with no error bars or significance tests reported. Table 2 reports averages only, so the difference between Zochi (4.63) and other systems (2–3) cannot be distinguished from sampling noise. The paper's own note that publicly available papers may be curated further weakens the inference from these numbers. The authors should either provide per-paper distributions and uncertainty estimates or refrain from drawing comparative conclusions from Table 2.
  3. [Section 4.2 and Section 3.3] Section 4.2 states that review scores are not reliable indicators for predicting future impact and that peer review is more suitable for filtering low-quality papers than identifying the highest quality ones. This caveat undermines the use of DeepReviewer-14B scores to conclude that current AI Scientist systems 'lack the execution capabilities needed to execute rigorous experiments and produce high-quality scientific papers.' If the review instrument itself is acknowledged to be an imperfect quality measure, the authors need an additional argument explaining why the specific defect categories in Table 3—especially Experimental Weakness—are nonetheless valid evidence for the implementation gap. Without such an argument, the evidence supports 'AI-generated papers receive low scores from this particular reviewer' rather than 'AI-generated papers have fundamentally deficient implementation.'
  4. [Section 3.2 and Section 1] The paper's central claim is that implementation is 'the fundamental bottleneck,' but the evidence in Section 3.2 shows only that current LLMs perform poorly on complex execution benchmarks. It does not directly compare the magnitude of the implementation deficit against the idea-generation deficit, nor does it show that idea generation is already near-saturated. The cited work on idea generation (e.g., Si et al., 2025) suggests novelty, but the paper does not quantify how many AI-generated ideas, when executed by humans, lead to successful research outcomes. To support 'fundamental bottleneck,' the paper would need either a direct comparative experiment or a more measured claim, such as 'a major bottleneck.' As written, the strong qualifier outruns the supplied evidence.
minor comments (5)
  1. [Abstract, Section 1] There are several spacing and grammar errors, such as 'systematic evaluation assess 28 research papers' and 'to execute rigorous experiments' missing a space before 'current.' A careful proofreading pass is needed.
  2. [Section 3.2, Table 1] The text states that 'Claude 3.5 Sonnet scored only 1.8% on PaperBench,' but the table reports 26.00% for OpenAI o1-high on the overall benchmark. The 1.8% figure is for a specific 'Execution' leaf node described later in the same section, not the overall PaperBench score. The sentence should be clarified to avoid confusing the overall benchmark score with the sub-task score.
  3. [Figure 3 and Section 3.1] The citation analysis in Figure 3 and Table 4 does not control for publication date, venue, field, or self-citations, and no statistical test is reported for the difference in average citations (10.3 vs. 25.0). At minimum, the authors should acknowledge these confounds and describe the trend as suggestive rather than causal.
  4. [Section 2.2, Eq. (1)] The formal definition in Eq. (1) is not operationalized anywhere; the 'max' notation and the variables (theta_AI, R_human, B_res) are not used again in the paper. This is acceptable for a position paper, but the notation should either be connected to later discussion or simplified to avoid an impression of formalism without content.
  5. [Appendix A] The sampling-time estimates in Appendix A are admittedly rough and are presented as an illustration rather than a rigorous analysis. This is acceptable, but the caveat should appear in the main text near Figure 4, and the assumptions (e.g., 40 tokens/sec, 12-hour experiment cycle) should be clearly labeled as speculative.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: external implementation benchmarks independently support the core claim; the DeepReviewer self-citation is a validity concern, not a circular reduction.

full rationale

This is a position paper rather than a derivation with predictions. The central claim that current AI Scientist systems are bottlenecked by implementation/verification capability is supported primarily by external, author-independent benchmarks (MLE-Bench, PaperBench, SciReplicate-Bench, CORE-Bench, ML-Dev-Bench) reported in Section 3.2. These benchmarks are not fitted to the paper's conclusion and provide independent evidence that LLM-based agents fail at code execution, result matching, debugging, and model-performance tasks. The Section 3.3 evaluation of 28 papers uses DeepReviewer-14B (Zhu et al., 2025), a model from the same authorship group, as the sole judge, and no calibration against human review or a human-authored baseline is reported; this is a legitimate concern about measurement validity and a self-citation, but it is not a definitional or by-construction reduction. The paper explicitly acknowledges the selection-bias limitation of the 28-paper sample and, in Section 4.2, notes that review scores are imperfect predictors of future impact. No equation in the paper is equivalent to its inputs, no parameter is fitted and then renamed a prediction, and no uniqueness theorem from the authors' prior work is invoked to force the conclusion. Therefore the core argument retains independent content; the minor self-citation warrants a low score, not a circularity finding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or formal entities; 'implementation gap' is a framing concept rather than an entity. The central empirical evidence rests on an LLM-based judge developed by the same group, on external benchmark scores, and on hand-chosen sampling-time estimates.

free parameters (2)
  • Sampling time assumptions in Appendix A = 40 tokens/s; T_search_API=10s; T_op_code=12h; T_op_lit=20min; Tsample_CS approximately 46,900s
    Figure 4 and the reinforcement learning cost argument depend on hand-chosen durations for code operation, literature search, and inference rates. The authors state these come from referenced literature and 'our experience,' not from measurements.
  • DeepReviewer acceptable-quality threshold = 6 on a 1-10 rating scale
    The interpretation of Table 2 depends on a threshold of 6 for acceptable quality, but no calibration to human accept/reject decisions is provided.
assumptions (4)
  • domain assumption DeepReviewer-14B scores are a valid proxy for the scientific quality of AI-generated papers.
    Section 3.3 uses DeepReviewer-14B as the sole judge of the 28 papers; no validation against human review for AI-generated papers is cited.
  • domain assumption Publicly available AI Scientist papers are usable evidence despite curation.
    Section 3.3 acknowledges that 'publicly available papers may be curated and not fully representative' but still treats the 28 papers as evidence about average output quality.
  • domain assumption External benchmark scores transfer to AI Scientist implementation capability.
    Section 3.2 uses PaperBench, MLE-Bench, SciReplicate-Bench, CORE-Bench, and ML-Dev-Bench as evidence; these tasks are related to, but not identical with, end-to-end AI Scientist workflows.
  • domain assumption The absence of AlphaFold-class AI Scientist breakthroughs is attributable to implementation rather than other factors.
    The introduction and conclusion infer the bottleneck from the lack of groundbreaking CS results, but do not control for alternative explanations such as research topic difficulty, evaluation standards, or short time since system deployment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Scientists Fail Without Strong Implementation Capability." pith.science (2026). https://pith.science/paper/FO6N5HIC

@misc{pith2026250601372,
  author       = {Pith},
  title        = {Pith review of: AI Scientists Fail Without Strong Implementation Capability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FO6N5HIC}},
  note         = {Machine review of arXiv:2506.01372}
}
read the original abstract

The emergence of Artificial Intelligence (AI) Scientist represents a paradigm shift in scientific discovery, with large language models (LLMs) taking the lead as the primary executor in the entire scientific workflow from idea generation to experiment implementation. Recent AI Scientist studies demonstrate sufficient capabilities for independent scientific discovery, with the generated research reports gaining acceptance at the ICLR 2025 workshop and ACL 2025, arguing that a human-level AI Scientist, capable of uncovering phenomena previously unknown to humans, may be imminent. Despite this substantial progress, AI Scientist has yet to produce a groundbreaking achievement in the domain of computer science on par with automated scientific tools. Based on extensive quantitative evidence from existing benchmarks in complex engineering tasks and a systematic evaluation assess 28 research papers generated by five advanced AI Scientist systems, we argue that \textbf{the fundamental bottleneck for AI Scientists lies in their capability to execute the requisite verification procedures.} Current AI Scientist systems lack the execution capabilities needed to execute rigorous experiments and produce high-quality scientific papers. To better illustrate the root cause of this \textbf{implementation gap}, we provide an in-depth discussion on the fundamental limitations of AI Scientist. This position paper aims to call for the participants in the community to bridge the implementation gap.

Figures

Figures reproduced from arXiv: 2506.01372 by the authors.

Figure 1
Figure 1. The roadmap of AI Scientist from 2024 to future, highlighting key milestones and fundamental challenges that must be overcome to bridge the implementation gap of AI Scientist. 6). We hope this position paper will contribute to a clearer understanding of the limitations of current AI Scientist, shedding light on the future development of AI Scientist. 2. Definition of the AI Scientist The emergence of automated scien… view at source ↗
Figure 2
Figure 2. The difference between AI Scientist and scientific tool. Sci￾entific tools generate predictions under human supervision, while AI Scientist autonomously leverage tools to address research questions. Scientific tools, originating from AI for Sci￾ence research, represent specialized AI sys￾tems designed to solve specific scientific problems by processing data and generat￾ing results within defined domains. These tools… view at source ↗
Figure 3
Figure 3. Analysis of AI Scientist publications on arXiv. The upper panel displays the average number of citations up to now, categorized by containing implementation details. The lower panel shows the growth in the total number of these papers with the same categorization. Empirical Evidence of Implementation Gap. Ad￾vanced LLMs achieve near-saturated performance on simple code generation benchmarks like HumanEval (Chen et a… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Estimated time to solve representative tasks for different agent types (AI vs. Human), which for AI agents also corresponds to single-sample RL sampling duration. Limitation 3: multi-agent col￾laboration. Ideal AI Scientist should seamlessly integrate into complex rese…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One Run Is Not an Idea: The Implementation Lottery in Automated Research

    cs.MA 2026-07 conditional novelty 7.0 of 10

    In audits of 312 LLM research assignments, variation across implementations of the same idea dominates rerun noise and reverses the selected idea winner in 25.6-43.6% of decisions.

Reference graph

Works this paper leans on

65 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [3]

    The relationship between reasoning and performance in large language models–o3 (mini) thinks harder, not longer.arXiv preprint arXiv:2502.15631,

    Marthe Ballon, Andres Algaba, and Vincent Ginis. The relationship between reasoning and performance in large language models–o3 (mini) thinks harder, not longer.arXiv preprint arXiv:2502.15631,

    arXiv

  2. [4]

    Superintelligent agents pose catastrophic risks: Can scientist ai offer a safer path?arXiv preprint arXiv:2502.15657,

    Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, et al. Superintelligent agents pose catastrophic risks: Can scientist ai offer a safer path?arXiv preprint arXiv:2502.15657,

    arXiv

  3. [6]

    Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657,

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657,

    arXiv

  4. [7]

    Mle-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    arXiv

  5. [8]

    doi: 10.1145/3641289

    ISSN 2157-6904. doi: 10.1145/3641289. URLhttps://doi.org/10.1145/3641289. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  6. [9]

    Corinna Cortes and Neil D Lawrence

    URLhttps://openreview.net/forum?id=6z4YKr0GK6. Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.arXiv preprint arXiv:2109.09774,

    arXiv 2014

  7. [10]

    Iris: Interactive research ideation system for accelerating scientific discovery.arXiv preprint arXiv:2504.16728,

    Aniketh Garikaparthi, Manasi Patwardhan, Lovekesh Vig, and Arman Cohan. Iris: Interactive research ideation system for accelerating scientific discovery.arXiv preprint arXiv:2504.16728,

    arXiv

  8. [11]

    Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning.arXiv preprint arXiv:2409.05556,

    Alireza Ghafarollahi and Markus J Buehler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning.arXiv preprint arXiv:2409.05556,

    arXiv

Show all 65 references
  1. [12]

    Towards an ai co-scientist.arXiv preprint arXiv:2502.18864,

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist.arXiv preprint arXiv:2502.18864,

    arXiv

  2. [13]

    Llms can realize combinatorial creativity: generating creative ideas via llms for scientific research.arXiv preprint arXiv:2412.14141, 2024a

    Tianyang Gu, Jingjin Wang, Zhihao Zhang, and HaoHong Li. Llms can realize combinatorial creativity: generating creative ideas via llms for scientific research.arXiv preprint arXiv:2412.14141, 2024a. Yang Gu, Hengyu You, Jian Cao, Muran Yu, Haoran Fan, and Shiyou Qian. Large la...

    arXiv

  3. [14]

    Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680,

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680,

    arXiv

  4. [15]

    Pasa: An llm agent for comprehensive academic paper search.arXiv preprint arXiv:2501.10120,

    Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. Pasa: An llm agent for comprehensive academic paper search.arXiv preprint arXiv:2501.10120,

    arXiv

  5. [16]

    Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278,

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278,

    arXiv

  6. [17]

    Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas.arXiv preprint arXiv:2410.14255,

    Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas.arXiv preprint arXiv:2410.14255,

    arXiv

  7. [18]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    14 AI Scientists Fail Without Strong Implementation Capability. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language mode...

    arXiv

  8. [19]

    Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation.arXiv preprint arXiv:2503.22708,

    Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bod- hisattwa Prasad Majumder, Daniel S Weld, and Peter Clark. Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation.arXiv preprint arXiv:25...

    arXiv

  9. [20]

    Aide: Ai-driven exploration in the space of code.arXiv preprint arXiv:2502.13138,

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code.arXiv preprint arXiv:2502.13138,

    arXiv

  10. [23]

    Llms get lost in multi-turn conversation

    Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120,

    arXiv

  11. [25]

    Drugagent: Au- tomating ai-aided drug discovery programming through llm multi-agent collaboration.arXiv preprint arXiv:2411.15692, 2024a

    Sizhe Liu, Yizhou Lu, Siyu Chen, Xiyang Hu, Jieyu Zhao, Yingzhou Lu, and Yue Zhao. Drugagent: Au- tomating ai-aided drug discovery programming through llm multi-agent collaboration.arXiv preprint arXiv:2411.15692, 2024a. Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Y...

    arXiv

  12. [26]

    Aigs: Generating science from ai-powered automated falsification.arXiv preprint arXiv:2411.11910, 2024b

    Zijun Liu, Kaiming Liu, Yiqi Zhu, Xuanyu Lei, Zonghan Yang, Zhenhe Zhang, Peng Li, and Yang Liu. Aigs: Generating science from ai-powered automated falsification.arXiv preprint arXiv:2411.11910, 2024b. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and Dav...

    arXiv

  13. [27]

    Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk

    URL https://www.arxiv.org/abs/2408.06292v3. Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery.Nature, 624(7990):80–85,

    arXiv

  14. [29]

    Sparks of science: Hypothesis generation using structured paper data.arXiv preprint arXiv:2504.12976,

    Charles O’Neill, Tirthankar Ghosal, Roberta Răileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciucă. Sparks of science: Hypothesis generation using structured paper data.arXiv preprint arXiv:2504.12976,

    arXiv

  15. [30]

    Mathis Pink, Qinyuan Wu, Vy Ai Vo, Javier Turek, Jianing Mu, Alexander Huth, and Mariya Toneva

    URLhttps://arxiv.org/abs/2502.00964. Mathis Pink, Qinyuan Wu, Vy Ai Vo, Javier Turek, Jianing Mu, Alexander Huth, and Mariya Toneva. Position: Episodic memory is the missing piece for long-term llm agents.arXiv preprint arXiv:2502.06975,

    arXiv

  16. [31]

    Ideasynth: Iterative research idea development through evolving and composing idea facets with literature-grounded feedback

    Kevin Pu, KJ Kevin Feng, Tovi Grossman, Tom Hope, Bhavana Dalvi Mishra, Matt Latzke, Jonathan Bragg, Joseph Chee Chang, and Pao Siangliulue. Ideasynth: Iterative research idea development through evolving and composing idea facets with literature-grounded feedback. InProceedin...

    2025

  17. [32]

    Iterative hypothesis generation for scientific discovery with monte carlo nash equilibrium self-refining trees.arXiv preprint arXiv:2503.19309,

    Gollam Rabby, Diyana Muhammed, Prasenjit Mitra, and Sören Auer. Iterative hypothesis generation for scientific discovery with monte carlo nash equilibrium self-refining trees.arXiv preprint arXiv:2503.19309,

    arXiv

  18. [33]

    Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination.arXiv preprint arXiv:2409.14634,

    Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S Weld. Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination.arXiv preprint arXiv:2409.14634,

    arXiv

  19. [34]

    Astroagents: A multi-agent ai for hypothesis generation from mass spectrometry data.arXiv preprint arXiv:2503.23170,

    Daniel Saeedi, Denise Buckner, Jose C Aponte, and Amirali Aghazadeh. Astroagents: A multi-agent ai for hypothesis generation from mass spectrometry data.arXiv preprint arXiv:2503.23170,

    arXiv

  20. [35]

    Spark: A system for scientifically creative idea generation.arXiv preprint arXiv:2504.20090,

    Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R Varshney, and Dilek Hakkani-Tur. Spark: A system for scientifically creative idea generation.arXiv preprint arXiv:2504.20090,

    arXiv

  21. [36]

    Agent laboratory: Using llm agents as research assistants.arXiv preprint arXiv:2501.04227,

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants.arXiv preprint arXiv:2501.04227,

    arXiv

  22. [37]

    Paper2code: Automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192,

    Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192,

  23. [38]

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto

    URLhttps://arxiv.org/abs/ 2407.00132. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

    arXiv

  24. [39]

    Canllmsgeneratenovelresearchideas? Alarge-scalehuman study with 100+ NLP researchers

    ChengleiSi,DiyiYang,andTatsunoriHashimoto. Canllmsgeneratenovelresearchideas? Alarge-scalehuman study with 100+ NLP researchers. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025

  25. [40]

    Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan

    URLhttps://openreview.net/ forum?id=M23dTGWCZy. Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.11363,

    arXiv

  26. [41]

    Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    arXiv

  27. [43]

    Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588,

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588,

    arXiv

  28. [44]

    SciMON: Scientific inspiration machines optimized for novelty

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific inspiration machines optimized for novelty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    arXiv

  29. [45]

    Large language models are better reasoners with self-verification

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575,

    2023

  30. [46]

    Yuhao Wu, Yushi Bai, Zhiqing Hu, Shangqing Tu, Ming Shan Hee, Juanzi Li, and Roy Ka-Wei Lee

    URLhttps://openreview.net/forum?id=bjcsVLoHYs. Yuhao Wu, Yushi Bai, Zhiqing Hu, Shangqing Tu, Ming Shan Hee, Juanzi Li, and Roy Ka-Wei Lee. Shifting long-context llms research from input to output.arXiv preprint arXiv:2503.04723, 2025a. Yuyang Wu, Yifei Wang, Tianqi Du, Stefan...

    arXiv

  31. [47]

    Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang

    18 AI Scientists Fail Without Strong Implementation Capability. Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Improving scientific hypothesis generation with knowledge grounded large language models.arXiv preprint arXiv:2411.02382,

    arXiv

  32. [48]

    The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066,

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066,

    arXiv

  33. [49]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Ha...

    arXiv

  34. [50]

    MOOSE-chem: Large language models for rediscovering unseen chemistry scientific hypotheses

    Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. MOOSE-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. InThe Thirteenth International Conference on Learning Representat...

    arXiv

  35. [51]

    Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback.arXiv preprint arXiv:2501.03916,

    Jiakang Yuan, Xiangchao Yan, Botian Shi, Tao Chen, Wanli Ouyang, Bo Zhang, Lei Bai, Yu Qiao, and Bowen Zhou. Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback.arXiv preprint arXiv:2501.03916,

    arXiv

  36. [52]

    A survey on test-time scaling in large language models: What, how, where, and how well?arXiv preprint arXiv:2503.24235,

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling in large language models: What, how, where, and how well?arXiv preprint arXiv:2503.24235,

    arXiv

  37. [53]

    Openresearcher: Unleashing ai for accelerated scientific research.arXiv preprint arXiv:2408.06941,

    Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, et al. Openresearcher: Unleashing ai for accelerated scientific research.arXiv preprint arXiv:2408.06941,

    arXiv

  38. [54]

    Deepreview: Improving llm-based paper review with human-like deep thinking process.arXiv preprint arXiv:2503.08569,

    Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Deepreview: Improving llm-based paper review with human-like deep thinking process.arXiv preprint arXiv:2503.08569,

    arXiv

  39. [55]

    19 AI Scientists Fail Without Strong Implementation Capability. A. Sampling Time Calculation for Different Types of AI Agents We referenced existing literature (Guo et al., 2025, Yang et al., 2025a, Muennighoff et al.,

    2025

  40. [56]

    20 AI Scientists Fail Without Strong Implementation Capability

    for statistics; however, even under these conditions, humans achieve a success rate of less than 50%. 20 AI Scientists Fail Without Strong Implementation Capability. Table 4:Timeline of AI Scientist Ideas and Code Implementations by Month 2024-08 2024-09 2024-10 2024-11 2024-1...

    2024

  41. [57]

    (Ghafarollahi and Buehler, 2024),(Raden- sky et al.,

    2024

  42. [58]

    (Pu et al., 2025a), (Yang et al., 2024), (Su et al., 2024), (Li et al., 2024a), (Hu et al., 2024), (Liu et al., 2025), (Wang et al., 2024b) (Weng et al., 2025), (Xiong et al.,

    2024

  43. [59]

    (Rabby et al., 2025), (Saeedi et al.,

    2025

  44. [60]

    (O’Neill et al., 2025), (Garikaparthi et al., 2025), (Sanyal et al.,

    2025

  45. [61]

    w/ Exp(Lu et al., 2024), (Li et al., 2024c) (Liu et al., 2024b), (Liu et al., 2024a) (Yuan et al., 2025), (Schmidgall et al.,

    2024

  46. [62]

    (Jiang et al., 2025), (Kon et al.,

    2025

  47. [63]

    (Schmidgall et al., 2025), (Jansen et al.,

    2025

  48. [64]

    (Yamada et al., 2025), (Seo et al.,

    2025

  49. [65]

    Regarding the statistics for the papers We have conducted a comprehensive search on arXiv to gather relevant publications in the AI Scientist field

    B. Regarding the statistics for the papers We have conducted a comprehensive search on arXiv to gather relevant publications in the AI Scientist field. This collection includes a series of papers from August 2024 to April 2025 for methods or systems, which are cited in Table

    2024

  50. [1987]

    Chain of ideas: Revolutionizing research via novel idea development with llm agents.arXiv preprint arXiv:2410.13185, 2024a

    Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, et al. Chain of ideas: Revolutionizing research via novel idea development with llm agents.arXiv preprint arXiv:2410.13185, 2024a. Ruochen Li, Liqiang ...

    arXiv

  51. [2009]

    Curie: Toward rigorous and automated scientific experimentation with ai agents.arXiv preprint arXiv:2502.16069,

    Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang, Yibo Huang, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, and Ang Chen. Curie: Toward rigorous and automated scientific experimentation with ai agents.arXiv preprint arXiv:2502.16069,

    arXiv

  52. [2013]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    arXiv

  53. [2020]

    Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation.arXiv preprint arXiv:2410.09403,

    Haoyang Su, Renqi Chen, Shixiang Tang, Xinzhe Zheng, Jingzhe Li, Zhenfei Yin, Wanli Ouyang, and Nanqing Dong. Two heads are better than one: A multi-agent system has the potential to improve scientific idea generation.arXiv preprint arXiv:2410.09403,

    arXiv

  54. [2021]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  55. [2023]

    Genome modeling and design across all domains of life with evo 2.BioRxiv, pages 2025–02,

    Garyk Brixi, Matthew G Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A Gonzalez, Samuel H King, David B Li, Aditi T Merchant, et al. Genome modeling and design across all domains of life with evo 2.BioRxiv, pages 2025–02,

    2025

  56. [2024]

    Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz

    1https://ai-researcher.net/social-iclr-2025 12 AI Scientists Fail Without Strong Implementation Capability. Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. A survey on rag with llms. Procedia Computer Science, 246:3781–3790,

    2025

  57. [2025]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang

    URL https://www.unite.ai/ googles-new-ai-co-scientist-aims-to-accelerate-scientific-discovery/. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms.arXiv preprint arXiv:2410.06992,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.