REVIEW 4 major objections 6 minor 2 cited by
Predicting Empirical AI Research Outcomes with Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A fine-tuned language model with paper retrieval predicts which of two AI research ideas will win on benchmarks, and beats human experts in the paper's comparisons.
desk verdict A useful benchmark and a careful evaluation, but the 77% accuracy is probably inflated by a proposed-vs-baseline shortcut that the stress tests don't isolate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a pairwise prediction pipeline whose inputs are idea summaries with empirical results removed, a goal defined by a set of benchmarks with metrics, and a binary label aggregated by majority vote over those benchmarks. The retrieval agent searches only papers published before June 1, 2024, decomposes the novel idea into sub-components, queries for transferable findings, summarizes whole PDFs instead of abstracts, and filters by relevance; the paper reports whole-paper summarization lifts accuracy from 38.8% to 53.0%. Fine-tuning GPT-4.1 on 6,000 historical idea pairs with raw outcome labels completes the mechanism, lifting accuracy to 77%. The contrast between the fine-tuned result and near-chance zero-shot results marks supervised pattern-learning over historical experimental outcomes as the component that carries the argument.
What would settle it
Search every retrieved paper list for the 1,585 test pairs and flag any retrieved item that is the test paper itself, a direct citation of it, or a paper quoting its exact benchmark scores; even one such hit at the reported retrieval scale would mean the 77% figure is inflated, and a re-run with those hits removed would show how much.
Extended reading notes
Core claim
The central claim is that the empirical success of a research idea can be predicted from its description alone, without running experiments, and that a fine-tuned retrieval-augmented language model does this better than human experts. The paper reports 77% accuracy on a 1,585-pair human-verified test set, 64.4% on a 45-pair NLP subset against 48.9% for human experts, and 63.6% on 33 unpublished idea pairs whose groundtruth comes from expert implementations. Off-the-shelf frontier models, including o3 and Claude 3.5 Sonnet, stay near chance even with the same retrieval augmentation, while fine-tuning GPT-4.1 on 6,000 historical pairs lifts accuracy from 51.9% to 77%. Fine-tuning on self-generated chain-of-thought rationales provides no gain, and stress tests indicate the model is not leaning on surface cues like idea length, recency, or famous lab names.
Load-bearing premise
The result would be void if any test pair's outcome was knowable to the model in advance, through the training data, the retrieval pool, or subtle cues in the idea summary, because then the measured accuracy would be recall rather than prediction.
Editorial extensions
If this is right
- Research groups can use the system to rank candidate ideas before implementation, shifting compute and human effort toward the ideas predicted to win.
- An AI ideation agent can use the same predictor as a reward model, filtering its generated proposals before any experiment is run.
- The 6,000-pair training set plus 1,585 verified test pairs establish a reusable benchmark for measuring future attempts at research-outcome prediction.
- The near-chance performance of off-the-shelf frontier models implies that capability must be elicited through fine-tuning on historical outcomes, not just longer prompts or better base models.
- Because self-augmented CoTs did not help, the paper argues that low-quality model-generated rationales add noise; future systems may need externally sourced or experimentally grounded reasoning traces.
Reading between the lines
- One extension the authors leave implicit: the same pipeline could be applied to other empirical disciplines whose papers report quantitative benchmark comparisons, such as systems research or computational chemistry, wherever groundtruth can be extracted as pairwise wins.
- A testable consequence of the 64.4% versus 48.9% gap is that human experts anchor on narrative features like complexity and novelty, so a system trained only on outcomes may be less susceptible; this could be checked by measuring how often expert rationales invoke such features.
- If the 63.6% unpublished-idea result generalizes, then outcome prediction becomes an economical pre-filter for AI-driven discovery loops, cutting the dominant cost of executing and validating ideas rather than the cost of generating them.
- The 77% figure should be stress-tested by increasing the temporal gap between training and test ideas; the current test set guarantees at least one post-cutoff idea per pair, but a longer gap would test whether the model extrapolates as AI research evolves.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces a benchmark and a system for the task of predicting, from textual descriptions of two research ideas, which idea will achieve better empirical results on a shared set of benchmarks. The benchmark is built by scraping idea pairs and outcome labels from conference papers, with 6,000 training pairs from pre-June-2024 papers and 1,585 human-verified test pairs containing at least one post-June-2024 idea. The system fine-tunes GPT-4.1 on the training pairs and augments it with a retrieval agent that searches pre-cutoff arxiv papers. On the test set it achieves 77% accuracy; on a 45-pair NLP subset it achieves 64.4% versus 48.9% for majority-vote human experts. The paper also reports robustness to a range of stress tests and 63.6% accuracy on 33 unpublished idea pairs from an AI-ideation study, and argues that the system can serve as a reward model for automated research.
Significance. If the 77% figure reflects genuine prospective prediction, this would be an important new capability and a reusable evaluation resource. The test set is large, human-verified with a measured reduction in labeling error from 11% to 2.5%, and the retrieval and fine-tuning design are clearly described. The human-expert comparison is a useful first step, and the stress-testing battery is more extensive than in most benchmark papers. The main reason for caution is that the benchmark construction pairs ideas within a single paper, making it possible for the system to latch onto the proposed-versus-baseline structure of papers rather than onto the empirical content of ideas; the current evidence does not fully exclude this explanation. The claims about human superiority and unpublished-idea generalization are also statistically underpowered.
major comments (4)
- [2.1, 3.2, 5] Pairs are created only within the same paper (Section 2.1, Step 3), and the label is read from the paper's own result tables. In this construction, one idea is usually the paper's proposed contribution and the other is a baseline, and the proposed idea's summary is written from the test paper while the baseline's summary is written from a referenced paper. A model fine-tuned on 6,000 pairs of the same construction could learn to identify which idea is described as the contribution ('proposed', 'introduced', more mechanism detail) rather than to predict which idea works better. None of the stress tests in Section 5 holds fixed or removes the proposed-vs-baseline asymmetry, and the 63.6% result on unpublished ideas is consistent with this shortcut. Please report the base rate of 'proposed idea wins' in the test set, accuracy conditioned on whether the proposed or baseline idea is the winner, and a control experiment that isolates proposedness (e.g., a probe classifier trained to detect which summary is the paper's contribution, or an evaluation on baseline-vs-baseline pairs).
- [2.1, Table 2] The contamination control hinges on 'published after June 1, 2024,' but the manuscript never defines whether this is the arXiv first-posting date or the conference publication date. If a test paper was posted on arXiv before the GPT-4.1 knowledge cutoff and only 'published' in a conference afterward, the base model may have seen its text, and the headline accuracy would overstate true predictive skill. Please specify the date definition, verify it for every test example, and describe the retrievability asymmetry that arises when the baseline idea is from a pre-cutoff paper (its original paper is retrievable while the new idea is not), including whether this asymmetry predicts the label.
- [4.3, Figure 1] The expert comparison is based on 45 NLP pairs selected by the authors, with no confidence intervals, no significance test, and no description of the selection rule. With 29/45 correct for the system and 22/45 for the majority vote, the difference is within sampling error; the 'ceiling' baseline that chooses the best annotator per topic using the test labels is also not an upper bound in any statistically meaningful sense. Please report CIs or bootstrap intervals, use a paired test (e.g., McNemar) where appropriate, and state how the 45 pairs were sampled.
- [6] The unpublished-idea experiment contains only 33 labeled pairs, and 63.6% (21/33) is not statistically distinguishable from chance (95% CI for the binomial proportion is approximately 46%–78%, p≈0.08). The abstract's wording 'demonstrating its potential as a reward model' goes beyond what this sample supports. In addition, since the dataset comes from reference [14], whose first author is also a co-author of this manuscript, the paper should state how temporal and practical independence was ensured (e.g., when the ideas were implemented and whether any of the authors of the present paper had access to the outcomes before the system was finalized).
minor comments (6)
- [Throughout] There are several typos: 'respctive' (Section 2.2), 'indidates' and 'stree-tests' (Section 5), 'valuation' (Section 5.2 text), 'resues' (Section 3.1), and 'ACMMM' (Table 1) should be 'ACM MM'.
- [Introduction/Section 4.2] The Introduction says off-the-shelf frontier models such as Claude 3.7 Sonnet perform no better than random, but the footnote in Section 4.2 says Claude 3.7 Sonnet was not evaluated because of its November 2024 cutoff; these statements conflict and should be reconciled.
- [Table 4] Please clarify whether all rows include retrieval; as printed, the reader cannot tell whether the +FT gains are relative to a GPT-4.1 baseline with or without the retrieval agent.
- [Figure 3] Figure 3 is not described in sufficient detail (no error bars, no list of models, no confidence intervals); please expand the caption and the text around it.
- [Section 5.1] The 'Famous Lab Names' perturbation is applied only to losing ideas; appending famous lab names to winning ideas as well would provide a stronger symmetry check.
- [Section 6] Please define precisely what the 'baseline' is for each unpublished idea (e.g., the standard prompting method used in the original study), since the task then becomes 'identify the novel idea,' which is related to but weaker than pairwise outcome prediction.
Circularity Check
No construction-level circularity: the 77% result is a cutoff-controlled holdout prediction, and the unpublished-idea evaluation rests on externally implemented outcomes.
full rationale
The central derivation is a predictive holdout: the system is fine-tuned on 6,000 pre-cutoff pairs and tested on 1,585 pairs each containing at least one idea published after GPT-4.1's June 1, 2024 cutoff (Section 2.1), so the headline accuracy cannot be a memorized reproduction of test labels. Retrieval is restricted to pre-cutoff papers (Section 3.1), and idea summaries are generated with instructions not to include empirical results; the 77% figure is therefore not a fitted parameter renamed as a prediction. The most plausible circularity candidates do not reduce by construction. Section 6's unpublished-idea evaluation uses reference [14], whose first author is also an author of this paper, but the outcomes there were obtained by independent human implementers at 103.4 hours per idea, making it externally falsifiable rather than a self-citation used to justify the claim. Section 4.3's 'ceiling human baseline' is admittedly selected on the same test set and is labeled a ceiling, so it is not presented as an honest predictive baseline; the headline human comparison uses the majority-vote 48.9% figure, which is genuinely elicited. The proposed-vs-baseline asymmetry inherent in within-paper pairs (Section 2.1, Step 3) is a real validity threat to construct generalization, and the paper itself concedes it 'cannot rule out the possible reliance on spurious features' (Section 8). But this is a shortcut/leakage concern about what the model has learned, not a definitional equivalence between input and label; the LM-designed stress tests (Section 5.2) provide some counter-evidence by showing accuracy remains above 61% even on subsets where hypothesized superficial features contradict the label. Overall, no step in the claimed derivation chain is equivalent to its own input by construction, so there is no significant circularity.
Assumptions & free parameters
free parameters (3)
- Top-15 retrieved papers per query =
15
- Minimum benchmarks per idea pair =
3
- Flagged-hypothesis accuracy threshold =
62%
assumptions (5)
- domain assumption Paper-reported benchmark results and win conditions extracted by LMs are accurate after human verification
- domain assumption The June 1, 2024 cutoff split prevents leakage of test ideas into GPT-4.1's training or fine-tuning data
- domain assumption Idea summaries generated from PDFs omit empirical results
- ad hoc to paper The 35 unpublished ideas from reference [14] are independent and uncontaminated despite author overlap
- ad hoc to paper The 45 NLP pairs selected for expert evaluation are representative of the test set
Cite this review
Pith. "Pith review of Predicting Empirical AI Research Outcomes with Language Models." pith.science (2026). https://pith.science/paper/CMFLGHS7
@misc{pith2026250600794,
author = {Pith},
title = {Pith review of: Predicting Empirical AI Research Outcomes with Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/CMFLGHS7}},
note = {Machine review of arXiv:2506.00794}
}
read the original abstract
Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs published after our base model's cut-off date for testing, and 6,000 pairs for training. We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with. In the NLP domain, our system beats human experts by a large margin (64.4% v.s. 48.9%). On the full test set, our system achieves 77% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation. We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests. Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent. Our system achieves 63.6% accuracy, demonstrating its potential as a reward model for improving idea generation models. Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 2 Pith papers
-
The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
A randomized execution study with 43 experts shows that LLM-generated research ideas lose more of their appeal than human ideas when actually implemented, reversing part of their ideation-stage advantage.
-
Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts
A position paper advocating large-scale training of event forecasting LLMs, with proposals for label selection, counterfactual training data, auxiliary rewards, and multi-source datasets.
Reference graph
Works this paper leans on
-
[14]
Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024. 10 Table 7: Typical errors found in the human verification process sorted by frequency. Human annotators will rewrite or directly remove the incorrect data. Error Type Example ...
arXiv 2024
-
[1]
Humans vs large language models: Judgmental forecasting in an era of advanced ai
Mahdi Abolghasemi, Odkhishig Ganbold, and Kristian Rotaru. Humans vs large language models: Judgmental forecasting in an era of advanced ai. arXiv preprint arXiv:2312.06941, 2024
arXiv 2024
-
[2]
Laradji, Laurent Charlin, and Christopher Pal
Shubham Agarwal, Issam H. Laradji, Laurent Charlin, and Christopher Pal. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788, 2024
arXiv 2024
-
[3]
Alpacafarm: A simulation framework for methods that learn from human feedback
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems , 36:30039–30069, 2023
2023
-
[4]
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Vikram Dhillo...
arXiv 2025
-
[5]
Approaching human-level forecasting with language models
Danny Halawi, Fred Zhang, Yueh-Han Chen, and Jacob Steinhardt. Approaching human-level forecasting with language models. arXiv preprint arXiv:2402.18563, 2024
arXiv 2024
-
[6]
Mlagentbench: Evaluating language agents on machine learning experimentation
Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023
arXiv 2023
-
[7]
Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. Forecastbench: A dynamic benchmark of ai forecasting capabilities. arXiv preprint arXiv:2409.19839, 2025
arXiv 2025
Show all 19 references
-
[8]
Can large language models provide useful feedback on research papers? a large-scale empirical analysis
Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas V odrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI , 1(8):AIoa2400196, 2024
2024
-
[9]
The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024
2024 arXiv
-
[10]
Large language models surpass human experts in predicting neuroscience results
Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, Anton Pashkov, et al. Large language models surpass human experts in predicting neuroscience results. Nature human behaviour, 9(2):305...
2025
-
[11]
Neurips 2021 summary, 2021
None. Neurips 2021 summary, 2021
2021
-
[12]
gradient descent
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search.arXiv preprint arXiv:2305.03495, 2023
2023 arXiv
-
[13]
Philipp Schoenegger and Peter S. Park. Large language model prediction capabilities: Evidence from a real-world forecasting tournament. arXiv preprint arXiv:2310.13014, 2023
2023 arXiv
-
[15]
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022
2022
-
[16]
The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025
2025 arXiv
-
[17]
Star: Self-taught reasoner bootstrapping reasoning with reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 1126, 2024
2024
-
[18]
Goal driven discovery of distributional differences via language descriptions
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. Advances in Neural Information Processing Systems, 36:40204–40237, 2023
2023
-
[19]
Forecasting future world events with neural networks
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. arXiv preprint arXiv:2206.15474, 2022. Appendix A Human Verification of Test Examples Tabl...
2022 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.