REVIEW 11 cited by
CycleResearcher: Improving Automated Research via Automated Review
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The automation of scientific discovery has been a long-standing goal within the research community, driven by the potential to accelerate knowledge creation. While significant progress has been made using commercial large language models (LLMs) as research assistants or idea generators, the possibility of automating the entire research process with open-source LLMs remains largely unexplored. This paper explores the feasibility of using open-source post-trained LLMs as autonomous agents capable of performing the full cycle of automated research and review, from literature review and manuscript preparation to peer review and paper refinement. Our iterative preference training framework consists of CycleResearcher, which conducts research tasks, and CycleReviewer, which simulates the peer review process, providing iterative feedback via reinforcement learning. To train these models, we develop two new datasets, Review-5k and Research-14k, reflecting real-world machine learning research and peer review dynamics. Our results demonstrate that CycleReviewer achieves promising performance with a 26.89\% reduction in mean absolute error (MAE) compared to individual human reviewers in predicting paper scores, indicating the potential of LLMs to effectively assist expert-level research evaluation. In research, the papers generated by the CycleResearcher model achieved a score of 5.36 in simulated peer reviews, showing some competitiveness in terms of simulated review scores compared to the preprint level of 5.24 from human experts, while still having room for improvement compared to the accepted paper level of 5.69. This work represents a significant step toward fully automated scientific inquiry, providing ethical safeguards and exploring AI-driven research capabilities. The code, dataset and model weight are released at https://wengsyx.github.io/Researcher/.
Forward citations
Cited by 11 Pith papers
-
Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review
In a benchmark of four AI Scientist systems on 15 FARS proposals, LLM reviewers rated FARS's own papers about twice as high as Sakana v1/v2, CycleResearcher, and Data-to-Paper outputs, but the evaluation lacks human v...
-
AI Can Learn Scientific Taste
Reinforcement learning on citation-preference pairs teaches a model to predict which papers will be cited more and to propose ideas that LLM judges rate as likely to be cited more—but "taste" here means citation impact.
-
FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
In FIRE-Bench's rediscovery test — question-only prompt, methods withheld — the best agent averaged 46.7 F1 and no agent reached 50, with failures concentrated in research planning and conclusion formation.
-
ReviewRL: Towards Automated Scientific Review with RL
ReviewRL combines arXiv retrieval, supervised fine-tuning, and reinforcement learning with a composite reward to generate paper reviews that better match human ratings and judged quality.
-
SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
SafeScientist adds prompt, discussion, tool-use, and output-review safety checks to an AI scientist, with a new domain benchmark, but its reported evaluation is internally inconsistent.
-
AI-Researcher: Autonomous Scientific Innovation
AI-Researcher runs an end-to-end ML research pipeline with LLM agents, and Scientist-Bench measures how close the resulting papers come to human-authored publications.
-
Prompt-to-Paper: Agentic AI System for Bioinformatics
An agentic bioinformatics manuscript system with deterministic RAG, real experiment execution, and a quality-driven rewrite loop raises automated scores by ~18 points on five case studies at ~$0.31 per paper.
-
Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
A review that categorizes large-model-empowered embodied AI into hierarchical and end-to-end decision-making, imitation and reinforcement learning, and world models.
-
Deep Research Agents: A Systematic Examination And Roadmap
A survey that organizes LLM-powered deep research agents into static versus dynamic workflows and single versus multi agent architectures, and reviews their benchmarks and open challenges.
-
A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Sign in to comment.