REVIEW 4 major objections 5 minor 2 cited by
Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read cmbagent, a system of about 30 LLM agents, claims to run an end-to-end cosmology analysis with no human in the loop.
desk verdict Real open-source multi-agent system with a sensible architecture, but the flagship 'PhD-level cosmology' claim is not yet backed by numbers—just plausible-looking plots. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the named Planning & Control strategy. In the planning phase, a planner agent proposes a step-by-step plan (sub-task, actions, responsible agent) and a plan reviewer critiques it, with this proposal-review loop repeated a configurable number of times until the plan is approved. In the control phase, a controller assigns each step to a researcher agent (interpretation and writing) or an engineer agent (code), where the engineer's code is formatted and executed locally by an executor; failed runs route back to the engineer with interpreter feedback, to a package installer if a dependency is missing, or to a terminator if retries are exhausted. Memory across steps is kept by injecting the previous step's final code and outputs into the next step's system message, while agents and chat history are reset each step, roughly halving session cost. Domain knowledge is supplied by context agents carrying full documentation of standard cosmology libraries and by retrieval-augmented agents that query vector embeddings of documentation or paper corpora, and planning yields a saved JSON plan with a recorded dollar cost for traceability.
What would settle it
Re-run the exact Union2.1 task with cmbagent, extract the full MCMC chains, and check the recovered $H_0$ and $\Omega_\Lambda$ posteriors against an independent published analysis of the same data; the automation claim collapses if the chains do not converge, disagree with the published mean and uncertainty by more than the reported error bars, or if inspection of the session log shows the final plots were not produced by the code the plan describes.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a multi-agent architecture with a deliberate separation between planning and execution can automate a complete research analysis—including downloading data, writing and running Python code, iteratively fixing failed executions, performing MCMC sampling, and summarising findings—without any human review step. The load-bearing demonstration is the cosmology task: given only a URL, a short description of the Union2.1 supernova data, and a request to fit a flat LCDM model with two free parameters, the system generated a six-step plan, executed each step locally using eight threads, produced distance-modulus and posterior-contour figures, and commented on the results. The authors claim the task was solved the first time it was run, and treat this as evidence that current LLMs, orchestrated correctly, can carry out PhD-level inference work autonomously rather than merely assist a human analyst.
Load-bearing premise
The cosmology demonstration is judged successful from plausible-looking plots, not from a quantitative check: the paper never gives the recovered parameter values, error bars, or a comparison to published Union2.1 constraints, so the claim that the system 'solved' a PhD-level task assumes that correct-looking output means correct inference.
Editorial extensions
If this is right
- If the cosmology result is right, a fully autonomous system can reproduce a standard research workflow—parameter inference via MCMC plus written interpretation—that previously required a human in the loop at every step.
- The DS-1000 improvement (66% to 78% on the evaluated subset) suggests the planning-and-control orchestration itself, independent of any single model, raises reliability on data-science coding tasks.
- Context agents that embed up-to-date library documentation can solve cosmology coding problems (e.g., CMB delensing efficiency) that three frontier LLMs fail systematically, pointing to documentation-in-context as a cheap substitute for fine-tuning.
- Because the system records plans, costs, execution logs, and videos, the same task can be re-run and inspected, making the claimed automation reproducible in principle.
Reading between the lines
- The reported gains rest on a 50-problem benchmark subset and on 14 cosmology problems, so extending the 'superior performance' claim to arbitrary research tasks would require running the same protocols on much larger and more varied problem sets.
- Because the cosmology demonstration is documented with plots and a full session log but without quoted posterior values, a natural next step is a quantitative re-analysis reporting $H_0$ and $\Omega_\Lambda$ with error bars alongside published constraints.
- If the approach generalises, the bottleneck for autonomous discovery shifts from executing analyses to judging their scientific meaning; human reviewers would focus on question formation and verification rather than step-by-step code.
- The context-reset technique—clearing chat history between steps while re-injecting only final outputs—is a cost-control idea that could transfer to other long-horizon agent tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents cmbagent, an open-source multi-agent system built on AG2 that uses a Planning & Control strategy to automate scientific research tasks without step-by-step human intervention. The system comprises about 30 LLM agents with specialized roles (planner, plan reviewer, controller, researcher, engineer, interpreter, installer, terminator, context agents for camb/class, RAG agents). The authors claim that cmbagent can execute code locally and successfully complete a 'PhD level' cosmology task—measuring cosmological parameters (H0, Omega_L) from Union2.1 supernova data via MCMC—with no human-in-the-loop. They also report superior performance over state-of-the-art LLMs on a 14-problem self-designed camb benchmark and on a 50-problem subset of DS-1000. The paper includes an appendix with an automatically generated 'research paper' produced by the integrated denario system.
Significance. If the central claims hold, cmbagent would be a notable open-source contribution to automated scientific discovery, combining code execution, retrieval, and multi-agent planning in a reproducible GitHub-distributed package. The camb context agent's documented approach of feeding auto-generated Read the Docs documentation into the LLM context is a practical and potentially transferable technique, and the DS-1000 evaluation provides an external, though small, benchmark. However, the paper's flagship cosmology demonstration is currently supported only by qualitative plots without quantitative validation, and the claimed 'superior performance over state-of-the-art LLMs' is not consistently backed by the reported experiments. The automatically generated appendix paper itself contains N/A metrics and malformed citations, which undercuts the broader 'research-quality papers' claim made in the Discussion. These gaps are load-bearing for the paper's central message, so the manuscript requires meaningful revision rather than minor polishing.
major comments (4)
- [Sec. 3.3 and Appendix C] The cosmology task's 'success' is not quantitatively validated. The paper does not quote the recovered posterior means and 1-sigma uncertainties for H0 and Omega_L, reports no convergence diagnostics (e.g., R-hat, effective sample size, acceptance rate), and does not compare the results to published Union2.1 constraints or other measurements. The plots in Figs. 7 and 8 cannot distinguish a converged, correct inference from a biased MCMC or a likelihood coding error. Since this single run is the principal evidence for the abstract's flagship claim of carrying out a 'PhD level cosmology task', please add the quantitative posterior summary and diagnostics from the saved chains, and include a comparison to known constraints.
- [Sec. 3.2 and Table 1] The abstract claims 'superior performance over state-of-the-art LLMs' on 'two benchmark sets', but Table 1 only compares cmbagent's One Shot mode with its Planning & Control mode on the 50-problem DS-1000 subset. No external LLM baselines are reported for DS-1000, so the 'superior' claim is supported only by the 14-problem camb benchmark. Please either add SOTA LLM baselines to the DS-1000 table or qualify the abstract's claim to reflect which benchmark includes such comparisons.
- [Sec. 3.1 and Fig. 2] The camb benchmark consists of 14 self-designed problems, with success rates computed over 10 runs per agent. The paper does not specify the scoring protocol (what counts as a success, whether numerical tolerances are used, how partial credit is handled), and the problem set is only available via an external link. To support the 'superior performance' claim, please provide the full problem set and scoring rubric, and report the per-problem success counts so the reader can assess the significance of the context agent's advantage given the small N.
- [Appendix E and Sec. 3.4] The automatically generated denario paper contains 'N/A' entries for the non-linear-regime median |ΔP/P| and for the lowest redshift bin, and contains malformed citations (e.g., 'Silva et al. 2024,?'). The Discussion's statement that 'We are now even able to write research-quality papers automatically' is directly contradicted by this appendix. Please either remove or substantially qualify the 'research-quality' claim, or replace the example with a corrected paper that has complete metrics and valid references.
minor comments (5)
- [Sec. 4] There is a typo in the Discussion: 'also offering offers One Shot' should read 'also offering One Shot'.
- [Fig. 2 caption] The caption uses 'CAMB context' while the text refers to the 'camb agent'; please make the terminology consistent throughout.
- [Sec. 2.2] The class context document is given only as a URL; consider archiving it in a persistent repository or including a version-stamped copy so the evaluation is reproducible.
- [Sec. 2.1 and Appendix C] The plan in Appendix C instructs the engineer to benchmark different MCMC implementations, but the paper does not report which sampler was selected or the timing results; including this detail would strengthen the reproducibility of the cosmology run.
- [Abstract and Sec. 2.1] The phrase 'no human-in-the-loop at any point' is strong given that the user supplies the Main Task and sets hyperparameters (nsteps, nreviews, nfails); please clarify that this refers to the absence of step-by-step human guidance, not to the absence of user-specified inputs.
Circularity Check
No significant circularity; the benchmark evaluations use external or externally checkable targets, and the cosmology demonstration is not derived from its own outputs.
full rationale
The paper's central claims are evaluated against targets that are independent of cmbagent's own outputs. DS-1000 (Lai et al. 2022) is a published external benchmark, and the camb evaluation (Sec. 3.1) scores code against objective properties of the camb library, whose outputs are computed by the library itself rather than by the LLM being tested. The cosmology task (Sec. 3.3, Appendix C) asks the system to infer H0 and Omega_L from the public Union2.1 dataset; final success is not defined by any parameter fitted by the paper, and the plots, while not quantitatively validated, are generated from the supernova data and the model, not from the claim being made. The few self-citations (Laverick et al. 2024; Villaescusa-Navarro et al. 2025) describe related prior or ongoing work and are not used to justify the benchmark results. The manuscript's weaknesses, such as unquoted posterior values, missing convergence diagnostics, and a single unvalidated cosmology run, are evidence-quality gaps rather than circular derivations. No step reduces a supposed prediction to a fitted input or to a self-citation chain.
Assumptions & free parameters
free parameters (2)
- hyperparameters nsteps, nreviews, nfails =
user-specified, defaults not quoted
- LLM backend choice =
gpt-4.1 for DS-1000 and cosmology, gemini-2.5-pro for the camb context agent
assumptions (4)
- domain assumption The LLM backends (gpt-4.1, gemini-2.5-pro) can reliably write correct code on the first or retry attempt.
- ad hoc to paper The plan reviewer LLM provides non-trivial feedback that improves the plan.
- domain assumption The flat Lambda-CDM model and Union2.1 dataset are appropriate for the demo task.
- domain assumption Success in the benchmarks is correctly scored.
Cite this review
Pith. "Pith review of Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery." pith.science (2026). https://pith.science/paper/UA64EWLX
@misc{pith2026250707257,
author = {Pith},
title = {Pith review of: Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery},
year = {2026},
howpublished = {\url{https://pith.science/paper/UA64EWLX}},
note = {Machine review of arXiv:2507.07257}
}
read the original abstract
We present a multi-agent system for automation of scientific research tasks, cmbagent (https://github.com/CMBAgents/cmbagent). The system is formed by about 30 Large Language Model (LLM) agents and implements a Planning & Control strategy to orchestrate the agentic workflow, with no human-in-the-loop at any point. Each agent specializes in a different task (performing retrieval on scientific papers and codebases, writing code, interpreting results, critiquing the output of other agents) and the system is able to execute code locally. We successfully apply cmbagent to carry out a PhD level cosmology task (the measurement of cosmological parameters using supernova data) and evaluate its performance on two benchmark sets, finding superior performance over state-of-the-art LLMs. The source code is available on GitHub, demonstration videos are also available, and the system is deployed on HuggingFace and will be available on the cloud.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 2 Pith papers
-
Agentic Exploration of Physics Models
A general-purpose LLM agent can discover physics models, including ODEs and spin Hamiltonians, by autonomously choosing experiments and fitting hypotheses to numeric data.
-
DeepInflation: an AI agent for research and model discovery of inflation
An LLM agent with symbolic regression finds simple inflation potentials that match target CMB observables, but the outputs are fitted to the targets rather than independently predicted.
Reference graph
Works this paper leans on
-
[1]
Thomas, S. A. 2012, PkANN - I. Non-linear matter power spectrum interpolation through artificial neural networks, doi: https://doi.org/10.1111/j.1365-2966.2012.21326.x
arXiv 2012
-
[2]
2010, Error estimation in astronomy: A guide
Andrae, R. 2010, Error estimation in astronomy: A guide. https://arxiv.org/abs/1009.2755
arXiv 2010
-
[3]
Andreon, S., & Hurn, M. A. 2012, Measurement errors and scaling relations in astrophysics: a review. https://arxiv.org/abs/1210.6232
work page Pith review arXiv 2012
-
[4]
Bevins, H. T. J., Gessey-Jones, T., & Handley, W. J. 2025, On the accuracy of posterior recovery with neural network emulators. https://arxiv.org/abs/2503.13263
arXiv 2025
-
[5]
Bourilkov, D. 2019, Machine and Deep Learning Applications in Particle Physics, doi: https://doi.org/10.1142/S0217751X19300199
-
[6]
Cagliari, M. S., Dizgah, A. M., & Villaescusa-Navarro, F. 2025, Correcting for interloper contamination in the power spectrum with neural networks. https://arxiv.org/abs/2504.06919
arXiv 2025
-
[7]
Campeti, P., Delouis, J. M., Pagano, L., et al. 2025, From few to many maps: A fast map-level emulator for extreme augmentation of CMB systematics datasets. https://arxiv.org/abs/2503.11643
work page Pith review arXiv 2025
-
[8]
Chantada, A. T., Landau, S. J., Protopapas, P., Scóccola, C. G., & Garraffo, C. 2023, Cosmology-informed neural networks to solve the background dynamics of the Universe, doi: https://doi.org/10.1103/PhysRevD.107.063523
Show all 38 references
-
[9]
2024, Inferring IGM parameters from the redshifted 21-cm Power Spectrum using Artificial Neural Networks
Choudhury, M., Ghara, R., Zaroubi, S., et al. 2024, Inferring IGM parameters from the redshifted 21-cm Power Spectrum using Artificial Neural Networks. https://arxiv.org/abs/2407.03523 Conceição, M., Krone-Martins, A., da Silva, A., & Ángeles Moliné. 2023, Fast emulation of co...
2024 arXiv
-
[10]
2020, Lightning-Fast Gravitational Wave Parameter Inference through Neural Amortization
Delaunoy, A., Wehenkel, A., Hinderer, T., et al. 2020, Lightning-Fast Gravitational Wave Parameter Inference through Neural Amortization. https://arxiv.org/abs/2010.12931
2020 arXiv
-
[11]
2021, Neural Network Acceleration of Large-scale Structure Theory Calculations, doi: https://doi.org/10.1088/1475-7516/2022/04/056
DeRose, J., Chen, S.-F., White, M., & Kokron, N. 2021, Neural Network Acceleration of Large-scale Structure Theory Calculations, doi: https://doi.org/10.1088/1475-7516/2022/04/056
2021 doi
-
[12]
Adami, K. Z. 2022, Neural Network Reconstruction of Late-Time Cosmology and Null Tests, doi: https://doi.org/10.1088/1475-7516/2022/02/023
2022 doi
-
[13]
2025, Neural network emulation of reionization to constrain new physics with early- and late-time probes
Facchinetti, G. 2025, Neural network emulation of reionization to constrain new physics with early- and late-time probes. https://arxiv.org/abs/2503.11261
2025 arXiv
-
[14]
Fremstad, D., & Winther, H. A. 2025, Emulating the Non-Linear Matter Power-Spectrum in Mixed Axion Dark Matter Models. https://arxiv.org/abs/2503.07277
2025
-
[15]
Greiner, M., & Enßlin, T. A. 2014, Log-transforming the matter power spectrum, doi: https://doi.org/10.1051/0004-6361/201323181
2014 doi
-
[16]
P., Bechtol, K., Carlin, J
Guy, L. P., Bechtol, K., Carlin, J. L., et al. 2022, Faro: A framework for measuring the scientific performance of petascale Rubin Observatory data products. https://arxiv.org/abs/2206.15447 Gómez-Vargas, I., Andrade, J. B., & Vázquez, J. A. 2023, Neural Networks Optimized by ...
2022 arXiv
-
[17]
Huber, S., & Suyu, S. H. 2024, HOLISMOKES – XII. Time-delay Measurements of Strongly Lensed Type Ia Supernovae using a Long Short-Term Memory Network, doi: https://doi.org/10.1051/0004-6361/202449952
2024 doi
-
[18]
2023, Constructing Impactful Machine Learning Research for Astronomy: Best Practices for Researchers and Reviewers
Huppenkothen, D., Ntampaka, M., Ho, M., et al. 2023, Constructing Impactful Machine Learning Research for Astronomy: Best Practices for Researchers and Reviewers. https://arxiv.org/abs/2310.12528
2023 arXiv
-
[19]
2018, Neural network-based preprocessing to estimate the parameters of the X-ray emission of a single-temperature thermal plasma, doi: https://doi.org/10.1093/mnras/sty161
Ichinohe, Y., Yamada, S., Miyazaki, N., & Saito, S. 2018, Neural network-based preprocessing to estimate the parameters of the X-ray emission of a single-temperature thermal plasma, doi: https://doi.org/10.1093/mnras/sty161
2018 doi
-
[20]
Spergel, D. N. 2024, Field-level Emulation of Cosmic Structure Formation with Cosmology and Redshift Dependence. https://arxiv.org/abs/2408.07699
2024 arXiv
-
[21]
T., Harrison, I., Calabrese, E., et al
Jense, H. T., Harrison, I., Calabrese, E., et al. 2024, A complete framework for cosmological emulation and inference with CosmoPower. https://arxiv.org/abs/2405.07903
2024 arXiv
-
[22]
Burns, J. O. 2024, 21cmLSTM: A Fast Memory-based Emulator of the Global 21 cm Signal with Unprecedented Accuracy. https://arxiv.org/abs/2410.07619 10
2024 arXiv
-
[23]
Kelly, B. C. 2011, Measurement Error Models in Astronomy. https://arxiv.org/abs/1112.1745
2011 arXiv
-
[24]
2024, Machine Learning-Based Identification of Contaminated Images in Light Curves Data Preprocessing, doi: https://doi.org/10.1088/1674-4527/ad339e
Li, H., Li, R.-W., Shu, P., & Li, Y.-Q. 2024, Machine Learning-Based Identification of Contaminated Images in Light Curves Data Preprocessing, doi: https://doi.org/10.1088/1674-4527/ad339e
2024 doi
-
[25]
I., Dai, M., Ponder, K
Malz, A. I., Dai, M., Ponder, K. A., et al. 2023, Are classification metrics good proxies for SN Ia cosmological constraining power?, doi: https://doi.org/10.1051/0004-6361/202346891
2023 doi
-
[26]
P., Santa, C., & Romano, A
Marulanda, J. P., Santa, C., & Romano, A. E. 2020, Deep learning merger masses estimation from gravitational waves signals in the frequency domain, doi: https://doi.org/10.1016/j.physletb.2020.135790
2020
-
[27]
T., et al
More, A., Canameras, R., Jaelani, A. T., et al. 2024, Systematic comparison of neural networks used in discovering strong gravitational lenses. https://arxiv.org/abs/2405.12975
2024 arXiv
-
[28]
2021, Cosmological parameters from the likelihood analysis of the galaxy power spectrum and bispectrum in real space, doi: https://doi.org/10.1088/1475-7516/2021/11/038
Oddo, A., Rizzo, F., Sefusatti, E., Porciani, C., & Monaco, P. 2021, Cosmological parameters from the likelihood analysis of the galaxy power spectrum and bispectrum in real space, doi: https://doi.org/10.1088/1475-7516/2021/11/038
2021 doi
-
[29]
McEwen, J. D. 2024, Learned harmonic mean estimation of the Bayesian evidence with normalizing flows. https://arxiv.org/abs/2405.05969
2024
-
[30]
Wandelt, B. D. 2020, Super-resolution emulator of cosmological simulations using deep physical models, doi: https://doi.org/10.1093/mnras/staa1428
2020 doi
-
[31]
2024, CircleZ: Reliable Photometric redshifts for AGN computed using only photometry from Legacy Survey Imaging for DESI
Saxena, A., Salvato, M., Roster, W., et al. 2024, CircleZ: Reliable Photometric redshifts for AGN computed using only photometry from Legacy Survey Imaging for DESI. https://arxiv.org/abs/2407.10788
2024 arXiv
-
[32]
C., & Valentino, E
Silva, E., Zúñiga-Bolaño, U., Nunes, R. C., & Valentino, E. D. 2024, Non-Linear Matter Power Spectrum Modeling in Interacting Dark Energy Cosmologies, doi: https://doi.org/10.1140/epjc/s10052-024-13487-x Sáez-Casares, I., Rasera, Y., & Li, B. 2023, The e-MANTIS emulator: fast ...
2024 doi
-
[33]
2006, Cosmological Constraints from the SDSS Luminous Red Galaxies, doi: https://doi.org/10.1103/PhysRevD.74.123507
Tegmark, M., Eisenstein, D., Strauss, M., et al. 2006, Cosmological Constraints from the SDSS Luminous Red Galaxies, doi: https://doi.org/10.1103/PhysRevD.74.123507
2006 doi
-
[34]
2025, Neural Network-based model of galaxy power spectrum: Fast full-shape galaxy power spectrum analysis
Trusov, S., Zarrouk, P., & Cole, S. 2025, Neural Network-based model of galaxy power spectrum: Fast full-shape galaxy power spectrum analysis. https://arxiv.org/abs/2403.20093
2025 arXiv
-
[35]
2011, The Non-Linear Matter Power Spectrum in Warm Dark Matter Cosmologies, doi: https://doi.org/10.1111/j.1365-2966.2011.19910.x
Viel, M., Markovic, K., Baldi, M., & Weller, J. 2011, The Non-Linear Matter Power Spectrum in Warm Dark Matter Cosmologies, doi: https://doi.org/10.1111/j.1365-2966.2011.19910.x
2011
-
[36]
A., & Speagle, J
Wang, B., Leja, J., Villar, V. A., & Speagle, J. S. 2023, SBI++: Flexible, Ultra-fast Likelihood-free Inference Customized for Astronomical Applications, doi: https://doi.org/10.3847/2041-8213/ace361
2023 doi
-
[37]
2019, Emulators for the non-linear matter power spectrum beyond ΛCDM, doi: https://doi.org/10.1103/PhysRevD.100.123540
Winther, H., Casas, S., Baldi, M., et al. 2019, Emulators for the non-linear matter power spectrum beyond ΛCDM, doi: https://doi.org/10.1103/PhysRevD.100.123540
2019 doi
-
[38]
2025, Ten-parameter simulation suite for cosmological emulation beyond ΛCDM
Yang, Y., Bird, S., & Ho, M.-F. 2025, Ten-parameter simulation suite for cosmological emulation beyond ΛCDM. https://arxiv.org/abs/2501.06296 Álvarez, S. I., Alonso, E. D., Sánchez, M. L., et al. 2023, One-dimensional Convolutional Neural Networks for Detecting Transiting Exop...
2025 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.