{"id":"49889f33-dbc4-447a-892d-1eaf8b5a308f","arxiv_id":"2412.00431","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent LLM system with retrieval and local code execution reproduced ACT DR6 lensing cosmological parameter constraints without human-written code, and generalized to two research-software tasks.","lead":"This paper describes a team of AI agents that write and run code to analyze cosmological data, and shows it reproducing published measurements of dark matter from the Atacama Cosmology Telescope. It matters because it is a concrete early test of whether large language model agents can automate real scientific analysis pipelines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-session success on the development task, with authors explicitly disclaiming generalization; the concrete bottleneck is the absence of repeated independent runs quantifying agreement between cmbagent and reference chains.","rationale":"The reader's weakest assumption identified exactly the load-bearing issue: the main demonstration is on the development task and the system was heavily prompt-engineered, so the result does not establish general capability. I agree. My stress-test pass did not find a different, sharper technical flaw. The paper is unusually honest: Section 2 says 'finding instructions that yielded a good behavior for our tasks required a significant amount of prompt engineering'; Section 3.1 explicitly says the main task is 'the same task that we used to develop our system' and is 'not a demonstration of the usefulness of the system beyond that specific task'; Section 4 flags the lack of a benchmarking framework and the risk of LLM over-confidence. The generalization tasks use classy_sz and cosmocnc, which are developed by the authors themselves, but that is not a correctness flaw given the paper's stated scope. The most precise concrete weakness is that the central numeric claim (reproduction of ACT DR6 contours) rests on a single session with a qualitative posterior comparison and no quantitative chain-comparison metric; the residual difference is attributed to emulator precision settings without a shown check. Because the authors already scope the claim as a proof of concept, the correct verdict is unchanged: conditional acceptance as work-in-progress, with the condition that broader claims await a benchmark with repeated runs and quantitative metrics. If the paper instead claimed general agentic capability, this concern would justify a reject or strong condition, but as written the claims match the evidence. I would keep the CONDITIONAL verdict, asking for repeated runs and quantitative comparisons before stronger claims.","tokens_in":11832,"tokens_out":1689,"duration_ms":15553,"concrete_test":"Run the same ACT DR6 main task at least 5 times from clean sessions with identical agent instructions, RAG databases, and task phrasing, recording success/failure and the GetDist parameter difference (e.g., maximum posterior shift and KL divergence) between cmbagent chains and the official ACT chains. If all runs converge and the parameter shifts stay within the MCMC noise, the single-session concern is mitigated; if some runs fail or diverge, the paper's claim of reproduction should be reported as a success rate rather than a single demonstration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the cmbagent MAS reproduced ACT DR6 lensing constraints. The authors openly state in Section 3.1 that this is the same task used to develop the system, so it cannot demonstrate general capability; the paper also acknowledges the statistical comparison is qualitative ('almost perfectly', 'statistically insignificant differences') and attributes residual differences to emulator precision settings. What would make the claim load-bearing for general capability is the implied reliability of the agentic pipeline for new tasks. Section 3.2 reports two generalization demonstrations, but both use research software authored by members of the same team, and neither is compared quantitatively against an independent reference. A single successful session, on a task for which the prompts, RAG content, and memory were engineered, does not constrain the failure rate of the pipeline. This is not an internal inconsistency; the paper explicitly frames itself as a work-in-progress proof of concept. The weakest assumption is therefore that one successful reproduction, with qualitative posterior overlap, supports the broader claim that the MAS can automate and optimize scientific workflows in cosmology. Since the authors themselves caution that current LLMs produce over-confident, physically incorrect responses, the missing evidence is repeated trials on held-out tasks with quantitative agreement metrics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents cmbagent, a multi-agent LLM system built on the autogen/ag2 framework, in which RAG agents, coder agents, planner/manager agents, and a human 'admin' interact to retrieve documentation, write Python code, execute it, and iterate. The main demonstration is a single end-to-end session in which the system wrote and ran a cobaya MCMC analysis of the ACT DR6 CMB lensing likelihood using classy_sz as the theory code, obtaining posterior contours that the authors state overlap the ACT collaboration chains. Two additional 'generalization' demonstrations use classy_sz to compute fEDE power spectra and cosmocnc to evaluate a cluster-count likelihood as a function of mass bias. The paper emphasizes its work-in-progress status, discusses cost and token usage, and releases code at GitHub. The authors explicitly note that the main task was the same task used to develop the system and that the current system relies on human feedback at every step.","tokens_in":12020,"tokens_out":6296,"duration_ms":63466,"significance":"The paper's value, if the claims are supported, is as an early proof-of-concept that a carefully orchestrated multi-agent LLM system can assemble and execute a real cosmological inference pipeline under human supervision. The strengths are concrete: the system is open source, the main task is a genuine state-of-the-art likelihood, session transcripts are linked, and the authors are candid about limitations such as the risk of LLMs producing plausible-looking but physically incorrect responses. However, because the headline result rests on a single session and a qualitative contour comparison, the current evidence is anecdotal; it does not establish reliability or automation in a measurable sense. The generalization tests are also not benchmarked against independent references, so their evidentiary weight is limited.","major_comments":[{"comment":"The claim that the cmbagent contours 'overlap almost perfectly' and differ by 'statistically insignificant' amounts is not supported by any quantitative comparison. I request a table or text reporting, for each cosmological parameter sampled by both chains, the posterior mean, the 68% credible interval, and the difference in units of the combined uncertainty, or an equivalent sample-based distance statistic. Without such a metric, a reader cannot judge whether the reproduction is at the precision claimed, particularly because the paper attributes the residual difference to emulator precision settings; that explanation should be tested by quantifying the emulator error or by recomputing one chain with matched precision settings.","section":"Section 3.1, Figure 2"},{"comment":"The main demonstration is a single session on the very task used to develop the system. The manuscript states in Section 2.3 that 'finding instructions that yielded a good behavior ... required a significant amount of prompt engineering' and in Section 3.1 that the result 'is not a demonstration of the usefulness of the system beyond that specific task.' A single successful in-sample run does not constrain the failure rate of the pipeline, which is the quantity needed to assess any automation claim. Please report repeated independent runs of the main task, including the number of runs that reached convergence, the number that required human intervention, and the scatter in the resulting parameter constraints.","section":"Section 2.3 and Section 3.1"},{"comment":"The two generalization demonstrations are not validated quantitatively against independent references. For the fEDE spectra in Figure 4, no reference spectra or accuracy metric is shown, so the reader cannot tell whether the code was correct or merely plausible. For the cosmocnc likelihood in Figure 3 and Appendix A, the only check is that the peak lies near the input value bias_sz = 0.8; no comparison with a human-generated likelihood curve, no numerical value of the peak, and no tolerance is given. In addition, both software packages are authored by members of the same team, so these tests do not demonstrate the ability to learn unfamiliar external software. I request at least one held-out task with an independent, pre-existing pipeline as ground truth, with a quantitative agreement metric.","section":"Section 3.2 and Appendix A"},{"comment":"The paper does not state whether the memory RAG agent's database contained session summaries from earlier development runs when the reported main-task session was executed. Because the memory agent is designed to draw on 'past tasks' and 'past material,' a pre-populated memory could make the reported success partially a retrieval result rather than a fresh agentic solution. Please document the memory database state at the start of each reported session and specify which memory entries, if any, were retrieved during the main-task and generalization runs.","section":"Section 2.2, memory RAG agent"}],"minor_comments":[{"comment":"The word 'etrieval-augmented' should be 'retrieval-augmented'.","section":"Abstract"},{"comment":"The phrase 'we emphasize to our strategy' should be 'we emphasize that our strategy'.","section":"Section 1"},{"comment":"The convergence diagnostic is named 'Gellman-Rubin'; this should be 'Gelman-Rubin'.","section":"Section 2.3"},{"comment":"The title 'Simons Obsevatory' should be 'Simons Observatory'.","section":"Figure 3"},{"comment":"The manuscript would benefit from specifying the exact software versions and commit hashes of cmbagent, cobaya, classy_sz, and cosmocnc, as well as the OpenAI model snapshot used, for each reported session; the current text only names gpt-4o-2024-05-13.","section":"Section 2.3 and Appendix A"},{"comment":"Setting temperature and TopP to small values is described as a way to make the system 'as deterministic and controllable as possible,' but these parameters do not guarantee determinism; consider rewording to 'low stochasticity' and reporting a test of run-to-run variability.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is honest about its limitations and the code release is a useful community resource. My main concern is evidential rather than ethical: the headline claim of reproducing an ACT DR6 analysis is supported by one session and a visual comparison, and the generalization tests are not independently benchmarked. This is a borderline manuscript; it could be a worthwhile proof-of-concept for an astro-im/methods audience after the quantitative evidence is added. I would not recommend accepting it as a demonstration of automated scientific workflow capability in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you need to know: this is a work-in-progress paper that actually ships code and does one impressive thing. In a single session, their cmbagent multi-agent system wrote and executed a full cobaya MCMC for the ACT DR6 lensing likelihood, on a laptop, in about 40 minutes and for $1.55, and produced contours that visually overlap the collaboration chains. No human-written code. That is a real result, and the architecture — planner, admin, engineer, executor, with RAG agents for experiments and software plus a memory agent — is a reasonable first template for this kind of system.\n\nCredit where it is due: the paper is strikingly honest about its limits. It says plainly in Section 3.1 that the main task is the same one used to develop the system, so it is not a demonstration of usefulness beyond that task. It admits that a significant amount of prompt engineering was required. And the limitations section warns that current LLMs produce over-confident, physically incorrect responses and that a benchmarking framework is the most urgent need. That is the right tone, and it matches the evidence.\n\nThe soft spots are exactly what the authors concede. The central claim is supported by one run, on the development task, with a qualitative contour comparison rather than a quantitative distance metric or repeated trials. The two generalization tasks use research software from the same group and are not compared against independent reference results. So the paper does not establish a general capability; it is an existence proof. The stress-test note is on target: without repeated runs on held-out tasks with agreement metrics, the failure rate of the pipeline is unconstrained. I do not see equation-level circularity or a fabricated claim — the benchmark is external ACT chains, and the authors explicitly disclaim generalization.\n\nWho should read this: anyone building LLM agents for quantitative physics analysis, and cosmologists who want to see whether agentic pipelines can actually run real likelihood code. As a paper, it reads like a solid work-in-progress report rather than a finished capability claim. If I were refereeing, I would ask for repeated sessions across seeds, a quantitative chain comparison (e.g., parameter shifts or a distance metric), and at least one held-out task untouched during development. Those are standard, achievable additions.\n\nIt deserves peer review as a work-in-progress tool paper, not a desk reject. With those additions it could become a genuinely useful reference for the community.","headline":"A transparent proof-of-concept that an LLM multi-agent system can reproduce an ACT DR6 lensing MCMC without human-written code, but the evidence rests on a single development-task session with qualitative comparison.","tokens_in":12591,"tokens_out":2113,"would_cite":true,"duration_ms":23888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent LLM system wrote and executed a full cosmological MCMC analysis without any human-written code, reproducing published ACT DR6 lensing constraints.","keywords":["multi-agent systems","large language models","retrieval-augmented generation","scientific workflow automation","Markov Chain Monte Carlo","cosmological parameter estimation","ACT DR6 lensing","research software"],"falsifier":"A decisive test would be to run cmbagent on a held-out cosmological analysis (for example, parameter estimation with the Planck PR4 lensing likelihood and a different theory code) and count how many independent sessions produce converged chains and correct contours with no human-written code; the central claim of general automation is false if most sessions fail or require substantial human code fixes.","tokens_in":11622,"feed_emoji":"🤖","tokens_out":5492,"duration_ms":48200,"temperature":0.7,"pith_summary":"The paper reports that a multi-agent system built from large language models, retrieval-augmented generation, and locally executed code can take over an entire state-of-the-art cosmological analysis without any human-written code. The central demonstration is a full Markov Chain Monte Carlo parameter estimation of the ACT DR6 CMB lensing likelihood that converged in about forty minutes, cost $1.55, and matched the published contours. The authors argue this points toward a more general strategy for automating quantitative scientific workflows, and they show preliminary evidence that the system also handles research software it was not developed on. If this is right, the hard part of reproducing or redoing a cosmological analysis—assembling documentation, writing configuration files, and executing the pipeline—could shrink from hours or days to minutes, and the same architecture could transfer to other quantitative sciences.","feed_headline":"AI agents ran a cosmology analysis solo, matching published results","feed_subtitle":"A multi-agent LLM system wrote and executed the full MCMC pipeline in 40 minutes for $1.55.","key_machinery":"The system is a multi-agent architecture with three agent types: manager agents (planner, chat manager, admin) that split a task into steps and control transitions; coder agents (engineer and executor) that write and run Python code; and RAG agents that retrieve context from vectorized databases of papers, tutorials, and past session summaries. A memory agent stores summaries of past sessions to avoid repeating mistakes. Controllability comes from low temperature and TopP settings, restricted allowed transitions, and mandatory human feedback at each step. The central mechanism is the planner-driven loop: plan, retrieve, write, execute, check, iterate—with a human only approving or redirecting.","core_discovery":"Within a single session, cmbagent wrote and executed the full MCMC analysis for the ACT DR6 CMB lensing likelihood, with no human-written code, and reproduced the results presented in Madhavacheril et al. (2024). The contours overlap almost perfectly, with statistically insignificant differences, and the full analysis took about 40 minutes at a cost of $1.55. The authors also report successful results on two tasks outside the core development: computing CMB power spectra for values of the parameter fEDE in classy_sz, and evaluating a cosmocnc galaxy cluster number-count likelihood as a function of mass bias. These successes are presented as evidence that the multi-agent architecture can generalize beyond the specific pipeline it was built around.","pith_inferences":["If the architecture transfers, agentic systems could serve as low-cost analysis apprentices that draft complete pipelines for junior researchers, with a senior human validating each step.","The strongest test of general capability would be running cmbagent on a likelihood or dataset never seen during development, and measuring success rate over many independent runs, since LLM behavior is stochastic even at low temperature.","A natural next benchmark is to let the system propose its own pipeline changes and measure whether it finds a faster or higher-fidelity configuration than the human baseline.","The memory agent's accumulation of past session summaries could be studied as a within-system learning signal: comparing first-run versus later-run token counts and error rates on similar tasks would quantify how much experience the system actually retains."],"forward_implications":["A full cosmological parameter analysis can in principle be handed to an agentic system that writes the configuration, runs the MCMC, and produces posterior contours without human-written code.","Reproducing a published likelihood analysis becomes a session of about forty minutes and roughly $1.50 in API cost on a laptop, rather than days of manual code assembly.","The same planner/RAG/coder architecture can be pointed at research software with sparse documentation (classy_sz, cosmocnc) and still produce working code for new parameters and likelihood evaluations.","Because the system logs plans and session summaries, an entire analysis becomes an auditable record of prompts, code, and outputs, useful as a cross-check on human pipelines.","The cost and token counts are small enough that repeated agentic runs are feasible for exploratory work, not just one-off demonstrations."],"supporting_citations":[{"why":"Supplies the open-source multi-agent programming framework that cmbagent is built on.","marker":"Wu et al., 2023"},{"why":"Provides the ACT DR6 lensing map and cosmological parameter results that the main task reproduces.","marker":"Madhavacheril et al., 2024"},{"why":"Provides the ACT DR6 lensing power spectrum likelihood and data used in the main task.","marker":"Qu et al., 2024"},{"why":"Provides cobaya, the MCMC sampling framework the agent uses for parameter inference.","marker":"Torrado and Lewis, 2021"},{"why":"Provides camb, the Boltzmann solver used as the theory code in the original analysis.","marker":"Lewis et al., 2000"},{"why":"Provides GetDist, used to convert MCMC samples into posterior contours.","marker":"Lewis, 2019"},{"why":"Provides classy_sz, a research code the agent handles in a generalization task.","marker":"Bolliet et al., 2024"},{"why":"Provides cosmocnc, a research code the agent handles in a second generalization task.","marker":"Zubeldia et al., 2024"},{"why":"Supplies retrieval-augmented generation, the method used to ground agents in documentation.","marker":"Lewis et al., 2020"}],"fun_headline_variants":["AI agents auto-run cosmology MCMC, match paper","Multi-agent AI reproduces ACT DR6 results for $1.55","LLM agents write and run full cosmology analysis","Cosmology results replicated by AI in 40 minutes","Autonomous AI agents finish cosmology pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of general usefulness rests on the assumption that the main task's success is not just the result of instructions, retrieval content, and human feedback tuned to that exact benchmark during development, since the paper itself notes that the main task is the same one used to develop the system.","fun_headline_variants_meta":{"raw":{"variants":["AI agents auto-run cosmology MCMC, match paper","Multi-agent AI reproduces ACT DR6 results for $1.55","LLM agents write and run full cosmology analysis","Cosmology results replicated by AI in 40 minutes","Autonomous AI agents finish cosmology pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1450,"prompt_tokens":811,"completion_tokens":639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":576}},"tokens_in":427,"tokens_out":639,"duration_ms":6161,"temperature":1.0,"reasoning_tokens":576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:22:55.912823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would be to run cmbagent on a held-out cosmological analysis (for example, parameter estimation with the Planck PR4 lensing likelihood and a different theory code) and count how many independent sessions produce converged chains and correct contours with no human-written code; the central claim of general automation is false if most sessions fail or require substantial human code fixes.","supporting_citations":[],"review_version":1}