{"id":"bac5a11e-83c2-4331-b3cf-91962f913d58","arxiv_id":"2412.17149","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-driven multi-agent system iteratively modifies agentic AI workflows, and the paper reports quality gains scored by the same LLM that drives the modifications.","lead":"This paper describes a multi-agent system that automatically tweaks other AI agent workflows using feedback from a small language model. The reported improvements are measured by the same model that proposes and scores the changes, so the gains may be self-fulfilling.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not independent evidence: the same Llama 3.2-3B model both proposes modifications and scores outputs, so the improvements are comparisons of the optimizer against its own objective.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation loop is circular because the same LLM both proposes changes and scores outputs. I agree with the REJECT verdict, so no verdict change is needed. The central contribution is an autonomous optimization loop, and the only quantitative evidence for its benefit is a set of scores produced by the same model that was optimized. That evidence cannot distinguish genuine quality gains from reward hacking or evaluator bias. The paper's Section 7 acknowledges these risks but does not design the experiments to exclude them. Even the internal stopping logic in Algorithm 1 is suspect: after an improvement, the convergence check compares the new score to itself and terminates immediately, so the described multi-step refinement is not actually implemented as written. To the paper's credit, the anonymous repository containing original and evolved agents and outputs is publicly available, which makes an external check feasible. The proposed concrete test—blinded human evaluation of the archived outputs, or equivalently a held-out stronger evaluator—would settle whether the reported improvements survive outside the optimizer's own metric. Until such a test is run, the case-study results are consistent with the null hypothesis that the framework merely specializes outputs to the idiosyncrasies of Llama 3.2-3B.","tokens_in":9147,"tokens_out":3199,"duration_ms":32540,"concrete_test":"Download the anonymous.4open.science/r/evolver-1D11 archive, extract the original and evolved outputs for all reported case studies, and have blinded human raters score them on the same criteria used in the paper, without revealing which outputs came from the evolved system. Compare per-case deltas with Figure 8 and compute inter-rater agreement with the Llama 3.2-3B evaluator. If the human-rated deltas are near zero or negative, the reported gains are artifacts of self-evaluation; if they reproduce, the core loop receives independent support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the Evaluation Agent's Llama 3.2-3B scores on qualitative criteria (clarity, relevance, depth, actionability) measure genuine output quality. This premise is not merely unverified; it is structurally circular. In Section 3.2 and Algorithm 1, the same LLM generates hypotheses, modifies code, evaluates outputs, and selects the variant that maximizes the evaluator's score. Any observed improvement is therefore an artifact of optimizing the very function used to measure improvement, independent of external quality. No independent ground truth, human evaluation, or second-model check is provided; the Section 4 scores are the optimizer's own objective. Section 7 explicitly concedes that LLM-based feedback and evaluation may introduce biases and reward hacking, but the experiments never test or control for this. There is also an internal control-flow inconsistency in Algorithm 1: after a successful update (line 21), line 27 checks |Si+1 − Sbest| < epsilon, which is zero because Sbest was just set to Si+1, so the loop stops after the first successful variant. That undermines the claim of iterative refinement. The public archive is a useful resource, but it cannot validate the metric itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent framework for autonomously optimizing agentic AI systems. The system comprises Refinement, Execution, Evaluation, Modification, and Documentation agents, with an LLM (Llama 3.2-3B) generating hypotheses, executing variants, scoring outputs, and selecting the best variant according to qualitative criteria. Algorithm 1 formalizes this as iterative hill-climbing on an evaluation score f. The authors report case studies in market research, medical AI architecture, career transition, outreach, LinkedIn content, meeting facilitation, lead generation, and presentation development, claiming significant improvements in output quality, relevance, and actionability. Public code and output data are posted in an anonymous repository.","tokens_in":9337,"tokens_out":2829,"duration_ms":28673,"significance":"If the claimed autonomous optimization were convincingly demonstrated, the framework would be relevant to the growing area of agentic AI system design. The paper's central idea---closed-loop LLM-driven refinement with specialized agents---is plausible and worth investigating, and the authors provide a useful public archive of original and evolved agent code and outputs. However, the empirical evidence as presented does not establish the claim: every reported improvement is measured by the same LLM evaluator that drives the optimization, and no external validation, control condition, or statistical analysis is provided. The paper's own limitations section (Section 7) concedes risks of bias and reward hacking that the experiments never test. The significance of the work therefore rests on an unverified and structurally circular measurement.","major_comments":[{"comment":"The central empirical claim is undermined by circularity. The Evaluation Agent uses Llama 3.2-3B to score outputs on criteria such as clarity, relevance, depth, and actionability, and the same LLM (via the Hypothesis and Modification Agents) generates the modifications that are then accepted or rejected based on those very scores. Consequently, the improvements reported in Section 4 and Figure 8 are not independent measurements; they are comparisons of the optimizer against its own objective. To support the claim of genuine quality improvement, the authors must provide external evaluation, such as human annotation, a held-out benchmark, or a second independently designed evaluator, and ideally a control condition such as random code mutation or modifications without evaluator feedback.","section":"Section 3.2, Algorithm 1, Section 4"},{"comment":"There is a control-flow inconsistency in Algorithm 1. After a successful update on lines 21-26, Sbest is set to Si+1, so the stopping condition on line 27, |Si+1 - Sbest| < epsilon, evaluates to 0 < epsilon. For any positive epsilon, this is true, and the loop terminates immediately after the first accepted variant. This contradicts the paper's description of iterative refinement and means the reported 'evolved' systems may be the result of only a single accepted modification. The stopping condition needs to be corrected, for example by comparing Si+1 against the pre-update Sbest or by tracking improvement relative to the previous iteration's score.","section":"Algorithm 1, lines 21-28"},{"comment":"The evaluation results are reported as single scores without repeated runs, confidence intervals, or measures of variance. Agentic system evolution is stochastic: different random seeds, LLM sampling temperatures, and initial conditions would produce different trajectories. The claim of 'consistent improvements' in Section 4.8 cannot be supported by one run per case study. The authors should report multiple independent runs and appropriate summary statistics, and ideally perform a significance test against a baseline such as random or fixed modifications.","section":"Section 4.8, Figure 8"},{"comment":"The Limitations section explicitly acknowledges that using LLMs for feedback, hypothesis generation, and evaluation 'may lead to inaccuracies, lack of explainability, and biases' and that poor criteria can result in suboptimal refinements. These caveats apply directly to the reported case studies, yet no experiment in the paper tests for evaluator bias, reward hacking, or sensitivity to the choice of criteria. The discussion of Section 7 should be connected to the results: at minimum, the paper should state that all reported scores are self-evaluations and therefore do not provide evidence of external validity.","section":"Section 7"}],"minor_comments":[{"comment":"The abstract and case studies use the phrase 'significant improvements' without statistical significance testing; the wording should be qualified as 'score improvements according to the framework's evaluator.'","section":"Abstract and Section 4"},{"comment":"The comparison report in Appendix A.4 treats breadth (e.g., identifying seven markets versus two) as self-evidently better without evidence that the additional markets are correctly identified or relevant; a qualitative breadth argument is not a substitute for accuracy evaluation.","section":"Appendix A.4"},{"comment":"The caption describes 'each pair of bars' while the text refers to box plots; the figure and its caption should be made consistent.","section":"Figure 8"},{"comment":"Several references, including Johnson and Liu (2023), Pan and Zhang (2024), and Miller et al. (2024), lack full bibliographic details and could not be easily verified; the authors should complete and check these entries.","section":"References"},{"comment":"The phrase 'Synthesis Framework framework' contains a duplicated word and should be corrected.","section":"Section 3.1"}],"recommendation":"reject","confidential_remarks":"The core empirical claim is not supported by the evidence as presented: the evaluation metric is identical to the optimization objective, and the stopping-condition bug means even the internal story of iterative refinement is inconsistent. These are load-bearing issues that would require new experiments (external evaluation, controls, repeated runs) rather than local edits. The public code archive is a constructive contribution, and a substantially revised version with independent validation could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The gains it reports are not independent evidence: the same Llama 3.2-3B model scores the outputs and proposes the modifications, so the evaluations are the optimizer's own objective. And the pseudocode in Algorithm 1 breaks after the first successful variant, because the stopping check compares Si+1 to Sbest after Sbest has been overwritten with Si+1. The paper's iterative-refinement claim is not actually implemented in the algorithm. To its credit, the paper ships a public archive with original and evolved agent code, outputs, and evaluation reports. The architecture is a reasonable composition of known parts: hypothesis generation, modification, execution, evaluation. The case studies are detailed, and the appendix walks through one refinement cycle concretely. The authors cite relevant prior work (ADAS, AgentInstruct, reflective tree search, reward hacking) and list limitations in Section 7. The problems are the core evidence and the algorithm. There is no external benchmark, no human evaluation, no second-model check, no control condition such as random code mutation, and no repeated runs with confidence intervals. Section 3.2 says Llama 3.2-3B does the evaluation; Algorithm 1 uses that score to accept or reject variants. Section 7 concedes the risks of LLM bias and reward hacking, but the results never address them. The algorithm's stopping rule is a real correctness bug, not a stylistic choice. The abstract's claim of 'optimal performance without human input' is not supported by the evidence. Who is this for? Practitioners might use the case studies as a template for agent tuning. Researchers should not treat the scores as evidence. It's a demo, not a validated method. I'd desk reject at a serious venue. With an independent evaluator, a baseline, and repeated runs, the framework might become publishable, but the current version doesn't get there.","headline":"A working engineering demo whose evidence is circular and whose algorithm breaks after the first success; the reported gains are the evaluator's own scores.","tokens_in":9879,"tokens_out":4798,"would_cite":false,"duration_ms":41432,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A loop of AI agents can autonomously refine other agentic AI systems, the authors claim, with case-study gains on every task.","keywords":["agentic AI","autonomous optimization","LLM-driven feedback loops","iterative refinement","multi-agent systems","evolutionary algorithm","Llama 3.2","qualitative evaluation"],"falsifier":"Run the evolved and original agents on the same tasks and have independent human raters (or a stronger, instruction-following LLM that was not part of the loop) score the outputs blind; if the evolved outputs are not preferred at a similar or greater rate, the loop's reported improvements do not generalize beyond its own evaluator.","tokens_in":8889,"feed_emoji":"🤖","tokens_out":4684,"duration_ms":40353,"temperature":0.7,"pith_summary":"This paper tries to establish that a fully autonomous loop of AI agents can optimize other agentic AI systems without human intervention. The loop generates hypotheses from an LLM's evaluation of the current output, modifies the agent code, re-runs it, and keeps the variant with the highest score, repeating until improvement falls below a threshold. The authors report nine case studies in domains from market research to medical AI architecting, claiming notable gains in clarity, relevance, depth, and actionability for every evolved system. If true, this would replace manual prompt and workflow tuning with an automated, domain-independent refinement process.","feed_headline":"AI-against-AI feedback loop refines agent systems autonomously","feed_subtitle":"Nine case studies show consistent quality, relevance, and actionability gains without manual intervention.","key_machinery":"The load-bearing mechanism is the refinement loop in Algorithm 1, driven by two collaborating sets of components: the Synthesis Framework (Refinement, Hypothesis Generation, and Modification agents) and the Evaluation Framework (an Evaluation Agent using Llama 3.2-3B, a Selection Agent, and a Memory Module). The Evaluation Agent assigns scores on qualitative criteria such as clarity, relevance, depth of analysis, and actionability, plus quantitative metrics; the Hypothesis Agent turns evaluative feedback into concrete code changes; the Selection Agent accepts a new variant only if its score is strictly higher than the best-known score; and the loop terminates when improvement falls below a threshold $\\epsilon$.","core_discovery":"The central claim is that the framework's iterative feedback loop—Evaluation, Hypothesis Generation, Modification, Execution, Selection—constitutes a general method for optimizing agentic AI solutions. Starting from a baseline code variant $C_0$, the loop scores outputs with an LLM on qualitative criteria, generates hypotheses $H_i$ for improvement, produces a new variant $C_{i+1}=M(H_i, C_{\\text{best}})$, and accepts it only if its score exceeds the best-known score. The process stops when the score gain is below $\\epsilon$ or a maximum iteration count is reached, and the best-known variant and output are returned. The paper claims this achieves optimal performance without human input and demonstrates significant improvements in output quality, relevance, and actionability in all case studies.","pith_inferences":["Because the evaluator, hypothesis generator, and selection judge are the same LLM (Llama 3.2-3B), some of the reported gains may reflect the model's preference for its own writing style rather than objective task quality; an independent evaluator would test this.","The paper's 'optimal' is relative to the predefined qualitative criteria; if those criteria miss important dimensions, the loop will optimize the wrong target, as the Limitations section concedes.","A natural extension is to use the same loop with a reward model trained on human preferences, or to combine multiple evaluators, which would make the reported gains more externally valid.","The loop is essentially a discrete hill-climbing search over code space; the choice of starting code $C_0$ and the hypothesis-generation temperature likely determine whether it escapes local optima."],"forward_implications":["Agentic systems can be optimized end-to-end without manual prompt engineering or hand-tuned role definitions.","The same loop applies across domains, since the case studies span market research, healthcare, supply chain, career coaching, social media content, drug discovery, and education.","Evolved systems show not only higher scores but also less score variability, indicating more consistent outputs.","The framework returns the best-known variant and its output, so each iteration is safe in the sense that performance never decreases relative to the stored best.","The stopping condition based on $\\epsilon$ gives a concrete rule for when to stop refining, usable in enterprise deployment."],"supporting_citations":[{"why":"Provides MLAgentBench, a benchmark for evaluating language agents, which the paper draws on to motivate the evaluation criteria for agentic systems.","marker":"Huang et al. (2024)"},{"why":"Introduces ADAS, the automated design of agentic systems, which the paper contrasts with its own focus on optimizing existing agent systems rather than inventing new ones.","marker":"Hu et al. (2024)"},{"why":"Identifies reward hacking in LLM feedback loops, a risk the paper claims its robust evaluation framework mitigates.","marker":"Pan et al. (2024)"},{"why":"Supplies the insight that self-improving agents adjust roles and interactions autonomously via feedback loops, which the paper cites as critical to its approach.","marker":"Sulc et al. (2024)"},{"why":"Demonstrates automated evaluators for refining digital agents in web navigation, a prior domain-specific instance of the general refinement idea the paper extends.","marker":"Pan and Zhang (2024)"}],"fun_headline_variants":["Self-optimizing AI agents via LLM feedback loops","Autonomous AI refines its own agent systems","No humans needed: AI loop boosts agent quality","AI agents self-improve through iterative feedback","LLM-driven evolution of agentic AI solutions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire improvement signal comes from one LLM's scores on qualitative criteria, so if Llama 3.2-3B's judgments do not track genuine output quality, every reported gain could be an artifact of the evaluator's own biases.","fun_headline_variants_meta":{"raw":{"variants":["Self-optimizing AI agents via LLM feedback loops","Autonomous AI refines its own agent systems","No humans needed: AI loop boosts agent quality","AI agents self-improve through iterative feedback","LLM-driven evolution of agentic AI solutions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00013,"raw_usage":{"total_tokens":1096,"prompt_tokens":884,"completion_tokens":212,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":140}},"tokens_in":500,"tokens_out":212,"duration_ms":2939,"temperature":1.0,"reasoning_tokens":140,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:44:12.069699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the evolved and original agents on the same tasks and have independent human raters (or a stronger, instruction-following LLM that was not part of the loop) score the outputs blind; if the evolved outputs are not preferred at a similar or greater rate, the loop's reported improvements do not generalize beyond its own evaluator.","supporting_citations":[],"review_version":1}