{"id":"350d4015-a963-412c-a7e5-03703215638c","arxiv_id":"2504.18316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A lead AI agent creates a custom team of helper agents for each bug, and this adaptive team fixes more benchmark bugs than one-shot prompting.","lead":"This paper describes an agentic debugging system in which one leading LLM agent decides how many helper agents to create and what roles they should play based on the buggy code. In a first test on 50 coding problems, this adaptive approach fixed more bugs than plain one-shot prompting, gaining 6% to 18% depending on the model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fix-rate gain is measured without isolating the adaptive mechanism; validation of the pipeline's pre-fix stages is absent, and the paper's own limitation statement supports not treating the 6–18% improvement as conclusive.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and the reader's weakest assumption concerns unmeasured planning/agent-profile quality. My analysis agrees with that as a real soft spot: the design's core mechanism is the dynamic creation of roles and strategies, yet the evaluation only records final fix outcomes and never validates the intermediate artifacts. The stronger and more decisive concern, which the reader also implicitly supports, is that the comparison to one-shot prompting does not control for the extra compute, retries, and reflective feedback that the adaptive pipeline inherently consumes. This is a confound, not an internal inconsistency: the paper's own claims in Section 1 ('significant improvement') and the abstract ('average improvement of 11%') are not supported as conclusively as the word 'significant' suggests, and the authors themselves scope the work as preliminary in Section 5 and the abstract. I therefore do not move the verdict to REJECT: the direction of the effect is consistent across four models, the mechanism is plausible, a public implementation is promised, and the paper explicitly frames the evaluation as preliminary. I also do not move to ACCEPT because the central quantitative claim needs a baseline that isolates the adaptive mechanism. UNCHANGED was also considered: the reader already issued CONDITIONAL, and the confound with extra attempts is arguably the same family of concern (lack of a static multi-agent baseline was named in the reader's rationale). I chose to keep CONDITIONAL while making the confound explicit, because the reader's weakest assumption focused on internal planning quality, whereas the more load-bearing issue is that the experiment cannot attribute the fix-rate gain to adaptation at all. Both point to the same remedy: add controlled baselines and measure intermediate quality. Hence agreement_with_reader is 'agree' rather than 'partial', since the reader's rationale already names the missing static multi-agent baseline and the lack of significance testing; my concern is a sharper form of the same critique.","tokens_in":7860,"tokens_out":1924,"duration_ms":17219,"concrete_test":"Re-run the same 50 DebugBench instances with two additional baselines using identical budgets: (1) a static multi-agent team of 3 fixed roles with iterative re-planning, and (2) single-agent iterative reflection with N attempts and the same total token/iteration budget as the adaptive system. If the adaptive design does not outperform both baselines, the 6–18% gain is attributable to multiple attempts / iterative feedback rather than to adaptive profiling; also report per-model 95% confidence intervals or exact binomial tests on the Table 2 counts.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the adaptive design (dynamic planning, agent profiling, prioritization, validation loop) improves bug fix rates by 6–18% over one-shot prompting (Table 2, Section 4). The load-bearing concern is that the evaluation conflates the proposed architecture with the effect of giving the model extra reasoning turns and multiple attempts. The only comparison is against one-shot prompting, which gets a single final answer; the adaptive design gets a plan, several agent reports, and iterative re-planning with an explicit instruction to propose a different strategy after failure. Any iterative-reflection or self-consistency method with multiple attempts and feedback could plausibly produce a comparable gain. Section 4 also reports no measurement of the intermediate products that the design depends on: code analysis quality, agent profile quality, priority correctness, or validity of the validation step. Section 3.2 states that code analysis 'is critical because it guides the creation of the debugging strategy,' but the evaluation never checks whether misclassification of a bug type actually leads to failure, so the central mechanism is unverified. With 50 instances and no significance test, the per-model gains (6%, 8%, 12%, 18%) are also consistent with some of the apparent benefit being sampling noise; the absence of a static multi-agent or single-agent multi-attempt baseline means the claimed 'adaptive' component is not the variable being tested. The paper itself disclaims conclusiveness: Section 5 lists extensive evaluation as future work, and the abstract calls the result 'initial evaluation' and 'promising results.' Thus the strongest claim, as stated with 'significant improvement,' overreaches what the experiment can establish, but the design remains plausible and worth conditional publication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an adaptive agentic architecture for LLM-based debugging. A main agent first analyzes the buggy code, then dynamically generates the profiles and prioritization of specialized agents, collects their reports, validates the fix, and iterates with an explicitly different strategy if validation fails. The authors evaluate the design on 50 DebugBench instances using four LLMs (Llama3, DeepSeek-coder, Mistral-Large, GPT-4), reporting fix-rate gains of 6% to 18% over one-shot prompting and showing that the number of created agents and iterations grows with problem complexity. They conclude that adaptive multi-agent debugging improves both fix rate and resource usage, and they outline future plans for larger evaluation and other software engineering tasks.","tokens_in":8101,"tokens_out":4607,"duration_ms":48490,"significance":"If the central claims were fully supported, the contribution would be a practical and timely one: it addresses a known weakness of static multi-agent LLM systems by making the team composition and task plan problem-dependent. The paper also has concrete strengths: it gives raw counts (Table 2), compares four models, reports resource-usage trends (Figures 2 and 3), and makes the implementation publicly available, which supports replication and further study. The key limitation is that the evidence does not isolate the adaptive mechanism: the only baseline is one-shot prompting, so the reported gains could plausibly come from extra reasoning steps, multiple attempts, or iterative reflection rather than from dynamic agent creation. Because the design's intermediate stages are not measured, the paper currently demonstrates feasibility of a pipeline rather than validating the specific adaptive mechanism it advocates.","major_comments":[{"comment":"The central claim that the adaptive design improves bug-fixing over non-agentic debugging is built on a comparison against one-shot prompting only. The adaptive pipeline receives a plan, multiple agent reports, and an explicit re-planning prompt after failure, so it has substantially more computational budget and more opportunities to change its answer than the one-shot baseline. The reported gains of 6% to 18% are therefore consistent with the effect of iterative reflection or multiple attempts, not necessarily with the effect of dynamic agent profiling and prioritization. To support the 'adaptive' claim, the evaluation needs additional baselines such as a static multi-agent team with the same number of agents, a single-agent iterative self-refinement baseline with a comparable number of LLM calls, and a self-consistency baseline; without these, the mechanism being tested is confounded with extra reasoning budget.","section":"Section 4.2, Table 2"},{"comment":"The design narrative in Section 3.2 states that code analysis 'is critical because it guides the creation of the debugging strategy and the specification of the necessary agents,' and the same section makes agent profiling, prioritization, and validation load-bearing parts of the approach. However, the evaluation in Section 4 never measures the quality of the code analysis, the generated agent profiles, the priority ordering, or the validation step. It only records final fix outcomes and the number of agents/iterations. This leaves open the possibility that the system succeeds despite poor intermediate decisions, or that the adaptive mechanism contributes nothing beyond retrying with a new prompt. The authors should report intermediate products (e.g., classification accuracy of the bug types, profile usefulness ratings, or a controlled ablation in which the analysis stage is corrupted) so that the central mechanism is actually tested.","section":"Section 3.2"},{"comment":"The statistical basis of the evaluation is under-specified. With 50 instances per model, the smallest reported gain (GPT-4: 41/50 vs 44/50) corresponds to only three additional fixed instances, and no confidence intervals, significance tests, or multiple runs are reported; the word 'significant' in Section 1 is therefore not supported by the evidence. In addition, the paper does not describe how the 50 instances were sampled from DebugBench or whether 'fixed correctly' means passing the benchmark's reference test suite, executing the code, or relying on the main agent's own validation. This makes it impossible to assess both the external validity of the comparison and the reliability of the success criterion. Please report the sampling procedure, the success-judgment protocol, and at least basic uncertainty estimates or error bars.","section":"Sections 4.1 and 4.2"}],"minor_comments":[{"comment":"The phrase 'outperforming traditional non-agentic debugging methods' overstates the comparison: the evaluation only compares against one-shot prompting, not against traditional debuggers or other non-agentic methods.","section":"Section 1"},{"comment":"There is a typo in the description of external module-aided planning: 'external planer' should be 'external planner.'","section":"Section 2.1"},{"comment":"The text contains a typo in the description of AgentFL: 'mutli-agent' should be 'multi-agent.'","section":"Section 2.2"},{"comment":"The table reports raw counts but not rates; presenting the rates and the average 11% gain in the table caption would help the reader verify the abstract's summary.","section":"Table 2"},{"comment":"Figures 2 and 3 report resource usage for Llama3 and GPT-4 only; the text should clarify why DeepSeek-coder and Mistral-Large are excluded and how the complexity levels ('low', 'medium', 'high') are defined, since those categories drive the adaptability claim.","section":"Section 4.2 and Figures 2 and 3"},{"comment":"The reference contains a typo ('arcelona' should be 'Barcelona'), and the ACM DOI placeholders in the reference format should be replaced with real DOIs in the camera-ready version.","section":"Reference [15]"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a plausible adaptive multi-agent debugging design, with a genuinely new component—the main agent reads the bug and then decides how many specialists to spawn and with what roles—but the evaluation does not yet isolate that component as the cause of the reported gain. The 6–18% fix-rate improvements over one-shot prompting are consistent with the hypothesis but not conclusive.\n\nWhat's new: existing multi-agent debuggers (AgentFL, FixAgent, MetaGPT) fix roles in advance. Here the roles and count are generated per bug from a code analysis step, and the validation loop explicitly asks for a different strategy after failure. That's a sensible composition of planning, multi-agent collaboration, and iterative reflection, and it's clearly described. The figures on agent counts and iterations by problem complexity are also a nice touch: simple syntax bugs often need one agent, harder bugs up to five.\n\nThe paper is honest about being preliminary. It calls the evaluation 'initial' in the abstract and lists extensive evaluation as future work. It also ships source code.\n\nWhere it's soft: the experiment has no significance tests, no confidence intervals, no repeated runs, and no described sampling procedure for the 50 DebugBench instances. More importantly, the only comparison is against one-shot prompting. The adaptive pipeline gives the model a plan, several agent reports, and multiple attempts with re-planning. Any method that grants extra reasoning turns and a chance to revise would plausibly improve fix rates by a similar margin. Without a static multi-agent baseline or a single-agent multi-attempt baseline, the claim that the adaptivity itself is what helps is not load-bearing in this data. The paper also never measures the intermediate products—code analysis quality, agent profile quality, priority correctness—so the central mechanism is unverified. The word 'significant' in the intro overreaches; statistically, with 50 instances, per-model gains of 8-18% could be sampling noise, though the direction is consistent across all four models.\n\nThat said, the design is a real contribution and the paper is well written. It deserves a serious referee, but the referee should ask for a better comparison set and more rigor before publication.\n\nMy take: it's a workshop-quality short paper that should go through review rather than be desk-rejected. I'd probably cite it as related work on adaptive agent design, but I wouldn't rely on the claimed effect size until the baseline issue is fixed.","headline":"Plausible adaptive multi-agent debugging design with a real novelty, but the evaluation conflates adaptivity with extra attempts; the design deserves review, the claimed effect does not yet.","tokens_in":8694,"tokens_out":2090,"would_cite":true,"duration_ms":19331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive agent teams fix 6–18% more bugs than one-shot prompting.","keywords":["adaptive agentic design","LLM debugging","multi-agent systems","agent profiling","planning","iterative reflection","bug fixing rate","agentic AI"],"falsifier":"Take the same 50 buggy programs and replace the main agent's generated agent profiles with fixed, random, or hand-written profiles for every task; if the fix rate and iteration counts are unchanged, then dynamic profiling is not what drives the reported gains.","tokens_in":7665,"feed_emoji":"🐞","tokens_out":4714,"duration_ms":42068,"temperature":0.7,"pith_summary":"This paper proposes a debugging system in which a main LLM agent does not follow a fixed recipe; it first reads the buggy code, decides what kind of problem it is, and then creates a team of specialized agents with roles and an execution order tailored to that problem. It combines planning, multi-agent collaboration, and iterative reflection: specialists report back, the main agent validates the result, and if the bug remains it issues a new plan explicitly different from previous ones. In an initial evaluation on 50 buggy Python programs across four LLMs, the adaptive design fixed 6% to 18% more instances than one-shot prompting, averaging an 11% improvement. The same evaluation shows resource adaptation: simple syntax errors were usually handled by one agent, while harder problems generated up to five agents and multiple iterations. If the result holds, adaptive agentic debugging could spend reasoning budget where the bug is hard and avoid wasting it on easy fixes.","feed_headline":"Adaptive agent teams fix 6–18% more bugs","feed_subtitle":"One leader LLM sizes the team per bug: simple syntax gets one agent, hard bugs up to five.","key_machinery":"The central mechanism is a two-tier agent architecture managed by a main agent acting as team leader. The load-bearing object is the dynamically generated agent profile: a prompt, created after code analysis, that defines a specialized agent's role and task, together with a priority ordering over the generated profiles. The main agent decomposes the debugging task, issues profiles, receives reports, validates the fix, and explicitly prompts itself to produce a different plan on the next iteration. This assembly of planning, multi-agent collaboration, and iterative reflection is what lets the number of agents track problem complexity.","core_discovery":"The central claim is that a statically configured multi-agent debugging pipeline is less effective than an adaptive one: letting the leading model inspect the code and compose the agent team makes both the fix rate and the resource allocation depend on the problem. The mechanism is a main agent that performs code analysis, agent profiling, and agent prioritization, then dispatches specialized agents that work autonomously and report back; the main agent validates the result and, if needed, iterates with a different strategy. On a benchmark of 50 buggy Python solutions, the adaptive design raised fix counts from 26 to 35, 32 to 38, 29 to 33, and 41 to 44 across the four tested models, gains the paper reports as 18%, 12%, 8%, and 6%. The paper interprets these numbers as evidence that the design improves debugging performance across LLMs while generating only as many agents and iterations as the complexity of the problem calls for.","pith_inferences":["The same leader-specialist loop should transfer to other software engineering tasks such as requirement elicitation, code generation, or test design, which the paper lists as future work; the load-bearing condition is that the leader's plan quality remains good on those tasks.","A testable extension would be to ablate the adaptive component by feeding specialized agents fixed, hand-written profiles for every bug and measuring whether fix rate and iteration count change; the paper does not report this comparison, but its hypothesis implies dynamic profiling matters.","If the leader's own analysis is the bottleneck, part of the observed gain may come from the extra reasoning tokens spent in analysis rather than from specialization; a cost-controlled comparison against a fixed team with an equal token budget would separate the two effects."],"forward_implications":["Fixed-size multi-agent debuggers are over-configuring easy bugs and under-configuring hard ones; a team that sizes itself should dominate on both fix rate and cost.","The method transfers across models that can follow role prompts, since the same leader-profiling loop produced positive gains on all four tested LLMs.","Because the main agent invents roles at runtime, the system can express debugging subtasks such as syntax checking, semantic verification, and test generation without retraining for each role.","Resource counts become a diagnostic: the number of agents and iterations generated per instance can serve as a proxy for perceived bug complexity and for where a given model struggles."],"supporting_citations":[{"why":"Supplies the 50 buggy Python instances used in the evaluation and the bug-type and complexity categorization.","marker":"[21]"},{"why":"Defines agenticness and its four components, framing adaptability as a missing property in prior debugging approaches.","marker":"[20]"},{"why":"Supplies the decomposition-first planning taxonomy that the main agent's task decomposition follows.","marker":"[10]"},{"why":"Prior evaluation used to choose the open-source and closed models tested in the experiments.","marker":"[15]"},{"why":"Represents the static multi-agent design that the paper contrasts with its adaptive alternative.","marker":"[9]"}],"fun_headline_variants":["Adaptive agent teams fix 6–18% more bugs","Dynamic agent roles boost LLM debugging","One agent sizes the team per bug","Adaptive debugging: fewer agents, better fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the main agent's code analysis and agent profiling being accurate enough to produce appropriate specialist roles; the paper measures only final fix outcomes, not the quality of those plans, so a weak planner could sink the system even when a fixed team would have solved the problem.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive agent teams fix 6–18% more bugs","Dynamic agent roles boost LLM debugging","One agent sizes the team per bug","Adaptive debugging: fewer agents, better fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1236,"prompt_tokens":921,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":537,"tokens_out":315,"duration_ms":3750,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:19:14.034449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 50 buggy programs and replace the main agent's generated agent profiles with fixed, random, or hand-written profiles for every task; if the fix rate and iteration counts are unchanged, then dynamic profiling is not what drives the reported gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 50 buggy Python instances used in the evaluation and the bug-type and complexity categorization."}],"review_version":1}