{"id":"6437c71d-6c20-43b5-8725-2e31d1a1a6db","arxiv_id":"2608.10424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding shared debugging memory, hyperparameter tuning enforcement, and Thompson-sampling backtracking to tree-search agents recovers wasted compute and improves MLE-bench scores with the same language model.","lead":"This paper studies why AI agents that write machine learning code waste computing time, and tests three fixes: sharing debugging knowledge across the search, forcing hyperparameter tuning, and smarter backtracking. The fixes improve agent performance on nine tabular data competitions without changing the underlying language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Debug-consultant gains may come from pre-seeded environment constraints: 69% first-step validity under treatment (3% baseline) is unexplained by a within-run registry that starts empty.","rationale":"The central claim is broad: large gains in autoresearch agents are achievable through agentic design alone, holding the LLM fixed. The decisive evidence is the debug consultant's 22-to-38 gold and 81%-to-100% valid-submission improvement, presented as a consequence of sharing runtime constraints across the search tree. My concern targets exactly that mechanism. The step-0 numbers in Table 5 are hard to reconcile with a registry that starts empty within each seed. If the registry is seeded from prior runs or from the authors' own knowledge of the LightGBM incompatibility, then the comparison is not between the same agent with and without a memory mechanism; it is between an agent given an environment-specific bug sheet and one without it. That would not demonstrate that global memory recovers wasted compute. One could respond that the prompt injection is itself an agentic design choice, but the paper's mechanism is adaptive learning, and the claimed reduction in redundant bug encounters (46% to 7.8%) would be partly tautological if the banned list is provided upfront. This is a load-bearing internal-validity question, not merely a generalization concern. The reader's weakest assumption is about transfer to other backbones and environments; I agree that is a real limitation, but it is secondary because one first needs to know whether the gains occur via the stated mechanism. The missing code link makes this impossible to verify, which is why I keep the paper in conditional status pending the ablation. If the ablation fails, the debug-consultant portion of the central claim should be downgraded, though the HPO and Thompson Sampling results may still stand independently.","tokens_in":34768,"tokens_out":10714,"duration_ms":104070,"concrete_test":"Release the code and run the treatment with the registry reset to empty at the start of every seed (no cross-seed memory and no pre-seeded BANNED entries), while logging registry contents at S0. If S0 valid-submission rates and gold counts fall toward baseline (3% and 22), the debug-consultant gains come from pre-seeding; if they remain near the reported 69% and 38, the within-run sharing mechanism is confirmed. A second arm that pre-seeds only the LightGBM entries would quantify their standalone contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's flagship result is the debug consultant: AIDE golds rise from 22 to 38 and valid submissions from 81% to 100%. The mechanism is presented in Section 3.1 as online, within-run learning: a shared registry starts empty, accumulates compressed errors as nodes crash, and injects constraints into later generations. Section 4.2, however, says the consultant 'supplies the environment's constraints before the agent writes its first line of code,' and Table 5 reports that 69% of treatment seeds already have a valid submission at S0, versus 3% for baseline. A within-run registry is empty before the first node executes, so it cannot explain this. Either the registry is initialized across runs or pre-seeded with known environment facts (the LightGBM early_stopping_rounds/verbose incompatibility is the obvious candidate), or there is an undocumented pre-flight environment probe. None of these is described, and the 'Code available here' link in the manuscript points to no URL, so the initialization cannot be checked. If the BANNED list is pre-seeded, the headline gain measures environment-specific prompt engineering rather than 'adaptive learning of the execution environment,' and the central claim that agentic design alone recovers wasted compute is not established by the main experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies four interventions aimed at recovering wasted compute in tree-search-based autoresearch agents (AIDE, ML-Master, and MLEvolve) applied to nine tabular Kaggle/MLE-bench tasks with a fixed GPT-5-mini backbone and a two-hour compute budget per run. The interventions are: (1) a global debug consultant that shares discovered runtime constraints across the search tree via a shared bug registry and injected BANNED/USE lists; (2) prompt- and control-loop-level enforcement of hyperparameter tuning; (3) Thompson Sampling with backtracking to replace random sibling selection; and (4) a diagnostic injecting adversarial EDA results to test whether agents act on analysis. Headline results include raising AIDE gold medals from 22 to 38 and valid-submission rate from 81% to 100% with the debug consultant, large AIDE gains from HPO guidance (with a documented degradation for ML-Master), and a 54.5% reduction in null runs from Thompson Sampling in a controlled comparison. The paper also includes detailed journals, case studies, per-seed scores, and an honest discussion of where interventions fail. The central claim is that large gains are achievable through agentic design alone, holding the underlying language model fixed.","tokens_in":35072,"tokens_out":6177,"duration_ms":60295,"significance":"If the central claim holds, the paper would be a meaningful contribution to agentic AutoML design: it identifies concrete structural causes of wasted compute and evaluates targeted fixes with extensive logs and per-seed results. The strengths are the detailed case studies (e.g., S5E3 and Wine showing the same early_stopping_rounds bug dominating baseline budgets), the controlled Thompson Sampling comparison in Table 16, and the transparent reporting of the ML-Master HPO degradation. However, the significance is currently undercut by a load-bearing ambiguity about how the debug consultant is initialized (Section 4.2 says it supplies environment constraints 'before the agent writes its first line of code' while Section 3.1 describes a within-run registry that starts empty), the absence of a working code link, and the use of a single backbone and environment in which one LightGBM API incompatibility dominates failures. These issues must be resolved before the paper's broad claim about agentic design alone is established.","major_comments":[{"comment":"There is an internal contradiction about the debug consultant's registry initialization that is load-bearing for the paper's central claim. Section 3.1 describes a within-run process: Step 1 starts 'When a node crashes', Step 2 accumulates records into a shared registry, and Step 3 injects constraints into subsequent generations. This implies the registry is empty before the first node executes. Yet Section 4.2 states that the consultant 'supplies the environment's constraints before the agent writes its first line of code', and Table 5 reports that 69% of treatment seeds already have a valid submission at step 0, versus 3% for the baseline. A within-run registry cannot be non-empty before any node has executed, so either the registry is pre-seeded with known environment facts (e.g., the LightGBM early_stopping_rounds/verbose incompatibility), or there is an undocumented pre-flight environment probe, or the definition of 'step 0' differs from what I assume. This matters because if the BANNED list is pre-seeded with the specific LightGBM API incompatibility, the headline gains are attributable to environment-specific prompt engineering rather than to the claimed adaptive, within-run learning of the execution environment, and the abstract's claim that large gains come from 'agentic design alone' is not established. Please clarify the initialization of the registry, define step 0 precisely, and if pre-seeding is used, disclose it and re-frame the mechanism accordingly, ideally with an ablation that removes the pre-seeded entries.","section":"Section 3.1 vs. Section 4.2 and Table 5"},{"comment":"The manuscript contains the text 'Code available here' (in Section 2) but no URL or repository identifier follows it. The debug consultant's mechanism, including whether the registry is pre-seeded, cannot be verified without the code. Since the central claims rely on implementation details (the sharing of the registry, the exact injection of BANNED/USE constraints, and the Thompson Sampling updates), the absence of a working code link is a serious reproducibility problem. Please provide a complete code release or, if the link was accidentally omitted, include it in the revision.","section":"Section 2 and Appendix 'Code available here'"},{"comment":"The headline Thompson Sampling results in Table 12 are obtained under a configuration that differs from the baseline in two additional hyperparameters: Table 13 changes initial drafts from 5 to 20 and max-debug depth from 20 to 5, alongside the TS selection strategy. The paper correctly includes the controlled comparison in Table 16 (AIDE+TS+More Drafts vs. AIDE+More Drafts), which isolates TS and shows the null-run reduction from 33 to 15. However, Table 16 also shows that TS loses the score comparison on 6 of 9 competitions (Cirrhosis, S5E3, S5E6, S5E7, S5E12) and ties a seventh, so the claim that the added stability comes 'at no cost to peak performance' should be hedged to 'at no cost in mean score beyond sampling error on most competitions.' I recommend presenting Table 16 (or an equivalent controlled comparison) as the primary evidence for the TS contribution, with Table 12 relabeled as the combined-intervention result, so that readers do not attribute the draft-count and max-debug-depth effects to TS alone.","section":"Section 4.4, Tables 12-13 and Table 16"}],"minor_comments":[{"comment":"Please define 'step 0' explicitly in the table caption and text. The current phrasing 'Fraction of seeds with ≥1 valid submission by step N' is ambiguous about whether step 0 is the initial draft before any execution or the first executed node, which is central to interpreting the 69% S0 value.","section":"Table 5 and Figure 5"},{"comment":"The gold-count comparisons (e.g., 22 vs. 38 for AIDE) are reported without uncertainty or a significance test. Since gold medals are derived from thresholds on the leaderboard, a per-competition Fisher exact test or a permutation test across seeds would strengthen the claim that the intervention changes medal counts.","section":"Tables 1 and 2"},{"comment":"The pooled correlation is computed after z-normalizing within each competition; please state this explicitly in the main text or table caption and provide the number of seeds per competition, since the per-competition correlations range from r=-0.27 to r=+0.63.","section":"Appendix B.5, Table 10"},{"comment":"The footnote notes that the ML-Master baseline in the EDA study was based on 'a different set of runs than the baseline used for the debug consultant.' Please clarify how this affects comparability and whether the baseline conditions were otherwise identical.","section":"Appendix C, Table 18 footnote"},{"comment":"The HPO quality scorer uses gpt-4o-2024-08-06 as the judge while the agent backbone is GPT-5-mini. Please confirm that the judge is independent of the agent's own model and discuss any risk of the judge favoring code patterns it recognizes from its training data.","section":"Section 4.3 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The most urgent issue is the internal contradiction regarding the debug consultant's registry initialization, combined with the missing code link. If the code release confirms that the registry is pre-seeded with environment-specific API incompatibilities, the paper would still be a useful empirical study of environment-aware prompting, but its abstract and framing as 'adaptive learning' and 'agentic design alone' would need substantial revision. The controlled TS comparison and the detailed per-seed data are strong assets, and the ML-Master HPO failure is reported honestly. I would like to see a revised version that either clarifies and justifies the initialization, or re-scopes the claims, before I can recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this paper for the debug consultant alone, and you should read it with the stress-test note in hand. The core mechanism is a shared registry of runtime constraints across a tree search, which is a genuinely new and useful design. The evidence that it works is rich: redundant bug encounters drop from 46% to 7.8%, first-attempt fix rates double, and the two case studies include full generated code showing how valid-node budget translates into ensembles and calibration. The ML-Master HPO degradation is reported honestly and is itself a finding about scaffold-transfer. The MLEvolve controlled TS study is clean and supports a stability claim. Those parts deserve credit.\n\nBut the main experiment has a load-bearing ambiguity. Section 3.1 describes a within-run registry that starts empty and accumulates errors as nodes crash. Section 4.2 says the consultant supplies the environment's constraints before the agent writes its first line of code, and Table 5 shows 69% of treatment seeds already have a valid submission at step 0 versus 3% for baseline. A within-run registry cannot explain that. Either the registry is pre-seeded with known environment facts (the LightGBM early_stopping_rounds incompatibility being the obvious candidate), or there is an undocumented pre-flight probe. The paper does not say. The 'Code available here' link points nowhere, so the initialization cannot be checked. This matters because the headline claim—that agentic design alone, via adaptive learning, recovers wasted compute—collapses to environment-specific prompt engineering if the BANNED list is seeded. This is not a minor omission; it is the difference between the paper's central contribution and a much less interesting result. The paper itself contains the contradiction in Section 4.2 versus Section 3.1, so either the text is misleading or the mechanism is different from what is described.\n\nOther soft spots are proportionate. The AIDE Thompson Sampling headline conflates the selection strategy with raising initial drafts from 5 to 20; the controlled comparison shows TS mainly reduces null runs rather than raising peak scores, which is a weaker but still defensible claim. Single-backbone evaluation on GPT-5-mini is a real limitation, and the paper acknowledges cost constraints. These are minor-to-moderate.\n\nMy bottom line: this deserves a serious referee, but only conditionally. The debug consultant idea should see the light of day; the ambiguity about registry initialization is checkable and fixable. I would ask the authors to describe exactly what is in the registry at step 0, release the code, and run an ablation with a truly empty registry. If the 69%-at-S0 effect disappears, the paper becomes a study in environment-specific prompting, still useful but not the contribution it claims. If the effect persists, the paper is a solid systems result. Either way, it is worth engaging with, provided the authors answer the initialization question before publication.","headline":"The debug consultant is a promising idea, but the flagship numbers depend on an unexplained initialization that the paper must disclose; the rest of the empirical work is solid, mostly honest, and worth engaging with.","tokens_in":35561,"tokens_out":2806,"would_cite":false,"duration_ms":29761,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Autoresearch agents waste compute on re-solving the same bugs, and this paper shows a shared debug consultant that spreads each lesson across the search tree recovers that waste, nearly doubling MLE-bench gold medals with the language…","keywords":["autoresearch","agentic data science","tree search agents","debug consultant","context sharing","hyperparameter tuning enforcement","Thompson sampling","MLE-bench"],"falsifier":"Run the debug consultant on the same nine MLE-bench tasks with a different backbone model, or with a LightGBM version where early_stopping_rounds is a valid argument: if redundant bug encounters do not collapse and the gold-medal gain (22 to 38 on AIDE) does not persist, the headline effect is an artifact of the single environment. A complementary test is to choose tasks where no single API error dominates, since the claim that sharing failure knowledge is generally valuable predicts the consultant should still reduce redundant debugging and improve valid-node rates there.","tokens_in":2026,"feed_emoji":"🐛","tokens_out":2669,"duration_ms":88008,"temperature":0.7,"pith_summary":"Autoresearch agents—LLM systems that write code to solve machine-learning tasks end-to-end—are receiving large industry investment, but much of the compute they consume is wasted rather than spent. This paper identifies four structural ways tree-search agents waste budget: re-discovering the same runtime bugs in every branch, stopping search before tuning hyperparameters, failing to explore the candidate space, and ignoring the data analysis they perform. The paper's central claim is that these failures are fixable by design alone: a 'debug consultant' that compresses each crash into a shared bug registry and injects the distilled lessons (banned patterns and proven fixes) into every branch, plus reward shaping that enforces hyperparameter tuning and Thompson Sampling that backtracks out of failing paths. Holding the language model fixed, these interventions lift AIDE's gold medals from 22 to 38 out of 90 runs on MLE-bench and raise its valid-submission rate from 81% to 100%, with comparable gains for ML-Master. If right, the paper implies current agents operate far below the ceiling their language models already permit, and that memory, reliability, and exploration belong on the agent-design agenda.","feed_headline":"Sharing bugs across the search tree doubles gold medals","feed_subtitle":"The same language model lifts MLE-bench golds from 22 to 38 once branches learn from each other's failures.","key_machinery":"The central object is the debug consultant: a shared registry that turns each node's crash into a compressed record, accumulates these records across the whole search tree, distills them into a short list of banned patterns and proven fixes, and injects that list into the prompt before every draft and debug step—so the agent learns its execution environment (library versions, valid API signatures) once instead of per branch. The registry is built by a three-step control loop (error compression, shared registry, constraint injection) plus deterministic rules that treat timeouts and empty logs as terminal dead ends. Two supporting mechanisms carry the other results: a 0–3 hyperparameter-tuning score assigned by an LLM judge and folded into node selection (an additive metric adjustment in AIDE, a $+0.25 \\times \\text{hpo\\_score}$ term in ML-Master's UCT reward), and Thompson Sampling in which each sibling node holds a $\\text{Beta}(\\alpha, \\beta)$ distribution over quality—updated by normalized rewards after each execution—with backtracking to the branch point where a repeated error first appeared.","core_discovery":"The discovery, stated on the paper's own terms, is that a large fraction of an autoresearch agent's budget is consumed by failures the agent has already met before, and that making failure knowledge global rather than local converts that wasted compute into productive iteration. The debug consultant does this by compressing each node's traceback into a short record (error type, signature, failed strategy), accumulating records across all branches into a registry that distills 'BANNED' patterns and 'PROVEN FIX' replacements, and injecting the distillation into every subsequent generation and debug step; within one run, a single branch's crash becomes every branch's lesson. The measurable consequences in AIDE's search journals are that redundant bug encounters fall from 46.0% to 7.8% of nodes, the valid-node rate rises from 54.7% to 79.0%, the median step to a first valid submission drops from 6 to 0, and seeds with more valid nodes score better (pooled $r = +0.22$). The same philosophy extends to the other interventions: hyperparameter-tuning guidance, enforced through an LLM-judged 0–3 score folded into the search reward, recovers up to +0.388 on a single competition, and Thompson Sampling with backtracking cuts null runs from 33 to 15 of 90 on AIDE while largely preserving peak scores.","pith_inferences":["The paper's own asymmetry result—the same HPO intervention that helps AIDE degrades ML-Master—suggests the headline debug-consultant gains may be less universal than the framing implies: with one environment and one backbone, the consultant may be partly memorizing a single dominant bug. A direct extension would be to test the consultant across backbones and library versions, separating environmen","The pooled correlation between valid-node count and final score ($r = +0.22$) suggests the operative mechanism is simply more successful iterations per budget; if so, any intervention that raises the valid-node rate—cheaper evaluation, faster execution, better debugging—should compound into better final solutions, a testable prediction the paper does not state.","Because the consultant's knowledge is environment-level (installed versions, valid API signatures), the registry is naturally reusable across runs: persisting it between runs of the same benchmark would turn each new task into a warm start, an extension the paper does not explore.","The adversarial-EDA diagnostic is a negative result with a constructive counterpart the paper leaves implicit: a control loop that forces the agent to consume its own EDA, for instance by requiring a feature-selection justification before training, could test whether EDA becomes valuable once adherence is enforced."],"forward_implications":["With the language model held fixed, purely structural changes to the agent can nearly double medal-level results on MLE-bench tabular tasks: AIDE golds rise 22 to 38 with 100% valid submissions, and ML-Master golds rise 18 to 29.","An agent that shares failure knowledge across the tree reaches a working solution at median step 0 and spends its remaining budget on real modeling iteration, producing qualitatively more sophisticated solutions such as calibrated ensembles and optimized thresholds where the baseline produced single models.","Enforcing structured hyperparameter tuning through the reward function recovers performance wherever agents stop early, with gains up to +0.388 on one competition; because the same intervention hurts ML-Master by pushing it into a crashing HPO implementation, scaffold interventions must be designed with the target agent's memory architecture in mind.","Thompson Sampling with backtracking primarily buys reliability: null runs drop from 33 to 15 of 90 on AIDE at negligible cost to peak scores.","Because baseline agents ignore even adversarially injected exploratory data analysis (it changed feature selection in only 5% of AIDE runs), closing the EDA-to-decision gap is a concrete remaining design target rather than an already-solved stage of the pipeline."],"supporting_citations":[{"why":"Supplies MLE-bench, the benchmark and medal-grading system against which every headline number (golds, valid-submission rate) is measured.","marker":"Chan et al., 2024"},{"why":"AIDE is the primary scaffold; the debug consultant and hyperparameter-tuning interventions are built into it and produce the main results.","marker":"Jiang et al., 2025"},{"why":"ML-Master is the second scaffold; its memory architecture is the load-bearing explanation for why the same HPO intervention fails there.","marker":"Liu et al., 2025"},{"why":"Provides the Agentic Context Engineering idea of treating context as an evolving playbook, which the debug consultant operationalizes into a shared bug registry.","marker":"Zhang et al., 2025"},{"why":"MLEvolve serves as the held-fixed-configuration testbed that isolates Thompson Sampling's contribution from the other hyperparameter changes.","marker":"Du et al., 2026"},{"why":"Documents agents getting trapped in dead-end solution paths on MLE-bench, motivating the failure-mode taxonomy the interventions target.","marker":"Toledo et al., 2025"}],"fun_headline_variants":["Cross-branch bug sharing doubles gold medals","Global debug consultant recovers wasted compute","Autoresearch agents recover compute by sharing bugs","One branch's bug becomes every branch's fix","Wasted compute reclaimed by cross-branch bug registry"],"cache_read_input_tokens":37760,"weakest_assumption_plain":"Every experiment uses the same single language model, GPT-5-mini, in one fixed environment where one particular LightGBM API incompatibility (early_stopping_rounds raising a TypeError) accounts for most baseline failures, so the large measured gains could be largely a product of that specific environment and model rather than of the interventions themselves.","fun_headline_variants_meta":{"raw":{"variants":["Cross-branch bug sharing doubles gold medals","Global debug consultant recovers wasted compute","Autoresearch agents recover compute by sharing bugs","One branch's bug becomes every branch's fix","Wasted compute reclaimed by cross-branch bug registry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001218,"raw_usage":{"total_tokens":5049,"prompt_tokens":1024,"completion_tokens":4025,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":3956}},"tokens_in":640,"tokens_out":4025,"duration_ms":25436,"temperature":1.0,"reasoning_tokens":3956,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:20:22.963368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the debug consultant on the same nine MLE-bench tasks with a different backbone model, or with a LightGBM version where early_stopping_rounds is a valid argument: if redundant bug encounters do not collapse and the gold-medal gain (22 to 38 on AIDE) does not persist, the headline effect is an artifact of the single environment. A complementary test is to choose tasks where no single API error dominates, since the claim that sharing failure knowledge is generally valuable predicts the consultant should still reduce redundant debugging and improve valid-node rates there.","supporting_citations":[{"cited_title":"2025 , url=","cited_arxiv_id":null,"evidence_quote":"AIDE is the primary scaffold; the debug consultant and hyperparameter-tuning interventions are built into it and produce the main results."},{"cited_title":"2025 , url=","cited_arxiv_id":null,"evidence_quote":"ML-Master is the second scaffold; its memory architecture is the load-bearing explanation for why the same HPO intervention fails there."},{"cited_title":"2025 , url=","cited_arxiv_id":null,"evidence_quote":"Provides the Agentic Context Engineering idea of treating context as an evolving playbook, which the debug consultant operationalizes into a shared bug registry."}],"review_version":1}