Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

No fixed harness is universally best for LLM-guided discovery; adaptive online selection beats fixed recipes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:35 UTC pith:FNNTEUVA

load-bearing objection A careful large-scale negative result plus reusable run pools; the adaptive-allocation claim is real but needs an out-of-sample test. the 2 major comments →

arxiv 2607.18235 v1 pith:FNNTEUVA submitted 2026-07-20 cs.CL cs.AI

Automated Discovery Has No Universally Superior Harness

Akshat Gupta , Jermaine Lei , Alexander Lu , Gopala Anumanchipalli , Leshem Choshen This is my paper
classification cs.CL cs.AI
keywords LLM-guided discoverydiscovery harnessesevolutionary searchtree searchgeneralizationhyperparameter selectionadaptive allocationstatistical evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the composite harnesses that steer LLM-guided discovery—the recipes deciding which candidate programs to mutate and expand—do not transfer across models and problems. Decomposing two popular harness families into their constituent design choices, the authors compare 30 budget-matched harnesses across 12 model–problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical tests. They find that no fixed harness is reliably superior, that full OpenEvolve-style variants generally underperform simpler alternatives, and that the best harness changes with the model and task. They also show that partial-run performance becomes strongly predictive of final outcomes by the halfway point, and exploit this in a budget-matched adaptive-allocation policy that prunes weak runs and reallocates compute to survivors, beating both commitment to a random fixed harness and a non-adaptive ensemble. The practical upshot: harness choice is better viewed as a model- and problem-dependent hyperparameter, to be set or adapted online, than as a universal recipe.

Core claim

On its own terms, this paper's central claim is that discovery harnesses have a generalization problem: across the 12 evaluated model–problem pairs (four LLMs from 3B to 120B parameters and three mathematical discovery tasks), no fixed harness is reliably superior to a budget-matched Sequential Best-of-N baseline, and OpenEvolve-style variants generally rank below simpler lightweight-exploration and PUCT-style alternatives. The evidence base is more than 3.1 million budget-matched LLM rollouts, five independent runs per harness, 100-run (or 30-run) baseline pools, bootstrap and permutation tests, and a cross-pair majority-win statistic. The paper further claims that early discovery progress

What carries the argument

The load-bearing machinery is a decomposition of composite discovery harnesses into individual design choices, grafted one at a time onto a greedy Sequential Best-of-N baseline: elite-archive size K, epsilon-greedy exploration probability, breadth–depth budget allocation (N children vs. T iterations), MAP-Elites/inspiration sampling and islands (OpenEvolve branch), and UCT/PUCT subtree-value parent selection with multiple parents (TTT-Discover branch). This lets the authors attribute gains and losses to specific choices rather than to whole recipes. Around it stands a statistical apparatus: repeated Sequential BoN reference pools, best-of-five bootstrap tests against that null, a cross-pair

Load-bearing premise

The suite of 12 model–problem pairs is built from only three mathematical tasks and four LLMs, all in the same evolutionary-search ecosystem; the claim that no fixed harness is universally superior presumes these pairs represent discovery problems broadly enough that a transfer failure here indicates a general failure.

What would settle it

A budget-matched, repeated-trial evaluation on discovery problems outside these three tasks—say, code generation, scientific equation discovery, or non-mathematical optimization—that finds a fixed harness significantly outperforming Sequential Best-of-N on a majority of pairs after multiplicity correction would refute the no-universal-harness claim. A second, independent refutation would be a task family where early-run scores do not rank-correlate with final outcomes above chance, which would undercut the adaptive-allocation rationale.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Treating harness choice as a hyperparameter means single-harness evaluations are insufficient; future discovery systems should be benchmarked across models and problems, not a single pair.
  • The released run pools and null distributions give future harness proposals a reusable statistical baseline, letting claims of improvement be tested against empirical reference distributions rather than a few runs.
  • Since early progress is predictive by the halfway point, practitioners can safely prune weak discovery runs at 25–50% checkpoints and redirect compute to stronger trajectories without hurting expected outcomes.
  • The budget-matched adaptive schedule (e.g., 12→5→2→1) yields higher average scores than spending the same compute on five full runs of one harness, so online adaptation is a concrete way to spend a fixed discovery budget.
  • Because OpenEvolve-style complexity often reversed gains from simpler intermediate variants, simpler harnesses with one or two added components (lightweight exploration or PUCT) may capture most of the available benefit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the generalization failure extends beyond mathematics, the same non-transfer is likely to appear in code generation, scientific modeling, and non-mathematical search—an empirical prediction that a broader task suite could test.
  • The adaptive-allocation result is essentially Successive Halving applied to harnesses; an implication the author leaves implicit is that the policy itself (start breadth, checkpoint positions, survivor counts) is a tunable object, so meta-optimizing the schedule may push average scores further.
  • The reported margins are modest in absolute score (about 1.4 points over Sequential BoN on average); whether they matter will depend on whether such differences compound across repeated discoveries or on harder problems, where the best harness is unknown in advance.
  • The paper's framing suggests a two-level search: the outer loop selects harnesses while the inner loop searches solutions; if true, discovery systems should be built with explicit outer-loop adaptation rather than a permanently fixed harness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies whether the choice of search harness in LLM-guided discovery transfers across models and problems. The authors decompose OpenEvolve-style and TTT-Discover-style harnesses into components, evaluate 30 budget-matched configurations on 12 model–problem pairs (4 LLMs × 3 mathematical tasks), and report three findings: (1) no fixed harness is significantly better than Sequential BoN after multiple-testing correction; (2) partial-run scores correlate with final performance, especially by 50% of the budget; (3) a budget-matched adaptive ensemble that prunes weak partial runs and reallocates compute outperforms single-harness commitment and an unpruned ensemble. The paper releases run pools and baseline null distributions.

Significance. This is a substantial empirical contribution: more than 3.1 million rollouts, matched budgets, repeated trials, bootstrap and permutation tests with 100,000 resamples, Holm correction, and public release of run pools. The negative result on fixed-harness generalization, if accepted, is useful for the field and supports treating harness choice as a hyperparameter. The adaptive-allocation result would be a valuable positive contribution, but it currently rests on an in-sample evaluation. The paper's strengths—careful statistical machinery, honest reporting of non-significance after correction, and release of reusable null distributions—are real and should be credited.

major comments (2)
  1. [Section 5.2, Table 1, Appendix G, Algorithm 3] The adaptive-ensemble evaluation is in-sample in a way that undermines Finding 3. The policy-level simulations resample with replacement from the same five-run pools that define the policy. With n=5 per harness, the empirical distribution of each harness is coarsely estimated, and the pruning rule selects runs that are lucky at the checkpoint; the final score of a retained survivor comes from the same run whose partial score drove the pruning. The reported SE (0.02) is Monte Carlo error conditional on the pools, not the sampling error of the pools themselves. The advantage of the best adaptive schedule over the unpruned portfolio (85.75 vs 84.54) is small and could plausibly shrink or reverse under a proper out-of-sample protocol. I recommend an out-of-sample evaluation (e.g., leave-one-run-out or a fresh run pool for each harness) or at minimum a bias-corrected estimate explicitly accou
  2. [Finding 3 / Section 5.2 / Appendix G] The claim that the adaptive ensemble outperforms the unpruned harness portfolio is based on point estimates only. The only vector-battle significance test reported is against the Sequential BoN reference (Appendix G). No test or confidence interval is given for the adaptive-vs-unpruned difference, which is the key control isolating the value of pruning. Given the small margins and the in-sample issue above, this comparison needs a direct significance test or interval, and the claim should be tempered until such evidence is provided.
minor comments (6)
  1. [Title / Abstract] The word "universally" overstates the scope. The evidence covers 12 model–problem pairs from three mathematical tasks; the abstract's own qualification "across the evaluated model–problem pairs" is more accurate. Suggest aligning the title with the actual scope.
  2. [Abstract / Section 1] The abstract says "We release all run pools" while the introduction says "we will release"; make the tense consistent.
  3. [Section 3.3 / Eq. (5)] In the text after Eq. (5), "ˆpcross(h)(h)" has a duplicated argument; the equation itself is clear.
  4. [Table 1 vs Appendix G] Main-text Table 1 uses column labels CP Q3B, CP Q4B, etc., while Appendix G uses Q2.5, Q3, GPT-20B. Use consistent model naming to avoid confusion.
  5. [Section 3.2] The five-run protocol is acknowledged as larger than prior work, but the low power of best-of-five significance tests should be noted more prominently when interpreting the non-significant cross-pair results. Absence of evidence after Holm correction is not strong evidence of absence.
  6. [Abstract / Section 1] The phrase "reusable statistical infrastructure against for future harness proposals" is garbled; should be "against which future harness proposals can be compared".

Circularity Check

0 steps flagged

No load-bearing circular step; the paper's claims are empirical comparisons against budget-matched baselines.

full rationale

The paper's central claims — that no fixed harness is universally superior, that early progress is informative, and that adaptive allocation improves average performance — are empirical results rather than derivations that reduce to their own inputs. The fixed-harness comparisons use separate baseline pools and bootstrap/permutation tests against Sequential BoN, and the strongest observed fixed configuration is explicitly reported as non-significant after Holm correction (Section 4.2), the opposite of a self-fulfilling claim. The adaptive-allocation experiment replays a generic pruning policy over released run pools: the policy prunes using partial-run scores, and the reported final score is the final score of the same replayed run. This is an in-sample policy evaluation with a small number of empirical runs per harness, which limits external validity, but it is not equivalent by construction to the claimed outcome; if early and final scores were uncorrelated, selection on early scores would not systematically improve average final scores. No load-bearing uniqueness theorem or ansatz is imported from the authors' prior work. The only apparent self-citation (AdaEvolve, Cemri et al. 2026, which shares an author) appears in related work and is not used to justify the paper's conclusions. I therefore find no circular step.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claim is empirical and does not rest on fitted free parameters or invented entities. The main assumptions are representativeness of the task/model suite, faithfulness of the harness decomposition, validity of the bootstrap/permutation nulls, and the use of resampled trajectories as a proxy for fresh runs in the adaptive-allocation evaluation.

axioms (4)
  • domain assumption The three chosen tasks are representative of LLM-guided discovery problems.
    The general claim about no universally superior harness relies on the 12 model–problem pairs spanning the relevant space of discovery tasks; Section 3.1 and Appendix C define the tasks.
  • domain assumption The component-wise reconstruction faithfully captures OpenEvolve and TTT-Discover search harnesses.
    Section 2 and Appendix B decompose the two systems into archive, exploration, breadth–depth, MAP-Elites, islands, UCT/PUCT, and multi-parent components; if the reconstructions omit a load-bearing ingredient, the negative conclusion about the full recipes may not apply to the real systems.
  • standard math Bootstrap resampling and permutation tests provide valid null distributions for the heavy-tailed final-score data.
    Section 3.2 and 3.3 rely on the bootstrap and permutation procedures to estimate p-values; these are standard methods but depend on the exchangeability of runs within each model–problem pair.
  • domain assumption Resampling full trajectories with replacement from the available run pools is a valid way to estimate the expected performance of allocation policies.
    Section 5.2 and Appendix G evaluate adaptive policies by replaying stored runs; this assumes the stored trajectories are representative of new runs of the same harnesses, which is not separately tested with fresh runs.

pith-pipeline@v1.3.0-alltime-deepseek · 180 in / 7193 out tokens · 92793 ms · 2026-08-01T15:35:37.772672+00:00 · methodology

0 comments
read the original abstract

Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components and systematically evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the specific problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure against for future harness proposals.

Figures

Figures reproduced from arXiv: 2607.18235 by Akshat Gupta, Alexander Lu, Gopala Anumanchipalli, Jermaine Lei, Leshem Choshen.

Figure 1
Figure 1. Figure 1: Harnesses in discovery systems are composite recipes whose ingredients do not transfer uniformly. Panel (a) shows how we decompose two representative discovery harness families into building blocks, diverging from the same root of greedy Sequential Best-of-N. Panel (b) summarizes the cross-pair majority-win analysis across all 12 model–problem pairs used in this paper, using Sequential BoN as the baseline.… view at source ↗
Figure 2
Figure 2. Figure 2: Archive and Exploration Ablations. The top row varies only Top-K archive size with ϵ = 0%; the bottom row fixes K = 1 and varies exploration probability ϵ. Nominal pair-level results with pˆh < 0.05 are indicated at the bottom of the plot. Larger archives alone yield no significant gains, while exploration helps some model–problem pairs but hurts others. a single bootsrap trial, we define the cross-pair ma… view at source ↗
Figure 3
Figure 3. Figure 3: Progression toward OpenEvolve-style search. Configurations progressively shift budget from breadth to depth and add MAP-Elites, inspiration sampling, and multiple islands. Added complexity does not produce monotonic gains, and full OpenEvolve-style variants often underperform simpler intermediates [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: TTT-Discover-style ablations. Configurations vary tree-search policy, branching width, and the number of parents expanded per iteration. Tree search improves selected pairs, especially Heilbronn Triangle, but later TTT-Discover-style additions do not consistently help. 4 Fixed Harnesses Do Not Generalize Uniformly Across Model–Problem Pairs The analyses described in the previous section answer complementar… view at source ↗
Figure 5
Figure 5. Figure 5: Fixed harnesses do not reliably generalize, but early progress is informative. Panel (a) shows the cross-pair majority-win statistic Pˆmaj for representative fixed harnesses across all 12 model–problem pairs along with the cross-pair p-values before and after Holm’s correction. Panel (b) shows checkpoint-to￾final rank correlation, motivating early-feedback-based allocation. 5.1 Early Discovery Progress Pre… view at source ↗
Figure 6
Figure 6. Figure 6: Adaptive allocation schematic. The comparison is budget matched across fixed repeated-run baselines, a non-adaptive full ensemble, single-stage Adaptive Harness Ensemble, and multi-stage Adaptive Harness Ensemble. Starting with the full-ensemble baseline, different hues denote distinct search configura￾tions; in the Adaptive Harness Ensemble rows, retained configurations keep their hue after pruning. Algor… view at source ↗
Figure 7
Figure 7. Figure 7: Progression from Sequential BoN to OpenEvolve-style search. Sequential BoN samples the parent from the top-1 elite archive E 1 t . Archive sampling generalizes this to the top-K elite archive E K t and, with probability ϵ, the full history Ht; each generated child is inserted into Ht+1 and also enters E K t+1 if it is elite. Holding the total generation budget B = NT fixed and decreasing N shifts search to… view at source ↗
Figure 8
Figure 8. Figure 8: UCT/PUCT as tree-based search. Sequential BoN selects pt using only the highest evaluator score in the full history Ht. UCT and PUCT instead score each program x ∈ Ht using the induced mutation tree: Qt(x) propagates the best score in the descendant set Dt(x), visitation counts adjust exploration, and PUCT additionally uses the rank-based prior Pt(x). The LLM generation step and history update remain the s… view at source ↗
Figure 9
Figure 9. Figure 9: Representative discovery tasks. The tasks include two geometric optimization problems and one non-geometric autocorrelation objective. • Heilbronn Triangle. The goal is to place points inside an equilateral triangle while maximizing the minimum area of any triangle formed by three points. The objective is globally coupled because moving one point can affect many of the induced triangles. • Second Autocorre… view at source ↗
Figure 10
Figure 10. Figure 10: Progression toward OpenEvolve-style search on circle packing. The four panels show the model-specific breadth–depth progression and available OpenEvolve variants. (e) Qwen2.5-3B-Instruct. (f) Qwen3-4B-Instruct. (g) GPT-OSS-20B. (h) GPT-OSS-120B [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Progression toward OpenEvolve-style search on the second autocorrelation inequal￾ity task, continued. Intermediate breadth–depth allocations can be competitive, while the OpenEvolve endpoint does not uniformly dominate across models and tasks. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Full epsilon-greedy grid across all three tasks. Each panel varies elite archive size and ϵ probability, with model-specific score distributions overlaid. D.3 Sequential BoN Baseline We first evaluate a Sequential Best-of-N (BoN) baseline for each model–problem pair. Sequential BoN is the K = 1, ϵ = 0 special case of the archive formulation in Equation 9: it always expands the best program found so far an… view at source ↗
Figure 12
Figure 12. Figure 12: Sequential BoN baseline distributions. Kernel-density estimates show final scores for each task and model. Dashed vertical lines mark model-specific means, and rug marks show individual runs. These 100-run distributions define the empirical reference used in the bootstrap comparisons. GPT-OSS models, and 100 runs for Qwen models) as the reference for the archive, exploration, UCT/PUCT, and OpenEvolve-styl… view at source ↗
Figure 13
Figure 13. Figure 13: Elite-archive sweep across all three tasks. Parent programs are sampled from a top-K archive under ϵ = 0%. Colored stars denote configurations whose five-run maximum significantly exceeds the bootstrapped Sequential BoN reference. The absence of a star indicates insufficient evidence under this test, not evidence of no effect. D.5 Adding Exploration Experimental setup. We isolate exploration by fixing the… view at source ↗
Figure 14
Figure 14. Figure 14: General-history exploration sweep across all three tasks. The elite archive is fixed at K = 1, while the epsilon probability is increased from 0% to 30%. Higher epsilon probability corresponds to more frequent sampling from the full program history. rather than a uniformly better parent-selection rule. Exploration may help recover diverse lineages, but it also spends budget on programs that may be less re… view at source ↗
Figure 15
Figure 15. Figure 15: UCT/PUCT exploration-constant ablations across all three tasks. PUCT and UCT are shown side by side for each value of C, with model-specific score distributions overlaid. The preferred exploration strength varies across model–problem pairs. For Heilbronn triangle, changing the breadth–depth allocation has a smaller and less consistent effect. GPT￾OSS-20B remains strong across the sweep, but the Qwen model… view at source ↗
Figure 16
Figure 16. Figure 16: UCT/PUCT breadth–depth ablations. The number of children generated per selected parent varies while the total rollout budget remains fixed. No single allocation is best across all model– problem pairs. produces only modest and model-dependent shifts. These results suggest that increasing the number of root programs is not by itself sufficient to improve search quality. The benefit depends on whether each … view at source ↗
Figure 17
Figure 17. Figure 17: UCT/PUCT parallel-parent ablations. Runs are grouped by the number of selected parent programs, P. Because total budgets differ by model, configurations are aligned by P rather than by an identical child batch size. This maps the baseline to 0, configurations that always beat the baseline to +1, and configurations that never beat the baseline to −1. The plotted points are ordered to match the experimental… view at source ↗
Figure 18
Figure 18. Figure 18: Full cross-pair majority-win probability leaderboard (max statistic). All 30 configura￾tions ranked by battle win rate using the max score across all dimensions of the config vector. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Full cross-pair majority-win probability leaderboard (median statistic). All 30 config￾urations ranked by battle win rate using the median score across dimensions. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Full cross-pair majority-win probability leaderboard (mean statistic). All 30 configu￾rations ranked by battle win rate using the mean score across dimensions. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficiency Matters in Autonomous Research

    cs.AI 2026-07 conditional novelty 5.5

    On twelve code-optimization tasks, no fixed search structure wins on efficiency, and fluid—a UCB portfolio over hill-climbing chains—nearly matches a per-task oracle on AUC of the reward frontier.

Reference graph

Works this paper leans on

70 extracted references · 26 linked inside Pith · cited by 1 Pith paper

  1. [7]

    Algorithm portfolios

    Carla P Gomes and Bart Selman. Algorithm portfolios. Artificial Intelligence, 126 0 (1-2): 0 43--62, 2001

    2001

  2. [8]

    Deep reinforcement learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  3. [9]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, volume 2025, pp.\ 21344--21377, 2025

    2025

  4. [11]

    Non-stochastic best arm identification and hyperparameter optimization

    Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics, pp.\ 240--248. PMLR, 2016

    2016

  5. [16]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesv \'a ri. Bandit based monte-carlo planning. In European conference on machine learning, pp.\ 282--293. Springer, 2006

    2006

  6. [19]

    Abandoning objectives: Evolution through the search for novelty alone

    Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19 0 (2): 0 189--223, 2011

    2011

  7. [20]

    Random search and reproducibility for neural architecture search

    Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pp.\ 367--377. PMLR, 2020

    2020

  8. [21]

    A system for massively parallel hyperparameter tuning

    Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. Proceedings of machine learning and systems, 2: 0 230--246, 2020

    2020

  9. [22]

    Hyperband: A novel bandit-based approach to hyperparameter optimization

    Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of machine learning research, 18 0 (185): 0 1--52, 2018

    2018

  10. [25]

    Eoh-s: Evolution of heuristic set using llms for automated heuristic design

    Fei Liu, Yilu Liu, Qingfu Zhang, Tong Xialiang, and Mingxuan Yuan. Eoh-s: Evolution of heuristic set using llms for automated heuristic design. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp.\ 37090--37098, 2026 a

    2026

  11. [33]

    Qwen2.5 technical report, 2025

    Qwen: , An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, ...

    Pith/arXiv arXiv 2025

  12. [34]

    The algorithm selection problem

    John R Rice. The algorithm selection problem. In Advances in computers, volume 15, pp.\ 65--118. Elsevier, 1976

    1976

  13. [35]

    Mathematical discoveries from program search with large language models

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625 0 (7995): 0 468--475, 2024

    2024

  14. [38]

    Agentsquare: Automatic llm agent search in modular design space

    Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic llm agent search in modular design space. In International Conference on Learning Representations, volume 2025, pp.\ 3841--3865, 2025

    2025

  15. [39]

    OpenEvolve : An open-source evolutionary coding agent

    Asankhaya Sharma. OpenEvolve : An open-source evolutionary coding agent. GitHub repository, 2025. URL https://github.com/algorithmicsuperintelligence/openevolve

    2025

  16. [40]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550 0 (7676): 0 354--359, 2017

    2017

  17. [46]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, volume 2024, pp.\ 12028--12068, 2024

    2024

  18. [47]

    Reevo: Large language models as hyper-heuristics with reflective evolution

    Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. Reevo: Large language models as hyper-heuristics with reflective evolution. Advances in neural information processing systems, 37: 0 43571--43608, 2024

    2024

  19. [50]

    arXiv preprint arXiv:2506.13131 , year=

    Alphaevolve: A coding agent for scientific and algorithmic discovery , author=. arXiv preprint arXiv:2506.13131 , year=

    Pith/arXiv arXiv

  20. [51]

    arXiv preprint arXiv:2509.19349 , year=

    Shinkaevolve: Towards open-ended and sample-efficient program evolution , author=. arXiv preprint arXiv:2509.19349 , year=

    Pith/arXiv arXiv

  21. [52]

    Nature , volume=

    Mathematical discoveries from program search with large language models , author=. Nature , volume=. 2024 , publisher=

    2024

  22. [53]

    arXiv preprint arXiv:2601.16175 , year=

    Learning to discover at test time , author=. arXiv preprint arXiv:2601.16175 , year=

    Pith/arXiv arXiv

  23. [54]

    arXiv preprint arXiv:2602.20133 , year=

    Adaevolve: Adaptive llm driven zeroth-order optimization , author=. arXiv preprint arXiv:2602.20133 , year=

    arXiv

  24. [55]

    Sharma, Asankhaya , year =

  25. [56]

    arXiv preprint arXiv:2602.02919 , year=

    DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolution , author=. arXiv preprint arXiv:2602.02919 , year=

    arXiv

  26. [57]

    nature , volume=

    Mastering the game of go without human knowledge , author=. nature , volume=. 2017 , publisher=

    2017

  27. [58]

    European conference on machine learning , pages=

    Bandit based monte-carlo planning , author=. European conference on machine learning , pages=. 2006 , organization=

    2006

  28. [59]

    arXiv preprint arXiv:2602.16805 , year=

    Simple Baselines are Competitive with Code Evolution , author=. arXiv preprint arXiv:2602.16805 , year=

    arXiv

  29. [60]

    arXiv preprint arXiv:2603.28052 , year=

    Meta-harness: End-to-end optimization of model harnesses , author=. arXiv preprint arXiv:2603.28052 , year=

    Pith/arXiv arXiv

  30. [61]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  31. [62]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  32. [63]

    arXiv preprint arXiv:2508.10925 , year=

    gpt-oss-120b & gpt-oss-20b model card , author=. arXiv preprint arXiv:2508.10925 , year=

    Pith/arXiv arXiv

  33. [64]

    Journal of Machine learning research , volume=

    Statistical comparisons of classifiers over multiple data sets , author=. Journal of Machine learning research , volume=

  34. [65]

    arXiv preprint arXiv:2602.23413 , year=

    Evox: Meta-evolution for automated discovery , author=. arXiv preprint arXiv:2602.23413 , year=

    arXiv

  35. [66]

    arXiv preprint arXiv:2606.09498 , year=

    Self-Harness: Harnesses That Improve Themselves , author=. arXiv preprint arXiv:2606.09498 , year=

    Pith/arXiv arXiv

  36. [67]

    arXiv preprint arXiv:2604.25850 , year=

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses , author=. arXiv preprint arXiv:2604.25850 , year=

    Pith/arXiv arXiv

  37. [68]

    arXiv preprint arXiv:2603.03329 , year=

    Autoharness: improving llm agents by automatically synthesizing a code harness , author=. arXiv preprint arXiv:2603.03329 , year=

    arXiv

  38. [69]

    arXiv preprint arXiv:2606.01779 , year=

    HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems , author=. arXiv preprint arXiv:2606.01779 , year=

    Pith/arXiv arXiv

  39. [70]

    arXiv preprint arXiv:2605.18747 , year=

    Code as Agent Harness , author=. arXiv preprint arXiv:2605.18747 , year=

    Pith/arXiv arXiv

  40. [71]

    2026 , publisher=

    Harness engineering for language agents: The harness layer as control, agency, and runtime , author=. 2026 , publisher=

    2026

  41. [72]

    International Conference on Learning Representations , volume=

    Large language models as optimizers , author=. International Conference on Learning Representations , volume=

  42. [73]

    arXiv preprint arXiv:2510.14150 , year=

    Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization , author=. arXiv preprint arXiv:2510.14150 , year=

    Pith/arXiv arXiv

  43. [74]

    arXiv preprint arXiv:2510.06056 , year=

    Scientific algorithm discovery by augmenting alphaevolve with deep research , author=. arXiv preprint arXiv:2510.06056 , year=

    arXiv

  44. [75]

    arXiv preprint arXiv:2511.17592 , year=

    GigaEvo: An Open Source Optimization Framework Powered By LLMs And Evolution Algorithms , author=. arXiv preprint arXiv:2511.17592 , year=

    arXiv

  45. [76]

    arXiv preprint arXiv:2507.19457 , year=

    Gepa: Reflective prompt evolution can outperform reinforcement learning , author=. arXiv preprint arXiv:2507.19457 , year=

    Pith/arXiv arXiv

  46. [77]

    Advances in neural information processing systems , volume=

    Reevo: Large language models as hyper-heuristics with reflective evolution , author=. Advances in neural information processing systems , volume=

  47. [78]

    arXiv preprint arXiv:2401.02051 , year=

    Evolution of heuristics: Towards efficient automatic algorithm design using large language model , author=. arXiv preprint arXiv:2401.02051 , year=

    Pith/arXiv arXiv

  48. [79]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Eoh-s: Evolution of heuristic set using llms for automated heuristic design , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  49. [80]

    arXiv preprint arXiv:1504.04909 , year=

    Illuminating search spaces by mapping elites , author=. arXiv preprint arXiv:1504.04909 , year=

    Pith/arXiv arXiv

  50. [81]

    Evolutionary computation , volume=

    Abandoning objectives: Evolution through the search for novelty alone , author=. Evolutionary computation , volume=. 2011 , publisher=

    2011

  51. [82]

    Advances in computers , volume=

    The algorithm selection problem , author=. Advances in computers , volume=. 1976 , publisher=

    1976

  52. [83]

    Artificial Intelligence , volume=

    Algorithm portfolios , author=. Artificial Intelligence , volume=. 2001 , publisher=

    2001

  53. [84]

    Artificial intelligence and statistics , pages=

    Non-stochastic best arm identification and hyperparameter optimization , author=. Artificial intelligence and statistics , pages=. 2016 , organization=

    2016

  54. [85]

    Journal of machine learning research , volume=

    Hyperband: A novel bandit-based approach to hyperparameter optimization , author=. Journal of machine learning research , volume=

  55. [86]

    Proceedings of machine learning and systems , volume=

    A system for massively parallel hyperparameter tuning , author=. Proceedings of machine learning and systems , volume=

  56. [87]

    arXiv preprint arXiv:2511.23473 , year=

    Thetaevolve: Test-time learning on open problems , author=. arXiv preprint arXiv:2511.23473 , year=

    Pith/arXiv arXiv

  57. [88]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Deep reinforcement learning that matters , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  58. [89]

    Uncertainty in artificial intelligence , pages=

    Random search and reproducibility for neural architecture search , author=. Uncertainty in artificial intelligence , pages=. 2020 , organization=

    2020

  59. [90]

    arXiv preprint arXiv:1912.12522 , year=

    NAS evaluation is frustratingly hard , author=. arXiv preprint arXiv:1912.12522 , year=

    Pith/arXiv arXiv 1912

  60. [91]

    arXiv preprint arXiv:2407.21787 , year=

    Large language monkeys: Scaling inference compute with repeated sampling , author=. arXiv preprint arXiv:2407.21787 , year=

    Pith/arXiv arXiv

  61. [92]

    arXiv preprint arXiv:2407.01502 , year=

    Ai agents that matter , author=. arXiv preprint arXiv:2407.01502 , year=

    Pith/arXiv arXiv

  62. [93]

    arXiv preprint arXiv:2605.09764 , year=

    LEVI: Stronger Search Architectures Can Substitute for Larger LLMs in Evolutionary Search , author=. arXiv preprint arXiv:2605.09764 , year=

    Pith/arXiv arXiv

  63. [94]

    arXiv preprint arXiv:2605.29268 , year=

    Compute Allocation in Evolutionary Search: From Depth-Breadth to Multi-Armed Bandits , author=. arXiv preprint arXiv:2605.29268 , year=

    Pith/arXiv arXiv

  64. [95]

    arXiv preprint arXiv:2605.15221 , year=

    Effective Harness Engineering for Algorithm Discovery with Coding Agents , author=. arXiv preprint arXiv:2605.15221 , year=

    Pith/arXiv arXiv

  65. [96]

    arXiv preprint arXiv:2604.20938 , year=

    Harbor: Automated harness optimization , author=. arXiv preprint arXiv:2604.20938 , year=

    Pith/arXiv arXiv

  66. [97]

    arXiv preprint arXiv:2605.09998 , year=

    Continual harness: Online adaptation for self-improving foundation agents , author=. arXiv preprint arXiv:2605.09998 , year=

    Pith/arXiv arXiv

  67. [98]

    arXiv preprint arXiv:2605.20086 , year=

    What Do Evolutionary Coding Agents Evolve? , author=. arXiv preprint arXiv:2605.20086 , year=

    Pith/arXiv arXiv

  68. [99]

    International Conference on Learning Representations , volume=

    Automated design of agentic systems , author=. International Conference on Learning Representations , volume=

  69. [100]

    International Conference on Learning Representations , volume=

    Agentsquare: Automatic llm agent search in modular design space , author=. International Conference on Learning Representations , volume=

  70. [101]

    arXiv preprint arXiv:2409.15254 , year=

    Archon: An architecture search framework for inference-time techniques , author=. arXiv preprint arXiv:2409.15254 , year=

    Pith/arXiv arXiv