{"id":"96c8121e-385f-4111-8657-a21f88a7a10c","arxiv_id":"2608.05651","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"A population handoff framework that uses a cheap LLM to explore and a strong LLM to refine, driven by a relay-gain reward, beats call-level routing across most tested evolution benchmarks.","lead":"The paper proposes RelayEvolve, a method that lets a cheap AI model explore many solution paths first and then hands the best, most varied candidates to a stronger AI model to refine. It reports the highest mean score in 11 of 12 benchmark settings, suggesting that in iterative search, budget should be allocated to the candidate population rather than to each individual call.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 11/12 headline rests on 3 runs per cell with no significance tests; several reported margins are far smaller than the stated run-to-run std (e.g., TXN 50%: +1.2 vs std ±153), so the central empirical claim is not yet established.","rationale":"The reader's stated weakest assumption is that Relay Gain is a valid reward for handoff timing. That is a legitimate design-level concern, and the ablations only probe it indirectly on Circle Packing (Square). But the more load-bearing issue for the paper's strongest claim—the 11/12 empirical superiority—is statistical: three runs per cell cannot support the count, and the table shows several margins smaller than one standard deviation. I therefore partially agree with the reader: their rationale already notes missing significance tests and overlapping error bars, but their weakest_assumption field points to relay gain rather than to the evidentiary basis of the headline. My proposed test (bootstrap/Wilcoxon on 10+ seeds per cell) would settle whether the 11/12 result is real. If the result survives, the remaining relay-gain concern is a secondary robustness issue; if it does not, the headline should be weakened. In either case the reader's CONDITIONAL verdict is appropriate.","tokens_in":12484,"tokens_out":7601,"duration_ms":77368,"concrete_test":"Obtain or regenerate per-run best scores for all 12 cells with at least 10 matched seeds per cell (same backend, same budget accounting). For each cell, compute a paired bootstrap 95% CI on the mean difference between RelayEvolve and the best non-RelayEvolve baseline, and a Wilcoxon signed-rank test. If fewer than 8 of 12 CIs exclude zero (or p<0.05), the 11/12 headline is not statistically supported and should be downgraded to 'competitive on some tasks'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RelayEvolve's central claim is empirical: it reports the highest mean score in 11 of 12 benchmark–budget settings. That claim is supported by only three runs per cell, with no significance tests or confidence intervals on the reported differences. Inspecting Table 1, many of the winning margins are within the noise of the method itself: TXN Scheduling at 50% and 75% shows RelayEvolve 3875.09 vs All-cheap 3873.89 (std 153.5); Circle Packing (Square) at 50% shows RelayEvolve 2.4136 vs All-strong 2.4084, with RelayEvolve std 0.1228. The one non-winning cell, Prism at 50%, is described as near saturation, which suggests the 11/12 count partly measures a plateau where several methods tie at the ceiling (e.g., Prism best score 26.2560 is reached by All-cheap, All-strong, and RelayEvolve). With n=3, a single favorable seed can flip the count. This is a correctness risk for the headline, not an internal inconsistency: the shared ShinkaEvolve backend and cost accounting make the comparison fair, but the statistical evidence is too thin to establish superiority. The absence of released code and undisclosed scheduler hyperparameters (h, k, λ, η, c, w, p, ε_rel, ε_F) further blocks reproducibility, although that is secondary to the underpowered comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies cost allocation between cheap and strong LLMs in evolutionary program search. It argues that existing call-level routing ignores the stateful nature of evolution, in which each candidate changes the population inherited by later mutations. The authors propose RelayEvolve, a training-free framework in which a cheap model explores multiple trajectories in short blocks, a Grow–Deepen bandit scheduler allocates blocks using a \"Relay Gain\" reward defined as the marginal improvement of a quality-diverse candidate bank, and handoff to a strong model occurs when this reward saturates or the cheap-model budget is exhausted. The same submodular relay objective is reused for offline seed curation, and the strong model then refines the selected population with the remaining budget. The paper motivates the design with three empirical findings about front-loaded progress, noisy early trajectories, and cheap-model early competitiveness, then evaluates RelayEvolve on four benchmarks at three budget levels, claiming the highest mean score in 11 of 12 benchmark–budget settings.","tokens_in":12794,"tokens_out":7551,"duration_ms":74719,"significance":"If the empirical claim were robust, this would be a useful contribution: it reframes budget allocation around evolving populations rather than individual model calls, and the unified submodular relay objective is a clean and potentially reusable mechanism. The controlled experimental setup, with all methods sharing the same ShinkaEvolve backend, the same evaluator, and the same cost accounting, is a strength, as is the explicit attempt to coordinate trajectory allocation, stopping, and seed selection under one objective. However, the headline result rests on only three runs per cell with no significance tests, several winning margins are smaller than the reported standard deviations, and the method's many hyperparameters and code are not disclosed. These issues currently prevent the central empirical claim from being accepted as established, although they are fixable within the manuscript's scope.","major_comments":[{"comment":"The headline claim of the highest mean score in 11 of 12 settings is supported by three independent runs per cell, with no significance tests or confidence intervals on the differences. Several reported margins are far smaller than the reported run-to-run standard deviations: in TXN Scheduling at 50% budget, RelayEvolve (3875.09 ± 153.52) is only 1.2 above All-cheap (3873.89 ± 129.43); in Circle Packing (Square) at 50% budget, RelayEvolve (2.4136 ± 0.1228) is only 0.0052 above All-strong (2.4084 ± 0.0237); and in Prism at 75% and 100% budgets, the mean margins over All-cheap are about 0.0044. In addition, the Prism best score 26.2560 is reached by multiple methods, so the 11/12 count partly reflects plateau ties. With n=3, a single seed can flip several of these outcomes. Please report per-seed results, paired or bootstrap significance tests, and confidence intervals on the differences, and increase the number of runs for at least the critical comparisons.","section":"Table 1, Main Results"},{"comment":"The method depends on at least ten hyperparameters—h, k, λ, η, c, w, p, ε_F, ε_rel, and the strong-model budget reserve—but no values or selection protocol are reported. The only value mentioned is the \"default split of 0.85\" in Figure 4(d). It is therefore unclear whether these parameters were fixed before the experiments, chosen on Circle Packing (Square) and then applied to other tasks, or tuned per cell; if any were selected after observing the full benchmark results, the comparison is optimistic. Please provide a full hyperparameter table and a clear tuning procedure, and release code or detailed pseudocode so that the reported numbers can be reproduced.","section":"Method: Adaptive Cheap Model Search and Population Handoff"},{"comment":"Relay Gain (Eq. 7) is the marginal improvement of the relay bank under the quality-diversity objective F, and it drives both the bandit scheduler (Eqs. 8–9) and the adaptive stopping rule (Eq. 10). The paper's design claim is that this set-level reward is a faithful proxy for the value of a block's candidates for downstream strong-model refinement. The only direct evidence is the relay-mechanism ablation on Circle Packing (Square) at the 100% budget (Figure 4a), which is itself based on three runs and one task. A concrete validation would be to compute, for each block, the correlation between its relay gain and the eventual contribution of that block's candidates to the final strong-model population. Without such evidence, the scheduler and stopping rule remain heuristic, and the central thesis about population handoff is only partially supported.","section":"Method: Relay Objective and Relay Gain"},{"comment":"The budget comparison is not fully documented. The paper states that B = β B_all-strong and caps runs at N = 200 generations, but it does not report, for any method or cell, whether the run terminated by exhausting its monetary budget or by reaching the generation cap, nor how much of the budget was actually spent. This matters because All-cheap is said to coincide across budget levels at less than 0.5B_all-strong; if RelayEvolve or other methods are generation-cap-limited at some budgets, the \"same budget\" comparison does not hold as straightforwardly as stated. Please report realized costs per method and the termination cause per cell.","section":"Experiments: Experimental Setup"}],"minor_comments":[{"comment":"\"where the optimization landscape is sufficiency challenging\" should read \"sufficiently challenging.\"","section":"Ablation Study"},{"comment":"The caption says \"Best results in each budget–benchmark column are bolded,\" but the bolded entries in the table are the mean scores, and the \"Best\" column values are not bolded. Please clarify the bolding convention.","section":"Table 1 caption"},{"comment":"The label \"Cheap/Strong Gain @ Gen20\" is not defined in the caption; please state explicitly that it is the ratio of cheap-model to strong-model best-so-far improvement over the first 20 generations.","section":"Figure 2(b)"},{"comment":"The deduplication step is not defined; please specify the equivalence criterion used to decide that two program candidates are duplicates.","section":"Eq. (6)"},{"comment":"The abstract and conclusion state that RelayEvolve \"achieves the highest mean score in 11 of 12 settings\"; since the means are based on three runs and no significance tests are reported, please qualify the claim as the highest observed mean, or add significance statements.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the 11/12 headline will be taken as established evidence by readers who do not inspect the per-cell standard deviations. I would ask the authors to add paired significance testing, release per-seed data and code, and disclose all hyperparameters and their selection procedure before this can be accepted. The conceptual contribution is promising, but the statistical and reproducibility evidence currently lags behind the strength of the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLM-driven program search. The core idea is real: instead of routing each model call, allocate budget to populations passed between a cheap and a strong model. The relay-gain reward (marginal improvement of a quality-diverse candidate bank) is a new construction, and using the same objective for online scheduling, stopping, and final seed selection is elegant. The empirical motivation section is genuinely useful: front-loaded progress, noisy early signals, cheap models capture roughly 0.8–1.0x of early strong-model gains. The experiments use a shared ShinkaEvolve backend with realistic token-cost accounting, and the ablations support the design choices.\n\nBut the headline claim of highest mean in 11 of 12 settings is not established by the data. Three runs per cell, no significance tests, and several winning margins sit inside the standard deviation: TXN at 50% budget (3875 vs 3874, std ±153) and Circle Packing Square at 50% (2.414 vs 2.408, std ±0.123) are effectively ties. The one non-win (Prism at 50%) is near saturation, so the 11/12 count partly reflects a plateau. With n=3, one seed flips the count. The reported budget-split sensitivity picks the default 0.85 on Circle Packing Square at 100% budget, then the paper uses that split everywhere; that is tuning on an evaluation task, though the magnitude seems modest. Hyperparameters are not fully disclosed, and no code is released, limiting reproducibility.\n\nNone of this is fatal to the idea. The conceptual framing is a step forward, and the method is coherent. But the evidence as presented supports \"competitive and often better,\" not \"11 of 12.\" A responsible revision would add significance testing (paired bootstrap across shared seeds, or more runs), release code and the full hyperparameter table, and report the budget-split choice as a selected hyperparameter with sensitivity across tasks.\n\nWho is this for? Anyone working on cost-efficient LLM evolution or algorithm discovery will find the population-handoff perspective worth engaging with. It deserves a serious referee, but the empirical section needs real work before the central claim can stand. I'd like to see this after revision, with stronger evidence.","headline":"A genuinely new population-handoff idea for cost-aware LLM evolution, but the 11-of-12 headline is underpowered with n=3 and no significance tests.","tokens_in":13420,"tokens_out":2406,"would_cite":true,"duration_ms":23849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By organizing LLM-evolution budgets around the population handed between a cheap and a strong model, RelayEvolve reports the highest mean score in 11 of 12 benchmark–budget settings.","keywords":["LLM-driven evolution","population handoff","relay gain","budget allocation","quality-diversity","bandit scheduling","program search","cost-efficient inference"],"falsifier":"Measure, over many independent runs with the same budget, the correlation between the Relay Gain accumulated during the cheap phase and the additional fitness the strong model gains in the refinement phase; a null or negative correlation would show that Relay Gain is not measuring handoff value. Equivalently, hold the handoff point fixed and replace relay-gain scheduling with random Grow–Deepen choices under the same cheap budget: if performance does not drop, the reward signal is not what carries the method.","tokens_in":12228,"feed_emoji":"🧬","tokens_out":6998,"duration_ms":64808,"temperature":0.7,"pith_summary":"RelayEvolve claims that in LLM-driven evolutionary search, the right unit of budget allocation is not the individual model call but the population of candidate programs passed from one model phase to the next. The paper supports this with empirical observations that search progress is front-loaded, early trajectory fitness is a noisy signal, and a cheap model recovers most of the early gains of a strong model at lower cost. On this basis it proposes a training-free scheme: a cheap model explores several trajectories in short blocks, a relay bank measures the marginal improvement of a compact quality-diverse candidate set, and this Relay Gain decides when to start or deepen trajectories and when to hand off to a strong model. The strong model then refines the curated seeds with the remaining budget, and the resulting method reports the highest mean score in 11 of 12 benchmark–budget settings.","feed_headline":"Population handoff wins 11 of 12 LLM-evolution settings","feed_subtitle":"What matters is the candidate set handed to the strong model, not which model answers the next query.","key_machinery":"The central object is the relay bank and its Relay Gain. The relay value of a bank $S$ relative to a candidate pool $C$ is $F_C(S)=\\lambda Q_r(S)+(1-\\lambda)D^q_C(S)$, where $Q_r$ is the average of the top-$r$ normalized qualities and $D^q_C$ is the quality-weighted diversity coverage of the pool by the bank; because both terms are monotone submodular, $F_C$ is too. Relay Gain is the marginal improvement $g_t=F_{C_{t+1}}(S_{t+1})-F_{C_{t+1}}(S_t)$ after a block of cheap-model generations. This single set-level reward drives the Grow–Deepen bandit scheduler, the adaptive stopping rule, and the final greedy curation of the seed population handed to the strong model.","core_discovery":"At the level of the paper's own claims, the discovery is that stateful LLM evolution is better served by optimizing the population handed from a cheap exploration phase to a strong refinement phase than by routing each mutation to a weaker or stronger model. The authors define the relay objective $F_C(S)=\\lambda Q_r(S)+(1-\\lambda)D^q_C(S)$ over a compact bank, use its marginal increase (Relay Gain) as the reward for a Grow–Deepen bandit scheduler, and trigger handoff when recent relative gains fall below a threshold. The final seed set is chosen by greedy submodular selection on the same objective, then used to initialize a shared strong-model population. The experimental claim is that this population-handoff organization achieves the highest mean score in 11 of 12 settings across four program evolution benchmarks and three budgets.","pith_inferences":["Relay Gain is used only as a scheduling reward in the paper; one natural extension would be to test it as a direct predictor of the strong model's downstream improvement, which would turn the scheduler into a forecasting device rather than a heuristic.","The same population-handoff objective could be applied with more than two model tiers or with heterogeneous block lengths, treating the handoff point as a continuous resource boundary rather than a single cheap-to-strong switch.","Because the handoff is stateful and population-based, RelayEvolve may be particularly compatible with multi-fidelity evaluators: relay gain measures improvement in the candidate bank independently of how each candidate was generated, so evaluation fidelity and model handoff could be scheduled together.","A testable extension is to hold the total budget and the final strong-model phase fixed while randomizing the cheap-model schedule; if relay-gain scheduling still wins over random scheduling, the claim that the reward signal carries the value is strengthened."],"forward_implications":["If RelayEvolve is right, then cost-aware evolutionary search should allocate budgets to phases and populations, and call-level routing methods that ignore population state will leave performance on the table.","Cheap models can front-load exploration cheaply, while strong model calls are best spent refining a compact, diverse set of promising candidates rather than scanning from scratch.","A single set-level reward (Relay Gain) can coordinate trajectory allocation, handoff timing, and seed selection, replacing separate heuristics for each decision.","Matching or outperforming both All-cheap and All-strong in most settings implies that the optimal model mix is task-dependent and that adaptive handoff is a practical way to find it under a fixed budget.","Because no training is required, the framework can be applied on top of existing LLM evolutionary backends and swapped model pairs."],"supporting_citations":[{"why":"Supplies the ShinkaEvolve evolutionary backend shared by all methods and the Bandit baseline that uses realized best-so-far improvement as reward.","marker":"(Lange, Imajuku, and Cetin 2026)"},{"why":"Provides the AdaEvolve adaptive allocation baseline and the evaluation setting that RelayEvolve follows.","marker":"(Cemri et al. 2026)"},{"why":"Basis of the quality-diversity principle behind the relay bank's top-quality plus coverage objective.","marker":"(Mouret and Clune 2015)"},{"why":"Supplies the quality-diversity framework used to justify rewarding banks that are both high-performing and diverse.","marker":"(Pugh, Soros, and Stanley 2016)"},{"why":"Gives the $(1-1/e)$ approximation guarantee relied on for greedy seed selection under the monotone submodular relay objective.","marker":"(Nemhauser, Wolsey, and Fisher 1978)"},{"why":"Motivates the recent-window UCB score used by the Grow–Deepen scheduler for non-stationary rewards.","marker":"(Besbes, Gur, and Zeevi 2014)"},{"why":"Supplies the SkyDiscover benchmark suite and Prism task used in the empirical evaluation.","marker":"(Liu et al. 2026c)"},{"why":"Defines the LEVI role-based routing baseline that RelayEvolve compares against.","marker":"(Tanveer 2026)"}],"fun_headline_variants":["Population handoff wins 11 of 12 LLM-evolution benchmarks","Don't route queries, hand off populations: 11 of 12 wins","Bandit-scheduled population handoff beats per-query routing 11/12","Stateful evolution: cheap models explore, strong refine, wins 11/12","For LLM evolution, spend on handoff not query: 11/12 wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Relay Gain, the marginal improvement of a quality-diverse relay bank, faithfully measures how much the strong model will gain from the handed-off population, so that it is a valid reward for scheduling cheap-model blocks and timing the handoff.","fun_headline_variants_meta":{"raw":{"variants":["Population handoff wins 11 of 12 LLM-evolution benchmarks","Don't route queries, hand off populations: 11 of 12 wins","Bandit-scheduled population handoff beats per-query routing 11/12","Stateful evolution: cheap models explore, strong refine, wins 11/12","For LLM evolution, spend on handoff not query: 11/12 wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3063,"prompt_tokens":971,"completion_tokens":2092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":1988}},"tokens_in":587,"tokens_out":2092,"duration_ms":12562,"temperature":1.0,"reasoning_tokens":1988,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:45:48.196389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, over many independent runs with the same budget, the correlation between the Relay Gain accumulated during the cheap phase and the additional fitness the strong model gains in the refinement phase; a null or negative correlation would show that Relay Gain is not measuring handoff value. Equivalently, hold the handoff point fixed and replace relay-gain scheduling with random Grow–Deepen choices under the same cheap budget: if performance does not drop, the reward signal is not what carries the method.","supporting_citations":[{"cited_title":"T.; Imajuku, Y.; and Cetin, E","cited_arxiv_id":null,"evidence_quote":"Supplies the ShinkaEvolve evolutionary backend shared by all methods and the Bandit baseline that uses realized best-so-far improvement as reward."},{"cited_title":"K.; Soros, L","cited_arxiv_id":null,"evidence_quote":"Supplies the quality-diversity framework used to justify rewarding banks that are both high-performing and diverse."},{"cited_title":"L.; Wolsey, L","cited_arxiv_id":null,"evidence_quote":"Gives the $(1-1/e)$ approximation guarantee relied on for greedy seed selection under the monotone submodular relay objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the recent-window UCB score used by the Grow–Deepen scheduler for non-stationary rewards."}],"review_version":1}