{"id":"69758af8-da60-4b73-a21e-f2f9fa8fa675","arxiv_id":"2607.13911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Dual-Surrogate Guided Search selects operator-parent actions for LLM-based heuristic generation by predicting the child's latent distribution and its expected per-instance utility, improving sample efficiency on several automated-heuristic-design benchmarks.","lead":"An LLM-based search system that automatically writes optimization heuristics gets a new steering module: before each expensive code-generation query, two learned surrogates score which archived heuristics to reuse as parents and which operator to apply. On five optimization tasks, the dual-surrogate rule ties or beats four published baselines on four tasks — at the cost of extra machinery, and with margins small enough that uncharacterized run-to-run noise could matter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Controlled evidence for DGS's central gain is within run-to-run noise: Table 2's H4-vs-H3 margin is 0.043, based on three-run means with no error bars or significance tests.","rationale":"The reader's weakest_assumption emphasizes transition-surrogate generalizability, with statistical fragility listed second. I agree that the transition-surrogate assumption is a real risk, but it is not the most load-bearing: the paper's own ablation 'w/o transition prediction' shows a non-negligible drop on OBP, providing at least indirect evidence that the transition model contributes. The sharper problem is that the central empirical comparison could be entirely consistent with run-to-run noise: a 0.043 margin on three-run means, with no variance estimates, cannot support the headline claim of improved sample efficiency. This is a concern about evidence rather than mechanism, and it directly targets the 'more than run-to-run noise' condition identified in the reader's strongest_claim. A single high-powered rerun of the controlled study would settle it. Because the mechanism is plausible, the math is coherent, and the paper's claims are modest, the appropriate verdict remains conditional rather than accept or reject. I marked agreement as partial because the reader did identify the statistical fragility but gave primacy to the transition-surrogate assumption.","tokens_in":13151,"tokens_out":6563,"duration_ms":68341,"concrete_test":"Rerun the controlled TSP comparison (H3 vs H4, and full DGS vs H4) with at least 20 independent seeds under identical budgets, recording per-seed best-so-far at generation 200 and the full trajectory. Compute a paired bootstrap 95% CI for the mean difference and a rank-based test (e.g. Mann-Whitney U or Wilcoxon) on final best scores. Also report the same for the Table 1 per-task ranks using repeated runs. If the CI for H4−H3 includes zero, the claimed advantage of joint surrogate action selection is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical claim is that learned joint operator-parent action selection improves sample efficiency over the best rule-based control. In the controlled study (Table 2), the margin between H4 (joint surrogate) and H3 (operator bandit + rank) is -6.406 vs -6.449, i.e. 0.043 on a three-run mean. The paper reports no error bars, confidence intervals, or significance tests anywhere, and run-to-run spread appears comparable to this margin: the same configuration appears in Table 3 as 'w/o β_u' at -6.464, and several ablation differences on TSP are 0.1-0.5 while OBP differences are around 0.3, suggesting large variance. With only three runs, a 0.043 difference is not separable from noise. The main comparison (Table 1) also depends on average ranks over per-task results placed in an unavailable supplementary file, so the reader cannot check whether DGS's best-on-three-tasks claim is consistent across settings. If this margin is noise, the paper's central contribution reduces to a reasonable but unsupported mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Dual-Surrogate Guided Search (DGS), a module for LLM-based automated heuristic design that selects, before each LLM query, an operator-parent action (o,S) from the current archive. DGS learns a shared latent representation of heuristic code (Eq. 3), trains an instance-conditioned utility surrogate ensemble (Eqs. 4-12) that predicts per-instance performance of a child latent, and trains a transition surrogate (Eqs. 14-17) that predicts a diagonal Gaussian over the child latent induced by an action. These are combined in the acquisition rule of Eq. (19) with exploration bonuses from utility-ensemble disagreement and transition uncertainty. The method is evaluated on TSP, KP, OBP, ASP, and CVRP-ACO against FunSearch, EoH, ReEvo, and MCTS-AHD under a 200-generation budget, with a controlled TSP comparison and ablations. The central claims are that DGS improves sample efficiency and that the joint operator-parent surrogate outperforms rule-based action selection; the main results are reported as average ranks and three-run means.","tokens_in":13385,"tokens_out":9334,"duration_ms":92348,"significance":"If the reported gains hold, the paper makes a useful and original contribution: it treats pre-generation action selection as a learned decision problem rather than a hand-coded rule, and the two-surrogate decomposition is a natural way to score actions before the child code exists. The objective functions are well-formed, Algorithm 1 is explicit, and the paper honestly frames the main result as 'competitive' rather than claiming uniform superiority. The utility rank loss (Eq. 8), the Gaussian NLL (Eq. 17), and the acquisition rule (Eq. 19) are internally coherent. However, the empirical evidence for the central claim is currently not strong enough: the key controlled margin is within run-to-run noise, and no error bars or significance tests are reported. The significance of the work therefore depends on the authors providing statistically grounded comparisons.","major_comments":[{"comment":"The load-bearing claim that the joint operator-parent surrogate outperforms the strongest rule-based control rests on H4 (-6.406) vs H3 (-6.449), a 0.043 margin on three-run means. The paper reports no error bars, confidence intervals, or significance tests for any table. With three runs, this margin is not distinguishable from run-to-run noise. Note also that Table 3 reports DGS Original at -6.351 on the same task while H4 is -6.406; the difference between two DGS variants is itself of the same order as the H4-H3 gap. This is the central controlled comparison for RQ2, so it must be supported by additional seeds and a statistical test (or, at minimum, error bars and per-run values).","section":"Table 2 and §Effect of Joint Operator-Parent Selection"},{"comment":"The main result summary uses average ranks, but the actual per-task scores are not in the body; they are stated to be in the supplementary material, which was not part of the manuscript as provided. The reader therefore cannot verify the claimed 'best on TSP, KP, and OBP' or the average ranks reported in Table 1. A central empirical claim should be reproducible from the submitted manuscript. Include a per-task table with means and standard deviations in the main text, or otherwise supply enough per-task evidence to check the average ranks.","section":"Table 1 and §Main Results"},{"comment":"The acquisition rule in Eq. (19) treats τ(x) as the distribution of the child latent for action x and uses sτ(x) = ||σ_τ(x)|| as an exploration bonus. The manuscript provides no diagnostic that this diagonal-Gaussian transition model is a reasonable approximation of actual LLM-generated child latents, nor that its predicted uncertainty is calibrated. The ablation 'w/o λτ' in Table 3 shows sensitivity to this term, but sensitivity does not establish validity. Please report held-out transition NLL or calibration/coverage checks on held-out transitions, or otherwise justify the Gaussian/parent-summary parameterization.","section":"Eqs. (14)-(17), §Transition Surrogate"},{"comment":"Table 3 is used to conclude that each component is necessary, but all values are three-run means without variance or significance tests. On OBP, many variants are within about 0.03 of each other (e.g., -2.166 vs -2.174); on TSP, the range is about 0.55. Without error bars, the ordering of configurations may not be stable. Report per-run values, standard deviations, or paired tests over the same seeds or initial archives.","section":"Table 3, §Ablation Study"}],"minor_comments":[{"comment":"The text says 'if a batch contains no valid within-instance pair, the rank term is set to zero.' Make this explicit in the equation or immediately after, since the current notation R+ could be empty.","section":"Eq. (7)"},{"comment":"The five EoH operators are only named; a one-sentence description in the main text would make the method self-contained without forcing the reader to consult the supplement.","section":"§Method Overview"},{"comment":"The caption says 'Mean best' without specifying training vs test scores; the text clarifies later, but the caption should state 'training score' explicitly.","section":"Table 2 caption"},{"comment":"With only three runs, min-max bands are not a robust uncertainty display. Consider plotting means with standard errors or per-run curves.","section":"Figure 2"},{"comment":"The paper does not state how β_u, λ_τ, K, and M_MC were chosen. Add a short paragraph on hyperparameter selection and sensitivity.","section":"§Experimental Setup"},{"comment":"No code-release or seed-management statement is included. For an empirical paper of this kind, please state whether code will be released and how seeds are handled.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The method is coherent and clearly written, but the empirical case for the central claim is currently thin. The 0.043 margin in Table 2 and the absence of variance information mean that the main result may not survive a stricter reanalysis. I would be willing to accept a revision that adds more seeds, significance tests, per-task tables, and validation of the transition surrogate. I do not see grounds for rejecting the approach itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new formulation—dual-surrogate action selection for pre-generation operator-parent choices in LLM-based AHD—and the equations hold together. The empirical case does not yet support the headline claim.\n\nWhat's new: the transition surrogate (Eq. 16) predicts a diagonal Gaussian over the child's latent from the parent latents and an operator embedding, which is a sensible way to score actions before the LLM is called. Combined with the instance-conditioned utility ensemble and the UCB-style acquisition (Eq. 19), it forms a coherent mechanism that I don't see in the cited EoH, FunSearch, ReEvo, or MCTS-AHD work. The paper is also honest: the abstract says \"competitive,\" and the main table shows DGS ranks second on CVRP-ACO and ties ReEvo on ASP.\n\nWhere it falls short is evidence. Every result is a three-run mean with no error bars, confidence intervals, or significance tests. The controlled comparison that isolates the method's contribution (Table 2) shows a 0.043 margin between the joint surrogate (H4) and the best rule-based control (H3). The paper's own ablation table shows TSP differences of 0.1–0.5 and OBP differences around 0.3, so run-to-run variance appears to be in that range; 0.043 on three runs is not separable from noise. I should note that the stress-test's claim that H4 and the w/o beta_u row in Table 3 are the same configuration is incorrect—that ablation removes the exploration term—but the broader statistical problem stands. Hyperparameters are unstated, per-task results are in an unavailable supplement, and no code is referenced. Surrogate training on the search's own archive is a legitimate self-training loop rather than a circularity in the main comparison, but it does mean the reported gains could be specific to this exact setup.\n\nWho should read it: anyone working on LLM-based AHD or surrogate-assisted algorithm design. The formulation is worth thinking about even if this draft doesn't prove it works.\n\nRecommendation: send it to peer review—the idea is important enough to spend referee time—but the authors need to supply error bars, more runs, the supplement, and ideally code. As written, I would not cite the empirical claims, though I might cite the formulation once it is published. My verdict: conditional, closer to reject than accept.","headline":"A genuinely new dual-surrogate formulation for pre-generation action selection in LLM-based AHD, mathematically coherent and honestly framed, but the empirical case is statistically under-powered and the central controlled margin is within noise.","tokens_in":13988,"tokens_out":2943,"would_cite":false,"duration_ms":30494,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that scoring pre-generation operator-parent actions with two learned surrogates—one predicting the child code's latent distribution, one predicting its utility—lets LLM-based automated heuristic design spend its limited qu","keywords":["automated heuristic design","large language models","surrogate-guided search","operator-parent action selection","uncertainty-aware acquisition","latent representation learning","sample efficiency","combinatorial optimization"],"falsifier":"Take a fixed archive and a fixed pool of candidate actions; train the two surrogates on a random half of the transition records; test whether the acquisition score ranks the held-out actions by actually observed child score better than chance (e.g., Spearman rank correlation). If correlation is at or below zero, the surrogate is not learning a valid action-to-utility map. Separately, rerun the controlled TSP comparison with at least 20 independent seeds: if the distribution of best scores overlaps heavily, the reported 0.043 mean gap is noise.","tokens_in":12913,"feed_emoji":"⚙️","tokens_out":7622,"duration_ms":69147,"temperature":0.7,"pith_summary":"The paper addresses the pre-generation decision in LLM-based automated heuristic design (AHD): given an archive of evaluated heuristics, which one(s) should be reused as parents, and which generation operator should transform them, before spending the next LLM query and black-box evaluation. The authors claim that this choice is best made by learning from accumulated code-score and transition records rather than by rules such as archive ranking, random sampling, fixed operator schedules, or tree policies. They propose Dual-Surrogate Guided Search (DGS), which trains a transition surrogate to predict the distribution of the child code's latent representation and an instance-conditioned utility surrogate to predict per-instance performance of sampled child latents, then combines predicted utility and both uncertainties in an acquisition rule to select the next action. Using a 200-generation budget across five heuristic-design tasks, DGS reports the best average rank on three tasks, a tie on one, and second on the fifth, with controlled ablations showing that jointly learning operator and parent beats rule-based operator selection and operator-only or parent-only surrogates. The core claim is that the bottleneck in LLM-AHD is not prompt engineering alone but allocating the next generation call to the most promising operator-parent pair.","feed_headline":"Two surrogates pick the next LLM heuristic step","feed_subtitle":"Learned action selection converts past evaluations into more productive LLM generation calls.","key_machinery":"The load-bearing machinery is the two-surrogate decomposition of an unobservable outcome: the transition surrogate (Eq. 16) turns an operator-parent action into a distribution τ(x) over the child's learned latent representation z, and the utility surrogate (Eq. 5) turns any latent into per-instance performance estimates via ensemble heads fed with interaction features ξ(z, q_j) = [z, q_j, z⊙q_j, |z−q_j|, cos(z,q_j)]. The acquisition rule (Eq. 19) sums expected utility, ensemble disagreement, and transition uncertainty to rank actions. The paper also introduces a shared task-adaptive latent representation z(a) = norm(W e(a) + ρ R_φ(e(a))) over frozen code embeddings, plus instance embeddings","core_discovery":"The central claim is that a pre-generation operator-parent action can be given a score before its child code exists, and that this score is a valid guide for allocating the next LLM query. The paper's mechanism: a transition surrogate maps an action x = (o, S) — a generation operator plus a set of archived parents — to a diagonal Gaussian over child latents, using a summary of parent latents and a learned operator embedding. An instance-conditioned utility surrogate, an ensemble of heads, predicts per-instance performance from a heuristic-instance interaction feature, and is trained with both pointwise MSE and within-instance rank supervision. The acquisition rule α(x) = E[μ_u(z)] + β_u E[σ_","pith_inferences":["The transition surrogate is trained on self-generated data, raising the risk of a feedback loop where the acquisition rule favors actions the surrogate already predicts well, suppressing genuinely novel exploration; a testable extension is to inject a fraction of random or bandit-chosen actions into the training data and measure whether surrogate validity and final performance improve.","The controlled margin between the joint surrogate and the strongest non-surrogate control is 0.043 on a three-run mean, so statistical significance is not established; a reader should ask for repeated seeds and confidence intervals before treating the advantage as robust.","The instance-conditioned utility surrogate points toward a broader use: a latent embedding trained to predict per-instance utility could rank pregenerated heuristics on unseen instances, turning the search module into an algorithm-selection tool beyond AHD.","The diagonal-Gaussian form of the transition surrogate is a strong simplification; if some operators produce highly multimodal child distributions, the acquisition rule's expectation over τ(x) could systematically underestimate uncertainty, suggesting mixture or normalizing-flow extensions."],"forward_implications":["DGS is operator-agnostic: the paper shows it remains effective when the operator set is expanded with a tree-path synthesis operator, so learned action selection can be layered onto any archive-based LLM-AHD loop.","Because the utility surrogate conditions on individual instances, the search can favor heuristics that generalize across the training set rather than merely maximizing aggregate score.","The utility-trained latent space makes offline heuristic selection more effective: a GP-UCB selector built on these latents reduces simple regret faster than one built on raw code embeddings.","The controlled comparison shows that jointly scoring operator and parent is necessary: operator-only or parent-only surrogates are weaker than the joint surrogate, and all surrogate variants except operator-only beat the rule-based controls."],"fun_headline_variants":["Dual surrogates guide LLM heuristic generation","Score actions before generating LLM heuristics","Surrogate-guided action selection for heuristic design","Two surrogates pick the next LLM heuristic step","Pre-generation scoring improves LLM heuristic search"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that a diagonal Gaussian over a learned latent, whose parameters come from parent-latent summaries and an operator embedding, can predict the distribution of the child code an LLM will actually generate, and that this model, trained online on data the same search produced, generalizes well enough that the acquisition score ranks actions in the same order as their true expected utility.","fun_headline_variants_meta":{"raw":{"variants":["Dual surrogates guide LLM heuristic generation","Score actions before generating LLM heuristics","Surrogate-guided action selection for heuristic design","Two surrogates pick the next LLM heuristic step","Pre-generation scoring improves LLM heuristic search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1380,"prompt_tokens":777,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":521,"tokens_out":603,"duration_ms":5496,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:21:42.915028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed archive and a fixed pool of candidate actions; train the two surrogates on a random half of the transition records; test whether the acquisition score ranks the held-out actions by actually observed child score better than chance (e.g., Spearman rank correlation). If correlation is at or below zero, the surrogate is not learning a valid action-to-utility map. Separately, rerun the controlled TSP comparison with at least 20 independent seeds: if the distribution of best scores overlaps heavily, the reported 0.043 mean gap is noise.","supporting_citations":[],"review_version":1}