{"id":"c9eafc3a-a322-4ded-8bd9-c2b0b7d4e1f4","arxiv_id":"2608.08156","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hybrid nested search, an LLM outer loop for structural sketches and a numerical inner loop for parameters, improves LLM-driven optimization and its measured gain tracks the parameter-tuning gap.","lead":"This paper proposes a hybrid optimization loop in which a language model writes the structure of a program while a classical numerical optimizer fills in the numeric constants. The combination beats either approach alone across test functions, cloud-system policies, and Bayesian inference reparameterizations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed universal superiority is not established: pure-numerical baselines receive tiny fixed budgets while hybrid gets K×100 inner evals with K unreported, and Table 1 itself shows hybrid losing to pure numerical on ackley and rosenbrock.","rationale":"The paper introduces a clean, well-motivated factorization: an LLM proposes structure with numeric holes, an inner solver tunes them, and the structure is scored on the tuned value. The de-aliasing proposition is a nice formalization, and the code release is a plus. However, the headline empirical claim—superior to both vanilla LLM search and pure numerical optimization across all three domains—is not supported by the evidence as presented. First, the pure-numerical baselines are not compute-matched. Section 2.4's cost model only compares hybrid to vanilla, and asserts c_ev/c_llm without reporting actual timings. The numerical baselines receive fixed budgets (2000 evals for meta, 120 for policies), while hybrid's inner loop uses B_in=100 per structure for K outer steps, with K never reported. Since the paper elsewhere says 'N=3 proposals', K could be as small as 3, giving hybrid at least 300 inner evals—more than the 120-eval policy baseline and nearly the 2000-eval meta baseline if K≈20. Without an equal-budget comparison, 'superior to pure numerical' could simply reflect more evaluations and an LLM prior. Second, the paper's own Table 1 contains direct counterexamples to the abstract's universal claim: on ackley with GLM-5.2, hybrid (3.3e-9) is worse than pure numerical (4.4e-12); on rosenbrock, hybrid (1.5e-12) is worse than numerical (4.2e-13); ellipsoid is a tie. The abstract's 'Across all three... superior to both' is therefore false as written. This is not a minor wording issue: the central claim is exactly the universal superiority, and the paper's contribution is framed around it. A concrete check—matching total fitness-evaluation budgets across arms and reporting K, with standard errors—would settle whether the advantage is due to decoupling or to compute. If the advantage persists, the paper is strong; if not, the claims must be narrowed to specific regimes (e.g., multimodal problems with high Δ). We therefore recommend keeping the conditional verdict, but with the additional explicit condition of compute-matched comparisons.","tokens_in":21942,"tokens_out":9476,"duration_ms":87585,"concrete_test":"Re-run all three domains with matched total fitness-evaluation budgets: fix K and run pure-numerical with exactly K*B_in evals (and, for vanilla, match wall-clock by adding idle or random restarts); report K, total evals, and wall-clock per arm. Also recompute Table 1 with standard errors across the N=3 seeds; if the ackley/rosenbrock rows flip sign or the pure-numerical advantage at equal budget persists, the universal-superiority claim is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.4 compares hybrid to vanilla only and asserts c_ev/c_llm ~ 1e-2–1e-3, but no wall-clock or total-evaluation budgets are reported. The pure-numerical baselines receive fixed small budgets: 2000 evals for direct CMA-ES (Appendix B.1) and 120 evals for policy CMA-ES (Appendix B.2). Hybrid's inner loop uses B_in=100 per structure for K outer steps, and K is never specified (Algorithm 1 leaves it as a hyperparameter; experimental sections only mention N=3 proposals). If K>=3, hybrid already uses >=300 inner evals, exceeding the 120-eval policy baseline; if K>=20, it exceeds the meta-optimizer budget. Thus 'superior to pure numerical optimization' may reflect larger evaluation budgets plus an LLM structural prior, not the decoupling itself. Compounding this, the paper's own Table 1 contradicts the abstract: on ackley with GLM-5.2 hybrid (3.3e-9) is worse than pure numerical (4.4e-12); on rosenbrock hybrid (1.5e-12) is worse than numerical (4.2e-13); ellipsoid ties. The abstract's 'Across all three, the hybrid optimizer is superior to both...' is therefore unsupported by the presented data. A matched-compute comparison is required before the central claim can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid nested search framework for LLM-driven optimization: an outer LLM loop proposes a structural sketch with numeric holes and a manifest of bounds/types/guesses, while an inner numerical solver (CMA-ES, VI, or NUTS) tunes the holes before the structure is scored and promoted. The central formal claim is Proposition 1, which states that vanilla joint search can discard the structural optimum when the parameter-tuning gap Δ(τ)=F(τ)-f(τ,θ̄) varies across structures, whereas hybrid search ranks by the tuned value and avoids this error up to an ε-accuracy margin. The paper also states a falsifiable prediction, Eq. (4), that hybrid advantage grows with the mean tuning gap E[Δ], and validates the framework on meta-optimizers, code-based policies, and approximate Bayesian inference tasks, with code released. The empirical sections report hybrid wins in many settings, but the abstract's claim of superiority over both vanilla and pure numerical baselines in all three domains is stronger than the data in Tables 1-2 actually support.","tokens_in":22262,"tokens_out":5076,"duration_ms":51369,"significance":"If the modest version of the claim holds, the paper makes a useful contribution: it gives a clean decomposition of LLM search into structural and parametric levels, connects the failure mode of joint search to a formal 'parametric aliasing' statement, and demonstrates that plugging in different inner solvers (CMA-ES, VI, NUTS) is feasible across quite different problem families. The Appendix contains substantial experimental breadth, including reparameterization findings and several worked artifacts, and the code release is a concrete asset. The formal Proposition 1 is a genuine, non-circular attempt to characterize when hybrid search helps. However, the strong universal-superiority claim is not established by the presented evidence: the paper's own tables contain ties and counterexamples, the comparison against pure numerical baselines is not on matched compute budgets, and all numerical cells are single point estimates without error bars or statistical tests. These issues are fixable within the scope of the manuscript, so the appropriate decision is a major revision rather than rejection.","major_comments":[{"comment":"The abstract states that 'Across all three, the hybrid optimizer is superior to both vanilla LLM-driven search and pure numerical optimization baselines,' but this is contradicted by the paper's own consolidated results. In Table 2, block A, the ellipsoid row is marked 'tie' for all three proposer models, and for schwefel with Opus 4.8 the hybrid delivered loss (405) is not better than the numerical baseline (358) and ties vanilla (405). In block C, the intra-cloud Cloudcast rows show only small gains over vanilla, and the GEPA block E contains explicit ties. The claim of universal superiority is therefore unsupported; the strongest defensible claim is that hybrid helps in many high-gap regimes and ties when Δ is small. Please revise the abstract and the corresponding conclusions to reflect the actual pattern of wins, ties, and occasional losses.","section":"Abstract; Table 2, block A and block C"},{"comment":"The comparison against pure numerical optimization is not matched in computation budget, and the key quantity K (the outer budget in Algorithm 1) is never reported. The text says the inner budget is B_in=100 (Section 3) and that N=3 proposals per arm are used, but it does not state whether K equals N or is something else. If K=3, then a hybrid arm uses 3×100=300 inner fitness evaluations, which already exceeds the 120-evaluation budget of the pure-CMA-ES policy baseline (Appendix B.2); if K≥20, it exceeds the 2000-evaluation direct-CMA-ES reference for meta-optimizers (Appendix B.1). Section 2.4 compares hybrid cost only to vanilla (C_hyb vs C_van) and asserts c_ev/c_llm ~ 1e-2-1e-3 without reporting wall-clock times or total evaluations. Because the central claim includes superiority over pure numerical solvers, a matched-compute comparison (e.g., equal total inner evaluations, or an explicit wall-clock budget) is necessary before that claim can be accepted.","section":"Section 2.4, Algorithm 1, Appendix B.1, Appendix B.2"},{"comment":"Every numerical cell in Tables 1 and 2 is reported as a single point estimate, with no error bars, no repeated-run variance, and no statistical tests, despite the text stating that Cleanup runs 'three seeds' and that the headline numbers are best-of-N selections with N=3. The Cleanup claim of '9/9 model×seed runs' is not accompanied by a table of per-seed values or a test of whether the margins are significant. Since the empirical case rests on comparing point estimates across multiple tasks and models, the lack of uncertainty quantification makes the win/tie/loss classification fragile, especially for cells that are close (e.g., the schwefel and ellipsoid rows). Please add repeated runs and appropriate confidence intervals or statistical comparisons for at least the headline cells, or explicitly frame the results as preliminary point estimates.","section":"Section 3, Table 2, Appendix B.3"},{"comment":"The de-aliasing guarantee in Proposition 1 depends on the inner solver being ε-accurate and on the manifest bounds (l_j,u_j) containing the true parametric optimizer, but the paper never measures ε and never tests sensitivity to bad manifest bounds. If the LLM-supplied bounds exclude the optimum, then the tuned value bF(τ) understates F(τ), and the 'hybrid never commits this error' statement in Proposition 1(2) no longer holds. Relatedly, Eq. (4) is called a falsifiable prediction, but it is partly definitional: the hybrid advantage max bF(τ) - max g_van(τ) contains the term Δ(τ) by construction when tuning is accurate, so the empirical correlation with E[Δ] is not fully independent evidence. Please either measure ε on the test problems, add an experiment with adversarially tight/wrong manifest bounds, or weaken the formal claims so they state the condition under which the guarantee applies.","section":"Section 2.1, Section 2.3, Proposition 1, Eq. (4)"}],"minor_comments":[{"comment":"The text contains a typo ('tipycally' for 'typically') and an uncited claim that hybrid search 'achieves lower loss than the oracle CMA-ES with much less function evaluations'; the second part should be substantiated with the reported evaluation counts or removed.","section":"Section 3.1"},{"comment":"The statement that the inner cost is 'negligible compared to the LLM' is an assertion, not a measurement; reporting wall-clock time or token costs for at least one task would make the cost argument concrete.","section":"Section 2.4"},{"comment":"The table uses 'tie†' without explaining the dagger in the table caption or in the surrounding text; please add a footnote or define the marker.","section":"Table 2"},{"comment":"The text says 'the two search strategies tie on ellipsoid' and then later says hybrid 'is the best performer again' on schwefel, but Table 2 shows that on schwefel with Opus 4.8 the hybrid matches vanilla and is worse than the numerical baseline; please align the narrative with the tabulated numbers.","section":"Section 3.1 vs Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-scale contribution with a useful idea and a large but somewhat uneven experimental appendix. The main revision needed is to match compute budgets, report uncertainty, and soften the abstract's universal-superiority claim. The formal part is sound in its modest reading, but the ε-assumption should be made explicit as an unverified condition. I would not reject because the core factorization is plausible and the empirical breadth is valuable; I would require the above fixes before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the hybrid nested loop—LLM proposes a structural sketch with numeric holes, inner solver tunes them—is the right way to think about LLM-driven search, and Proposition 1 is a correct, clarifying decomposition. The manifest interface is a small but practical contribution: it makes the LLM declare which numbers are tunable and their ranges. The experiments are broad, and the Bayesian inference section, with the SBC gate and the banana case where tuning one curvature constant buys +60 nats, is the most convincing part. Code is released.\n\nWhere it falls down: the abstract says hybrid is superior to both vanilla and pure numerical across all three domains, but the paper's own Table 1 contradicts that—hybrid loses to pure numerical on ackley and rosenbrock, and there are ties elsewhere. That is an accuracy problem. The compute comparison is also not matched. Pure numerical baselines get 2000 evals (meta-optimizers) or 120 evals (policies), while hybrid runs K outer steps with B_in=100 inner evals each, and K is never reported. If K is at least 20, hybrid gets more evaluations than the baseline, so \"superior to pure numerical\" may just mean \"given a larger budget.\" The cost argument in Sec 2.4 is about LLM tokens, not wall-clock or total evals, so it does not repair this. There are also no error bars—all cells are best-of-N=3 point estimates—so close calls are not statistically supported.\n\nThe manifest-bound assumption is the real theoretical soft spot: if the LLM's bounds exclude the true optimum, the inner solver cannot find it and the de-aliasing guarantee fails. The paper never measures epsilon or tests sensitivity to bad bounds. Eq. (4) is less a falsifiable prediction than a restatement of Proposition 1(3), though the across-task correlation is a reasonable sanity check.\n\nNet: this is a solid workshop-level contribution with a real idea, an honest limitations section, and a formal core that holds up. The empirical claims need to be reined in and redone with matched budgets, variance estimates, and a sensitivity analysis on manifest bounds. I would send it to review and ask for major revision, not desk-reject it.","headline":"The core idea is right and the formal result is clean, but the empirical claims overreach and the baselines are not compute-matched.","tokens_in":22774,"tokens_out":3017,"would_cite":false,"duration_ms":30514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting LLM search into structural sketches plus a nested numerical tuner beats vanilla LLM search and pure numeric optimization across three domains; the gain tracks how badly the model guesses constants.","keywords":["hybrid nested search","LLM-driven optimization","parametric aliasing","bilevel optimization","evolutionary algorithms","CMA-ES","Bayesian reparameterization","simulation-based calibration"],"falsifier":"On closed-form objectives where the true structural value $F(\\tau)$ is computable, measure the inner solver's actual error $|\\hat{F}(\\tau) - F(\\tau)|$ under the stated budget, then run the harness on families with deliberately large measured tuning gaps $\\mathbb{E}[\\Delta]$ in two variants, one with manifest bounds shrunk around the LLM's guess and one with bounds widened, and check whether the hybrid advantage tracks $\\mathbb{E}[\\Delta]$ as Eq. (4) predicts; discovering a domain with a large gap and no advantage, or an advantage where the gap is negligible, would settle that the de-aliasing account is not the operative mechanism.","tokens_in":21730,"feed_emoji":"🧩","tokens_out":16992,"duration_ms":136973,"temperature":0.7,"pith_summary":"The paper argues that when a language model searches for a good program by trial and error, it is doing two incompatible jobs at once: inventing the program's structure (control flow, algorithm design) and fixing the numerical constants embedded in it. The proposed fix is hybrid nested search: the LLM proposes a structural sketch with numeric holes and a declared manifest of bounds, and a standard numerical optimizer (CMA-ES, variational inference, or NUTS) fills the holes before the structure is scored. The paper formalizes why this split matters, showing that joint LLM search can discard a genuinely good structure whose default constants are miscalibrated, an error the inner loop provably avoids. It validates the split across meta-optimization, cloud-systems and social-dilemma policies, and approximate Bayesian inference, where hybrid beats both vanilla LLM search and pure numerical optimization and the advantage grows as the LLM's parameter guesses get worse. The design rule a sympathetic reader takes away: give structure to the model, give numbers to a solver.","feed_headline":"Nested tuner rescues good algorithms the LLM would discard","feed_subtitle":"Tuning the numbers inside LLM-proposed code beats vanilla LLM search and numeric-only baselines across three domains.","key_machinery":"The load-bearing object is the sketch decomposition: every candidate program is split into a structure $\\tau$ (the program topology, written as text) and a parameter vector $\\theta$ (the numeric holes), with the LLM required to emit a manifest per hole declaring bounds, type, and a suggested value $\\bar{\\theta}$ that warm-starts the inner solver. Around this sit two further components. First, the bilevel objective $F(\\tau) = \\max_{\\theta \\in \\Theta(\\tau)} f(\\tau,\\theta)$, the structural value: a structure's worth is its fitness at its parametric best, and the hybrid loop always evaluates and promotes on the tuned estimate $\\hat{F}(\\tau) \\approx F(\\tau)$. Second, Proposition 1, the parametric de-aliasing result: ranking structures by the untuned score $f(\\tau,\\bar{\\theta}(\\tau))$ is equivalent to ranking by $F(\\tau) - \\Delta(\\tau)$, so the tuning gap $\\Delta(\\tau) = F(\\tau) - f(\\tau,\\bar{\\theta}(\\tau)) \\geq 0$ can invert the true structural ranking whenever a good structure carries unguessable constants; scoring on the tuned value removes the inversion up to a margin of $2\\varepsilon$. The tuning gap $\\Delta$ is the quantity that carries the whole empirical argument: the paper measures it per task and shows the hybrid advantage tracks it.","core_discovery":"The central claim is that decomposing the LLM's optimization operator by variable type, structure to the language model and continuous parameters to a nested numerical solver, removes what the paper calls parametric aliasing, and that this removal is both provable and measurable. The hybrid loop scores each proposed structure $\\tau$ by the tuned value $\\hat{F}(\\tau) \\approx \\max_\\theta f(\\tau,\\theta)$ rather than by the untuned guess $f(\\tau,\\bar{\\theta}(\\tau))$. Proposition 1 shows that vanilla joint search can reject the structural optimum in favor of an inferior structure whenever the tuning-gap difference $\\Delta(\\tau^*) - \\Delta(\\tau')$ exceeds the true value difference $F(\\tau^*) - F(\\tau')$, whereas hybrid never commits this error beyond a margin of $2\\varepsilon$ given an $\\varepsilon$-accurate inner solver. The falsifiable prediction is that the hybrid advantage over vanilla grows with the mean tuning gap $\\mathbb{E}[\\Delta(\\tau)]$, the fitness a structure leaves on the table under the LLM's guessed parameters, and vanishes when the model's guesses are already near-optimal. Across the three experimental domains the prediction holds: hybrid wins or ties in nearly every reported cell, with ties concentrated in low-gap control regimes, and the largest wins appear exactly where the untuned guess is worst, most cleanly in the Cleanup dilemma (a policy that ranks worst untuned and best tuned) and in the curved Bayesian target (a +60-nat gain from tuning a single hidden curvature constant).","pith_inferences":["The de-aliasing failure is not specific to LLMs: any search process that guesses structure and constants jointly, including human designers, faces the same inversion, so the manifest-plus-inner-tuner pattern is a transferable design for human-in-the-loop algorithm and policy design.","A direct quantification of the $\\varepsilon$-accuracy assumption would sharpen the empirical story, since the paper never measures how often the inner solver reaches the true optimum within its stated budget; an experiment that compares $\\hat{F}(\\tau)$ against the analytically known $F(\\tau)$ on the closed-form objectives would show how much of the $2\\varepsilon$ margin is actually consumed.","Because the LLM controls the manifest bounds, the protocol adds a failure surface the paper does not probe: a model that under-declares a hole's range silently breaks the inner solver, suggesting automatic bound refinement or manifest validation against the incumbent as a natural extension.","The paper's finding that weaker proposers carry larger tuning gaps suggests a scaling law worth testing: hybrid search's marginal value may be largest for small, cheap models with poor guess calibration, making the harness an efficiency tool as much as a capability upgrade."],"forward_implications":["LLM-driven program discovery systems should stop asking the model to set constants; any text-based outer optimizer (a simple (1+1) loop or reflective prompt evolution) can be paired with any inner numerical solver and inherits the de-aliasing guarantee, making the factorization a design pattern rather than a specific algorithm.","The hybrid advantage over vanilla search is predictable from a measurable quantity, the mean tuning gap $\\mathbb{E}[\\Delta]$, and vanishes on problems where the LLM's parameter guesses are already near-optimal.","Where the inner solver already covers a class of parameters (NUTS warmup's linear mass matrix, for instance), the corresponding reparameterizations add nothing and the methods tie, while nonlinear transformations no mass matrix can fix are exactly where hybrid wins, so the payoff is tied to the inner solver's blind spots.","Inner solvers compose: variational inference can find a nonlinear curvature that NUTS's linear metric cannot, after which NUTS samples the straightened space with a roughly 37x effective-sample-size gain, pointing toward pipelines that chain two tuners.","On expensive-evaluation domains the cost balance inverts and the same factorization holds with a cheaper inner solver; the paper's accounting (overhead factor $1 + (c_{\\mathrm{ev}}/c_{\\mathrm{llm}})B_{\\mathrm{in}}$) shows the pattern pays most when LLM proposals cost far more than fitness evaluations."],"supporting_citations":[{"why":"Defines the vanilla LLM-in-the-loop program discovery paradigm that the paper's hybrid harness is measured against.","marker":"(Romera-Paredes et al., 2024)"},{"why":"Provides the premise that language models are poor continuous optimizers, the motivation for handing parameters to a numerical inner loop.","marker":"(Yang et al., 2023)"},{"why":"Supplies CMA-ES, the zero-order inner solver that tunes the LLM's parametric holes in the meta-optimizer and policy experiments.","marker":"(Hansen & Ostermeier, 2001)"},{"why":"Supplies the gradient-based variational inference inner solver for the differentiable Bayesian reparameterization tasks.","marker":"(Kucukelbir et al., 2017)"},{"why":"Supplies NUTS, whose warmup mass-matrix adaptation serves as the inner tuner for the MCMC instantiation.","marker":"(Hoffman & Gelman, 2014)"},{"why":"Supplies GEPA, the reflective prompt-evolution outer optimizer used to show the inner tuner is orthogonal to the outer search.","marker":"(Agrawal et al., 2025)"},{"why":"Supplies simulation-based calibration, the reference-free gate that certifies the discovered reparameterizations as correct.","marker":"(Talts et al., 2018)"},{"why":"Frames the No Free Lunch bound that the paper uses to delimit when the factorization can and cannot pay off.","marker":"(Wolpert & Macready, 1997)"},{"why":"Supplies the Cleanup sequential social dilemma benchmark, the setting of the paper's cleanest de-aliasing demonstration.","marker":"(Hughes et al., 2018)"}],"fun_headline_variants":["Nested hybrid optimizer beats pure LLM or numeric search","LLM proposes structure, numeric optimizer tunes: hybrid wins","Tune the numbers inside LLM code: hybrid beats both","Decouple structure from parameters: nested search wins","LLM maps structure, numeric solver sets parameters: hybrid superior"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole guarantee rests on the inner optimizer actually reaching the true best parameters within its fixed budget and the bounds the LLM declares; if those bounds or that budget exclude the real optimum, the tuned score still understates a structure's potential and the de-aliasing advantage can fail.","fun_headline_variants_meta":{"raw":{"variants":["Nested hybrid optimizer beats pure LLM or numeric search","LLM proposes structure, numeric optimizer tunes: hybrid wins","Tune the numbers inside LLM code: hybrid beats both","Decouple structure from parameters: nested search wins","LLM maps structure, numeric solver sets parameters: hybrid superior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":3073,"prompt_tokens":1061,"completion_tokens":2012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":1931}},"tokens_in":677,"tokens_out":2012,"duration_ms":14311,"temperature":1.0,"reasoning_tokens":1931,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:19:39.753997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On closed-form objectives where the true structural value $F(\\tau)$ is computable, measure the inner solver's actual error $|\\hat{F}(\\tau) - F(\\tau)|$ under the stated budget, then run the harness on families with deliberately large measured tuning gaps $\\mathbb{E}[\\Delta]$ in two variants, one with manifest bounds shrunk around the LLM's guess and one with bounds widened, and check whether the hybrid advantage tracks $\\mathbb{E}[\\Delta]$ as Eq. (4) predicts; discovering a domain with a large gap and no advantage, or an advantage where the gap is negligible, would settle that the de-aliasing account is not the operative mechanism.","supporting_citations":[],"review_version":1}