{"id":"8953633e-752c-407c-89ad-32a7ca280836","arxiv_id":"2507.02892","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-SAEA uses an LLM scoring expert and an LLM decision expert to dynamically select surrogate models and infill criteria, and it reports competitive results on expensive-optimization benchmarks, though the gain over random selection is not statistically significant.","lead":"The paper proposes LLM-SAEA, which uses two language-model experts to pick which surrogate model and sampling rule should run at each step of an expensive optimization. It is a plausible way to cut manual tuning, but the experiments do not show a significant advantage over random selection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own ablation (Table 3) shows LLM-SAEA is not statistically distinguishable from random, alternating, or Q-learning action selection, so the claimed effectiveness of LLM dynamic configuration is not supported by the data.","rationale":"The reader's conditional verdict is appropriate and my analysis does not move it. The central contribution, LLM-based dynamic configuration, is not validated against trivial dynamic-selection baselines: the paper's own Table 3 shows no statistically significant advantage over V-Random, V-Alter, or V-Q. The abstract and Section 4.3.1 overstate this result. The unvalidated LLM-SE reward signal is a plausible cause, but the decisive issue is the missing statistical support for the core mechanism. I credit the authors for publishing code, reporting per-problem win/loss counts, and acknowledging the non-significance against ESA and AutoSAEA; these choices make the discrepancy visible. The verdict remains conditional: the method is plausible and documented, but the central claim needs stronger evidence and the text needs to be aligned with the reported statistics.","tokens_in":29594,"tokens_out":5596,"duration_ms":56198,"concrete_test":"Using the released code, re-run the F1-F15 (10D) ablation with 50 independent runs matched by random seed across LLM-SAEA, V-Random, V-Alter, and V-Q, then apply paired Wilcoxon signed-rank tests per problem and a paired Friedman test across problems. If LLM-SAEA still fails to show significant superiority (p < 0.05) over at least one of V-Random or V-Alter, the claim that LLMs provide effective dynamic configuration should be withdrawn or replaced with 'competitive with trivial selection.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that LLM-driven online configuration improves SAEA performance (Abstract; Section 4.3.1). That claim requires LLM-SAEA to beat non-LLM dynamic-selection baselines. Table 3 provides the direct test: LLM-SAEA significantly outperforms V-Random on only 5 of 15 problems (1 loss, 9 ties), V-Alter on 4 of 15, and V-Q on 5 of 15, with Friedman p-values of 0.37, 0.82, and 0.82. None is significant. The surrounding text nonetheless states that the results confirm the effectiveness of LLMs in dynamic configuration, which contradicts the paper's own statistics. The likely mechanism is the unvalidated LLM-SE score: a 0-1 value produced by GPT-3.5-turbo from rank and objective magnitudes (Algorithm 3, Fig. 2), whose correlation with actual action quality is never measured, and whose stochasticity is uncontrolled (no temperature/seed reporting). If these scores are noisy, Eqs. 7-8 propagate noise and LLM-DE degenerates to softmax/roulette over near-random averages, exactly the pattern Table 3 shows. The Section 4.2 admission that LLM-SAEA is not significantly better than ESA or AutoSAEA reinforces that the headline superiority is limited to static-configuration baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-SAEA, a surrogate-assisted evolutionary algorithm that uses two LLM-based modules—a decision expert (LLM-DE) and a scoring expert (LLM-SE)—to configure online the surrogate model and infill criterion from a fixed action set of eight (model, criterion) combinations. LLM-DE outputs recommended actions with confidence labels; LLM-SE assigns a 0–1 score to the newly evaluated solution, and average action scores and selection frequencies are updated by Eqs. (7)–(8). The method is tested on five classic benchmarks plus the CEC2005 F1–F15 problems at dimensions 10 and 30, with 1000 function evaluations and 20 independent runs, and compared against eight existing SAEAs and twelve ablated variants. The paper reports that LLM-SAEA significantly outperforms the static-configuration baselines on a majority of problems, while explicitly acknowledging that it is not significantly better than ESA or AutoSAEA in the Friedman test. The ablation studies compare LLM-SAEA with fixed-action variants, a sequential variant, random, alternating, and Q-learning action selection.","tokens_in":29979,"tokens_out":5942,"duration_ms":61500,"significance":"If the central claim were fully established, LLM-SAEA would be a useful contribution: it would show that LLM-based online configuration can replace hand-designed dynamic configuration mechanisms in SAEAs, with public code and a component-wise ablation framework. The paper has genuine strengths: the experimental design includes Wilcoxon rank-sum tests, Friedman tests with Hommel correction, multiple benchmark suites, and an openly available implementation. The comparisons against static-configuration SAEAs are mostly convincing, and the self-reflection and collaboration ablations (Tables 4 and 5) show positive evidence for those components. However, the paper's own Table 3 shows that LLM-SAEA is not statistically distinguishable from random, alternating, or Q-learning action selection, which directly undermines the load-bearing claim that the LLM's dynamic configuration is effective. The unvalidated and apparently uncontrolled LLM scoring mechanism adds further uncertainty. The contribution is therefore promising but not yet supported at the level claimed in the abstract and Section 4.3.1.","major_comments":[{"comment":"The sentence \"These results confirm the effectiveness of LLMs in the dynamic configuration of SAEAs\" is not supported by the reported statistics. Against the dynamic-selection controls, LLM-SAEA wins on only 5 of 15 problems versus V-Random and V-Q and on 4 of 15 versus V-Alter, and the Friedman p-values are 0.37, 0.82, and 0.82, respectively. Since none of these comparisons reaches significance, the data are consistent with LLM-driven configuration being no better than random or simple online-selection rules. The claim should be softened, or the experiment strengthened, for example with more runs, more problem instances, or a direct measurement of selection quality.","section":"Section 4.3.1, Table 3"},{"comment":"The LLM-SE reward signal is not validated. The 0–1 score produced by GPT-3.5-turbo is the only feedback used to update average action scores, and Eq. (7) propagates this score into every subsequent decision through Eq. (6) and Algorithm 2. The paper never reports the correlation between the LLM score and an actual quality measure, such as normalized improvement in objective value, nor does it report the LLM temperature, random seed, or repeated-call variance. Without such information, one cannot distinguish a meaningful credit-assignment signal from stochastic noise; noisy scores would make softmax/roulette selection approach uniform selection, which is exactly the pattern observed in Table 3. A validation study, or replacement of the LLM score with a deterministic surrogate reward plus controlled LLM inference, is needed before the dynamic-configuration claim can be accepted.","section":"Section 3.2.2, Algorithm 3, Fig. 2, Eqs. (7)-(8)"},{"comment":"The final executed action is chosen by random.choice(a*), not directly by the LLM's ranking. Because the LLM-DE prompt explicitly requires that each action be explored, a* can contain many or even all eight actions, so the uniform draw over a* can dilute whatever preference the LLM produces. This design may explain why LLM-SAEA is statistically indistinguishable from V-Random in Table 3. The authors should report the distribution of |a*| over the optimization process and either select actions according to their scores or probabilities rather than uniformly, or demonstrate that the LLM-DE output substantively constrains the random draw.","section":"Algorithm 1, lines 13-14; Algorithm 2"}],"minor_comments":[{"comment":"The prompt heading \"Soring Expert\" is a typo for \"Scoring Expert.\"","section":"Fig. 2"},{"comment":"The legend of Fig. 4 labels Action 1 as (GP, EI) and Action 2 as (GP, LCB), but Section 3.1 defines the action set with (GP, LCB) first and (GP, EI) second; the legend should match the formal definition of A.","section":"Fig. 4 vs. Section 3.1"},{"comment":"The abstract says \"another LLM\" acts as the decision expert, but Section 4.1 states that GPT-3.5-turbo-0125 is used for both expert roles; the wording should say the same LLM with different prompts, or the experimental setup should use two different models.","section":"Abstract and Section 4.1"},{"comment":"The pseudocode contains rendering artifacts, such as \"while A* ≠ ∅\" appearing as \"while /u1D440... ≠ )uni2205(vardo\" in Algorithm 1 and \"Set A* ← ∅\" appearing as \"← )uni2205(var\" in Algorithm 2; these should be cleaned before publication.","section":"Algorithm 1 and Algorithm 2"},{"comment":"The complexity expression uses an iteration-count symbol that is not consistently named in the text, and it should account for the internal DE optimizer used by the local-search actions in Eq. (5), which consumes 100D+1000 surrogate-based evaluations per call rather than a single constant cost C_SAEA.","section":"Section 3.3, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an empirical method paper with no evidence of circular fitting to the benchmarks; the authors report their statistical results honestly, including the nonsignificant comparisons against ESA and AutoSAEA. The main problem is the gap between the dynamic-configuration claim and the nonsignificant results in Table 3 against random, alternating, and Q-learning selection. I recommend major revision rather than rejection because the static-configuration comparisons are mostly convincing and the LLM expert framework may be salvageable with additional validation, controlled LLM inference, and a revised set of claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's worth knowing: this is a clean empirical study of a plausible idea—using two LLMs, one to score actions and one to choose them, for online configuration of surrogate models and infill criteria. The paper is transparent about its statistics and releases code, which is more than many papers do. But the central claim—that the LLM configuration loop helps—is not supported by the paper's own ablation. In Table 3, LLM-SAEA is statistically indistinguishable from V-Random, V-Alter, and V-Q (Friedman p-values 0.37, 0.82, 0.82). The text nonetheless says the results confirm the effectiveness of LLMs in dynamic configuration. That's a direct mismatch.\n\nOn the positive side, the collaboration-of-experts separation is a new combination, and the self-reflection mechanism is a reasonable idea. The comparison against static-configuration baselines is extensive, and the paper honestly notes the lack of significant improvement over ESA and AutoSAEA.\n\nThe soft spots beyond the overclaim: LLM-SE's 0-1 score is never validated against actual action quality, and no temperature or seed is reported, so the stochasticity of the LLM is uncontrolled. Since Eqs. 7-8 propagate that score, noise there would explain why the method degenerates to random-like behavior. This is not a fatal flaw in the method per se, but it is a hole in the evidence.\n\nThis paper is for readers working on LLM-based algorithm configuration or SAEAs who want to see a well-executed example of why naive LLM scoring may not beat simple baselines. It deserves a serious referee because the question is timely and the empirical work is substantial, but the referee should require the authors to align claims with statistics and to report controls. I would accept it for peer review with that expectation.","headline":"The collaboration-of-experts idea is worth discussing, but the paper's own Table 3 does not show that LLM-driven selection beats random, alternating, or Q-learning selection.","tokens_in":30403,"tokens_out":2424,"would_cite":false,"duration_ms":25131,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two large language models, one scoring solutions and one choosing algorithm components, can configure a surrogate-assisted evolutionary algorithm online, and the resulting method outperforms several existing SAEAs on most benchmark…","keywords":["large language model","surrogate-assisted evolutionary algorithm","expensive optimization","algorithm configuration","online configuration","infill sampling criterion","collaboration-of-experts","benchmark optimization"],"falsifier":"Record the LLM-SE score and the actual improvement in best-so-far objective for every selected action in the released code on F1–F15 (10D), and compute their rank correlation; a near-zero correlation would indicate that the reward signal feeding the update equation is not what drives performance, and a rerun with the true improvement as the score would then settle the matter.","tokens_in":29374,"feed_emoji":"🤖","tokens_out":10718,"duration_ms":107117,"temperature":0.7,"pith_summary":"The paper claims that the hand-designed part of surrogate-assisted evolutionary optimization—choosing which surrogate model and which infill sampling criterion to use at each step—can be delegated to two collaborating large language models. It proposes LLM-SAEA, in which a scoring expert LLM rates each newly evaluated solution from 0 to 1, and a decision expert LLM uses those ratings plus selection frequencies to recommend actions, with a softmax-and-roulette fallback when it is unsure. On 10- and 30-dimensional benchmark problems, the method reports lower average function error than several existing SAEAs on most test cases, and ablation studies support the contribution of both LLMs. If the claim holds, expensive black-box optimization would no longer require a human expert to pre-select a model-criterion pair or to engineer a reward function for online selection.","feed_headline":"Two LLM experts beat fixed optimizer setups on most tests","feed_subtitle":"For costly black-box problems, the method automates choosing the surrogate model and infill rule at every step.","key_machinery":"The load-bearing mechanism is the collaboration-of-experts loop, in which two LLM roles are connected by an online action-value statistic. An 'action' is a named pair (surrogate model, infill sampling criterion) chosen from eight fixed combinations. The decision expert (LLM-DE) receives the current budget, iteration, and each action's average score $s_a(t)$ and selection frequency $f_a(t)$, then returns a recommended action set with self-generated confidence labels ('certain' or 'uncertain'); uncertain actions are replaced by a softmax over average scores followed by roulette-wheel selection. The scoring expert (LLM-SE) converts the rank and objective value of each newly evaluated solution into a score $r_a \\in [0,1]$, and the action's average is updated by $s_a(t+1) = \\frac{f_a(t)s_a(t) + r_a}{f_a(t) + 1}$ and $f_a(t+1) = \\frac{f_a(t)+1}{t+1}$. This loop makes the LLMs the online configurator, replacing hand-coded heuristic rules, reward engineering, or bandit update laws.","core_discovery":"On the paper's own terms, the central discovery is that LLM-SAEA—a surrogate-assisted evolutionary algorithm whose online configuration is driven by two LLMs—achieves lower function error than eight existing SAEAs on the majority of 20 benchmark problems at both dimensions, while keeping the best average Friedman ranking. The algorithm maintains a fixed set of eight actions, each pairing a surrogate model (GP, RBF, PRS, or KNN) with an infill criterion (LCB, EI, prescreening, local search, L1-exploitation, or L1-exploration). At each iteration, the decision-expert LLM reads each action's average score and selection frequency and proposes an action set; each proposed action is executed to generate one new expensive evaluation, and the scoring-expert LLM assigns the resulting solution a score in $[0,1]$, which is folded into the action's running average by the update equations. For actions the decision expert labels 'uncertain', selection falls back to softmax probabilities and roulette-wheel choice. The paper also reports that the advantage over the reinforcement-learning-based ESA and the multi-armed-bandit-based AutoSAEA is not statistically significant in the Friedman test, while the advantage over static-configuration baselines is.","pith_inferences":["The same two-expert loop could be applied to other online algorithm-selection problems with a finite portfolio and an observable per-step outcome, such as choosing acquisition functions in Bayesian optimization or mutation operators in evolutionary strategies.","Replacing the scoring expert with the true improvement in best-so-far objective would test whether the LLM's ratings or the decision prompt is the load-bearing component; the paper does not run that experiment.","The framework is not tied to the fixed eight actions; an LLM could propose new surrogate/infill pairs, converting the method from a configurator into an open-ended algorithm designer.","The experiments use a single hosted LLM, so sensitivity to model choice and response randomness is untested; temperature and model changes are the natural next variables to explore."],"forward_implications":["For expensive problems, the added LLM calls are a small per-iteration overhead compared with the cost of one function evaluation, so the automation comes at low practical cost.","The ablation comparisons against single-action, sequential, random, and alternating variants imply that the performance gain comes from the LLM configuration loop, not from any one surrogate–infill pairing.","Because the configuration is updated online, the method can shift between exploration and exploitation as the evaluation budget runs down, something static SAEAs cannot do.","The Friedman test does not show a statistically significant advantage over ESA and AutoSAEA, so the paper's superiority claim is clearest against the static-configuration baselines.","The release of source code makes the prompts, the action set, and the update equations directly repeatable."],"supporting_citations":[{"why":"It supplies the catalogue of surrogate models and infill criteria from which the eight actions are assembled.","marker":"[8]"},{"why":"It provides the function-error metric and the Wilcoxon/Friedman statistical testing conventions used throughout the comparisons.","marker":"[13]"},{"why":"It is the reinforcement-learning dynamic-configuration baseline and the source of the local-search action settings.","marker":"[15]"},{"why":"It is the multi-armed bandit auto-configuration baseline that constitutes the closest learned competitor.","marker":"[27]"},{"why":"It is a static-configuration baseline whose alternating selection rule is reused in the V-Alter ablation variant.","marker":"[37]"},{"why":"It defines the differential evolution operator that the surrogate/infill actions use to generate offspring.","marker":"[61]"},{"why":"It defines the F1–F15 benchmark suite on which the main comparisons and ablations are run.","marker":"[62]"}],"fun_headline_variants":["LLM duo assigns scores, picks optimizer configs each step","LLM-SAEA auto-selects surrogate and infill, beats fixed setups on most tests","Two LLM experts configure SAEA online, win most benchmark cases","LLM-driven SAEA auto-tunes surrogates and infill, topping most tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on the scoring LLM's 0-to-1 ratings being a stable and truthful measure of how much an action helped; if those ratings are noisy or biased, the decision LLM's choices degrade toward random selection.","fun_headline_variants_meta":{"raw":{"variants":["LLM duo assigns scores, picks optimizer configs each step","LLM-SAEA auto-selects surrogate and infill, beats fixed setups on most tests","Two LLM experts configure SAEA online, win most benchmark cases","LLM-driven SAEA auto-tunes surrogates and infill, topping most tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2897,"prompt_tokens":984,"completion_tokens":1913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1829}},"tokens_in":600,"tokens_out":1913,"duration_ms":15756,"temperature":1.0,"reasoning_tokens":1829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:13:58.518769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the LLM-SE score and the actual improvement in best-so-far objective for every selected action in the released code on F1–F15 (10D), and compute their rank correlation; a near-zero correlation would indicate that the reward signal feeding the update equation is not what drives performance, and a rerun with the true improvement as the score would then settle the matter.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the catalogue of surrogate models and infill criteria from which the eight actions are assembled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the function-error metric and the Wilcoxon/Friedman statistical testing conventions used throughout the comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the reinforcement-learning dynamic-configuration baseline and the source of the local-search action settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the multi-armed bandit auto-configuration baseline that constitutes the closest learned competitor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is a static-configuration baseline whose alternating selection rule is reused in the V-Alter ablation variant."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the differential evolution operator that the surrogate/infill actions use to generate offspring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the F1–F15 benchmark suite on which the main comparisons and ablations are run."}],"review_version":1}