{"id":"476d2f9d-ee47-4a38-81ce-315d6f4ff4d1","arxiv_id":"2412.03250","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Dynamic mutation prompts sampled from a power-law distribution improve the convergence speed of LLaMEA when driven by GPT-4o, but not GPT-3.5-turbo.","lead":"This paper tests whether large language models can follow mutation-rate instructions when rewriting optimization algorithm code, and then adds a power-law-sampled dynamic mutation rate to an LLM-driven evolutionary framework. It reports that GPT-4o can follow these instructions and that dynamic rates speed up convergence of the LLaMEA framework, while GPT-3.5-turbo cannot.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig. 5 compares dynamic-rate runs on GPT-4o to an external [18] baseline that may have used a different model and prompt, so the claimed benefit of dynamic rates is confounded.","rationale":"The reader's weakest assumption is precisely the load-bearing point: Fig. 5 is the only evidence for the headline claim, and its baseline is external. Section 4 states the new experiments use GPT-4o-2024-08-06 and GPT-3.5-turbo-0125, but [18] may use a different GPT version; the paper itself demonstrates in Fig. 4 that model capability strongly affects mutation adherence, so a model-driven improvement is plausible. The prompt also differs: the dynamic condition replaces the generic mutation instruction with a rate-specified concrete request, and Fig. 4 shows prompt engineering materially changes mutation outcomes. Therefore, the comparison conflates model, prompt, and dynamic rate. The proposed concrete test would isolate these factors by running the default prompt on the exact GPT-4o model and by adding a fixed-rate control. If the default-prompt GPT-4o baseline reproduces [18]'s curve, the external baseline is validated; if the fixed-rate control matches the dynamic result, the 'dynamic' aspect is not the active ingredient. This does not overturn the paper—its adherence experiments are valuable—but it prevents over-attribution. The verdict remains CONDITIONAL until such controls are provided.","tokens_in":8723,"tokens_out":5691,"duration_ms":53205,"concrete_test":"Re-run LLaMEA with the original default mutation prompt ('Either refine or redesign to improve the algorithm') on GPT-4o-2024-08-06, using the exact evaluation pipeline of the new experiments (BBOB dimension 5, 3 instances, AOCC metric, 100 code instances, 5 runs). Compare this new GPT-4o baseline curve against both the [18] raw data and the dynamic-rate curve from Fig. 5b. Additionally, run a fixed-rate condition (e.g., requested 10% each mutation) with the same explicit prompt style used for dynamic rates; if the fixed-rate curve matches the dynamic-rate curve, the improvement is not specific to the power-law dynamics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that dynamic mutation rates improve LLaMEA convergence with GPT-4o—rests on Fig. 5, which compares new dynamic-prompt runs against 'raw data directly from [18]' (caption). This external baseline is unverified on three axes: (1) model version: the new runs use GPT-4o-2024-08-06, while [18] likely used a different GPT version, and the paper itself shows model capability strongly affects mutation adherence (Fig. 4); (2) prompt: the baseline default mutation prompt is 'Either refine or redesign to improve the algorithm' (Section 3), whereas the dynamic condition injects a numeric requested rate into the prompt, so any explicit-rate prompt might improve results regardless of the distribution; (3) run conditions: number of runs, BBOB instances, evaluation details, and random seeds are not confirmed identical to the new experiments. Because the baseline was not re-run under the new pipeline, the observed improvement in Fig. 5 cannot be causally attributed to the power-law dynamic rate; it could be a model upgrade or a prompt-engineering effect. The paper does not provide a statistical test for the GPT-4o comparison, and it only mentions 'not significantly' for GPT-3.5-turbo, leaving the difference in Fig. 5b unexplained.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper augments the LLaMEA framework with dynamic mutation prompts: instead of the default mutation instruction, a requested mutation rate is sampled from a power-law distribution and inserted into the prompt. The authors test five manually constructed and six automatically generated prompts on GPT-3.5-turbo and GPT-4o, measuring how close the delivered code change is to the requested rate. They then run LLaMEA with the best prompt per model plus dynamic rates and compare convergence against raw baseline data from [18]. They report that GPT-4o follows the rate instructions much better than GPT-3.5-turbo, and that with GPT-4o the dynamic-rate version converges faster than the baseline.","tokens_in":8955,"tokens_out":6165,"duration_ms":55453,"significance":"The prompt-adherence study is a useful, falsifiable empirical contribution: it shows a clear qualitative difference between two GPT models and that longer, more explicit prompts improve rate adherence for GPT-4o. The second claim—that power-law dynamic rates improve convergence of LLaMEA—is potentially important for automated algorithm design, but in its current form it rests on an unverified external baseline and on very few runs without statistical tests. The theoretical grounding in fast mutation operators is appropriate, and no fitted parameters are used beyond beta=1.5, which is a strength. Reproducibility would be stronger with a released code/data artifact, which the manuscript does not mention.","major_comments":[{"comment":"The convergence improvement attributed to dynamic mutation rates is obtained by comparing new runs against 'raw data directly from [18]' (Fig. 5 caption). Because the baseline was not re-run under the exact pipeline of this paper, the comparison is confounded: the model version (GPT-4o-2024-08-06 vs. whatever [18] used), the mutation prompt (default vs. explicit numeric rate), the BBOB instances, the number of runs, and random seeds are not shown to be identical. Given that Section 5.1 demonstrates that model capability changes mutation behavior substantially, this confound must be removed by re-running the default LLaMEA baseline under identical conditions or by providing a direct equivalence argument.","section":"§5.2, Fig. 5"},{"comment":"The claim that dynamic rates 'improve the convergence speed' with GPT-4o is based on five runs per condition with no significance test, confidence interval, or effect size. The paper mentions 'not significantly' only for GPT-3.5-turbo; the GPT-4o difference is reported without any uncertainty quantification. Please add an appropriate statistical comparison (e.g., Mann-Whitney U or bootstrap over independent runs) and report the number of runs and seeds.","section":"§5.2, Fig. 5"},{"comment":"The central metric 'percentage of code modification' is never defined. It is used in Eq. (2) to compute MSE, in Eq. (3) for prompt selection, and in Figs. 4 and 6 to claim that GPT-4o adheres to requested rates. The authors should specify exactly how code difference is measured (e.g., line-based diff, token-level Levenshtein, AST edit distance) and how percentages are computed, since different definitions can change the prompt ranking.","section":"§3, Eq. (2)"},{"comment":"The TDW-score in Eq. (3) weights the five requested rates by P r(x_i), but Eq. (1) defines P r over α/n, where n is the number of lines of the parent code. The paper never states how a requested percentage x is mapped to α/n (e.g., α = floor(n*x/100)). Without this mapping, the TDW-score and the resulting choice of 'best prompt' are not reproducible. Please make the mapping explicit and justify using percentages 2, 5, 10, 20, and 40 as a discretization.","section":"§3, Eq. (1) and Eq. (3)"}],"minor_comments":[{"comment":"Section 2 says LLaMEA uses an evolutionary strategy '(1, 1) or (1 + 1)', but Section 4.1 describes only the (1 + 1) strategy; please clarify which strategy was actually used in the experiments.","section":"§2"},{"comment":"In Eq. (2), the term log(x_i / x_i) is identically zero, so the first equality contains a redundant term; the notation should be simplified and checked.","section":"§3, Eq. (2)"},{"comment":"The caption of Fig. 4 does not explain the visual encoding of the columns (e.g., which prompt and mutation rate each column corresponds to) or how the green dotted-dashed line is computed; the reader should be able to interpret the panels without relying on the body text.","section":"Fig. 4"},{"comment":"The bullet list in Section 4.1 contains formatting issues, such as 'Code generation budget100 code instances' and 'the experiment is repeated 3 times for each combination'; please use consistent spacing and specify that these are independent runs.","section":"§4.1, experimental setup"},{"comment":"The phrase 'advanced automated cue engineering' in Section 6 should be 'prompt engineering', and in Section 2 'easily expend on' should be 'easily extend'.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not the idea but the evidence: the convergence claim rests on comparison with an external baseline that is not re-run. If the authors can make the baseline comparison watertight, the paper would be a reasonable contribution to the LLM-based algorithm design literature. I would not recommend rejection, because the prompt-adherence result is informative and the fix is local."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the prompt-adherence study is real work: the authors test manual and LLM-generated prompts across two GPT models, five requested mutation rates, and repeated runs, and they show convincingly that GPT-3.5-turbo ignores explicit rate instructions while GPT-4o gets reasonably close, especially with longer, more specific prompts. That part is useful and well-supported. Second, the headline claim that dynamic mutation rates improve LLaMEA convergence is much weaker than the abstract suggests. The key figure compares new GPT-4o runs against baseline data taken directly from the earlier LLaMEA paper. That baseline was not re-run under the same conditions: model version may differ, the prompt differs, and run counts or evaluation details are not verified. Since the paper itself shows model version strongly affects adherence, this comparison is confounded. You cannot cleanly attribute the observed improvement to the power-law distribution. Any explicit-rate prompt might have helped relative to a prompt with no rate at all.\n\nThe paper also has smaller soft spots. The code-difference metric is never precisely defined (line-based? token-based?), no code or data are released, and the GPT-4o convergence comparison lacks significance testing — the paper only says \"not significantly\" for GPT-3.5-turbo. Five runs for the dynamic-rate experiment is thin.\n\nThat said, the authors are honest about the weaker model's failure and about the automatic prompts not helping. The dynamic-rate idea is a simple, theory-grounded extension, not overclaimed. The prompt-adherence findings stand on their own and are directly actionable for anyone working on LLM-driven algorithm design.\n\nWho gets value? People building LLaMEA-style frameworks or using LLM mutation in evolutionary loops. The adherence results are worth citing; the convergence improvement should be treated as a hypothesis until the baseline is re-run under identical conditions.\n\nMy recommendation: send it to peer review, but require the authors to match the baseline, define the code-difference metric, and release code and data. The core empirical finding about prompt adherence is solid; the convergence story needs redoing.","headline":"Prompt adherence results are solid, but the convergence claim rests on an unverified external baseline.","tokens_in":9490,"tokens_out":2028,"would_cite":true,"duration_ms":20393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that prompting an LLM with a heavy-tailed sampled mutation rate can steer the amount of code it changes, and demonstrates faster LLaMEA convergence with GPT-4o while GPT-3.5-turbo ignores the instruction.","keywords":["mutation control","large language models","evolutionary computation","metaheuristic algorithm design","prompt engineering","heavy-tailed power-law distribution","LLaMEA","BBOB benchmark"],"falsifier":"Re-run the unmodified default LLaMEA (the bare 'refine or redesign' prompt) with GPT-4o-2024-08-06 under the exact experimental configuration used for the dynamic-rate runs and compare the mean best-so-far curves. If the default matches or beats the dynamic-rate curve, the claimed improvement fails. A second check is to measure GPT-4o's delivered code difference at a requested 2% rate: if it is not close to 2% across many trials, the premise that the model obeys the mutation rate is wrong.","tokens_in":8508,"feed_emoji":"🧬","tokens_out":9015,"duration_ms":76919,"temperature":0.7,"pith_summary":"This paper is trying to establish that the mutation step in an LLM-driven evolutionary algorithm can be controlled simply by telling the model how much code to change. It proposes drawing each requested mutation rate from a heavy-tailed power-law distribution and inserting that rate into the prompt, and it reports that GPT-4o adjusts the amount of code it rewrites accordingly. On the same benchmark suite used by the LLaMEA framework, this dynamic-rate mutation scheme converges faster than the recorded fixed-rate baseline. The paper also reports that GPT-3.5-turbo largely ignores the requested rate, which shows that prompt-based mutation control only works with a sufficiently capable model. A sympathetic reader would care because uncontrolled mutations waste evaluations and API cost, and a controllable mutation budget makes automated algorithm design more economically viable.","feed_headline":"GPT-4o follows mutation-rate prompts; evolution converges faster","feed_subtitle":"Heavy-tailed mutation requests beat the fixed default; GPT-3.5-turbo can't follow them.","key_machinery":"The load-bearing mechanism is a dynamic mutation prompt: at every mutation step, a code-difference target is sampled and made part of the instruction to the LLM. The target follows the power-law $Pr(\\alpha/n) \\propto \\alpha^{-\\beta}$ used in the fast mutation operator, with $n$ set to the number of lines of the parent code, so most mutations request small changes while rare mutations permit large redesigns. The evaluated prompt designs form a ladder from a plain rate request (Prompt 1) through mandatory, numeric, and line-count-specific formulations (Prompts 2-5), plus model-specific automatically generated prompts (Prompts 6-11). Adherence is scored by the mean squared error between log requested rate and log delivered code difference, aggregated as a target-distribution-weighted score, and the best manual prompt is then tested inside the full LLaMEA loop.","core_discovery":"The central claim is that prompt phrasing can act as a control channel for the mutation operator of an LLM-based evolutionary algorithm. The authors modify LLaMEA's default mutation instruction, which only says to refine or redesign, by adding a requested code-difference rate and then test eleven prompts at rates of 2%, 5%, 10%, 20%, and 40%. They find that GPT-4o's delivered code difference moves toward the requested rate as the prompt becomes more explicit, while GPT-3.5-turbo keeps making large, scattered changes regardless of the request. When the requested rate is sampled at each iteration from the heavy-tailed distribution of Equation 1 with $\\beta=1.5$, the best manual prompt improves the mean best-so-far convergence of LLaMEA with GPT-4o compared with the default setting. The paper concludes that dynamic mutation prompting yields better convergence speed and adaptability for GPT-4o, while the automatically generated prompts tested here do not yet beat the baseline.","pith_inferences":["The paper fixes the power-law exponent at $\\beta=1.5$; a natural extension is to tune $\\beta$ per problem class or to adapt it during the run, since the heavy-tailed schedule's benefit likely depends on how multimodal the landscape is.","The control mechanism should transfer to other LLM-based code optimization tasks, because it only requires a measurable edit distance; using a semantic diff instead of a line-based diff would be a stricter test of whether the model changes the intended functionality.","The negative result with GPT-3.5-turbo suggests a threshold effect: below some model capability, prompt-based mutation control is ineffective, so practitioners may need to enforce the mutation budget outside the model, for example by rejecting children whose measured code difference exceeds the requested rate."],"forward_implications":["A per-iteration mutation budget becomes a tunable parameter of the evolutionary loop, so LLaMEA can alternate between small refinements and occasional large redesigns instead of leaving that choice entirely to the LLM.","Prompt-based mutation control only pays off with a model capable of following quantitative instructions; the paper's experiments show GPT-3.5-turbo cannot.","The best manual prompt outperforms the simple automatically generated prompts, so automatic prompt engineering for mutation control is not yet a drop-in replacement.","Drawing mutation rates from a heavy-tailed power-law balances exploration and exploitation, which is the mechanism the paper credits for faster convergence on the benchmark suite."],"supporting_citations":[{"why":"Defines the LLaMEA framework and supplies the default mutation prompt and the baseline convergence data that the dynamic-rate experiments are compared against.","marker":"[18]"},{"why":"Introduces the heavy-tailed power-law mutation operator whose rate distribution this paper adapts into dynamic mutation prompts.","marker":"[7]"},{"why":"Defines the BBOB noiseless benchmark functions used to evaluate the generated metaheuristics.","marker":"[8]"},{"why":"Provides the IOHexperimenter platform used to run the benchmark evaluation and record best-so-far curves.","marker":"[15]"},{"why":"Defines the Area Over the Convergence Curve (AOCC) metric used as the performance measure in selection.","marker":"[20]"}],"fun_headline_variants":["GPT-4o adapts to mutation-rate prompts; GPT-3.5 fails","Dynamic mutation prompts speed up LLM-driven evolution with GPT-4o","Heavy-tailed mutation requests boost LLM evolution convergence","Prompt-controlled mutation helps GPT-4o evolve algorithms faster","Prompting mutation rates lets GPT-4o evolve better algorithms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed speed-up over default LLaMEA depends on the earlier baseline runs having used exactly the same experimental conditions, such as the same model versions, the same 100-generated-code budget, and the same benchmark evaluation; if those baseline runs differed, the comparison could be misleading.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o adapts to mutation-rate prompts; GPT-3.5 fails","Dynamic mutation prompts speed up LLM-driven evolution with GPT-4o","Heavy-tailed mutation requests boost LLM evolution convergence","Prompt-controlled mutation helps GPT-4o evolve algorithms faster","Prompting mutation rates lets GPT-4o evolve better algorithms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3124,"prompt_tokens":953,"completion_tokens":2171,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2081}},"tokens_in":569,"tokens_out":2171,"duration_ms":13821,"temperature":1.0,"reasoning_tokens":2081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:36:14.796991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the unmodified default LLaMEA (the bare 'refine or redesign' prompt) with GPT-4o-2024-08-06 under the exact experimental configuration used for the dynamic-rate runs and compare the mean best-so-far curves. If the default matches or beats the dynamic-rate curve, the claimed improvement fails. A second check is to measure GPT-4o's delivered code difference at a requested 2% rate: if it is not close to 2% across many trials, the premise that the model obeys the mutation rate is wrong.","supporting_citations":[{"cited_title":"In: Proceedings of the genetic and evolutionary computation conference","cited_arxiv_id":null,"evidence_quote":"Introduces the heavy-tailed power-law mutation operator whose rate distribution this paper adapts into dynamic mutation prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the BBOB noiseless benchmark functions used to evaluate the generated metaheuristics."}],"review_version":1}