Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Controlling the Mutation in Large Language Models for the Efficient Evolution of Algorithms

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that prompting an LLM with a heavy-tailed sampled mutation rate can steer the amount of code it changes, and demonstrates faster LLaMEA convergence with GPT-4o while GPT-3.5-turbo ignores the instruction.

desk verdict Prompt adherence results are solid, but the convergence claim rests on an unverified external baseline. read the letter →

arxiv 2412.03250 v1 pith:X3GIL7II submitted 2024-12-04 cs.NE

Haoran Yin , Anna V. Kononova , Thomas Bäck , Niki van Stein This is my paper · ORCID
classification cs.NE
keywords mutationcontrollargelanguagemodelsevolutionarycomputationmetaheuristicalgorithmdesignpromptengineeringheavy-tailedpower-lawdistributionLLaMEABBOBbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the mutation step in an LLM-driven evolutionary algorithm can be controlled simply by telling the model how much code to change. It proposes drawing each requested mutation rate from a heavy-tailed power-law distribution and inserting that rate into the prompt, and it reports that GPT-4o adjusts the amount of code it rewrites accordingly. On the same benchmark suite used by the LLaMEA framework, this dynamic-rate mutation scheme converges faster than the recorded fixed-rate baseline. The paper also reports that GPT-3.5-turbo largely ignores the requested rate, which shows that prompt-based mutation control only works with a sufficiently capable model. A sympathetic reader would care because uncontrolled mutations waste evaluations and API cost, and a controllable mutation budget makes automated algorithm design more economically viable.

What carries the argument

The load-bearing mechanism is a dynamic mutation prompt: at every mutation step, a code-difference target is sampled and made part of the instruction to the LLM. The target follows the power-law $Pr(\alpha/n) \propto \alpha^{-\beta}$ used in the fast mutation operator, with $n$ set to the number of lines of the parent code, so most mutations request small changes while rare mutations permit large redesigns. The evaluated prompt designs form a ladder from a plain rate request (Prompt 1) through mandatory, numeric, and line-count-specific formulations (Prompts 2-5), plus model-specific automatically generated prompts (Prompts 6-11). Adherence is scored by the mean squared error between log requested rate and log delivered code difference, aggregated as a target-distribution-weighted score, and the best manual prompt is then tested inside the full LLaMEA loop.

What would settle it

Re-run the unmodified default LLaMEA (the bare 'refine or redesign' prompt) with GPT-4o-2024-08-06 under the exact experimental configuration used for the dynamic-rate runs and compare the mean best-so-far curves. If the default matches or beats the dynamic-rate curve, the claimed improvement fails. A second check is to measure GPT-4o's delivered code difference at a requested 2% rate: if it is not close to 2% across many trials, the premise that the model obeys the mutation rate is wrong.

Watch

Extended reading notes

Core claim

The central claim is that prompt phrasing can act as a control channel for the mutation operator of an LLM-based evolutionary algorithm. The authors modify LLaMEA's default mutation instruction, which only says to refine or redesign, by adding a requested code-difference rate and then test eleven prompts at rates of 2%, 5%, 10%, 20%, and 40%. They find that GPT-4o's delivered code difference moves toward the requested rate as the prompt becomes more explicit, while GPT-3.5-turbo keeps making large, scattered changes regardless of the request. When the requested rate is sampled at each iteration from the heavy-tailed distribution of Equation 1 with $\beta=1.5$, the best manual prompt improves the mean best-so-far convergence of LLaMEA with GPT-4o compared with the default setting. The paper concludes that dynamic mutation prompting yields better convergence speed and adaptability for GPT-4o, while the automatically generated prompts tested here do not yet beat the baseline.

Load-bearing premise

The claimed speed-up over default LLaMEA depends on the earlier baseline runs having used exactly the same experimental conditions, such as the same model versions, the same 100-generated-code budget, and the same benchmark evaluation; if those baseline runs differed, the comparison could be misleading.

Editorial extensions

If this is right

  • A per-iteration mutation budget becomes a tunable parameter of the evolutionary loop, so LLaMEA can alternate between small refinements and occasional large redesigns instead of leaving that choice entirely to the LLM.
  • Prompt-based mutation control only pays off with a model capable of following quantitative instructions; the paper's experiments show GPT-3.5-turbo cannot.
  • The best manual prompt outperforms the simple automatically generated prompts, so automatic prompt engineering for mutation control is not yet a drop-in replacement.
  • Drawing mutation rates from a heavy-tailed power-law balances exploration and exploitation, which is the mechanism the paper credits for faster convergence on the benchmark suite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes the power-law exponent at $\beta=1.5$; a natural extension is to tune $\beta$ per problem class or to adapt it during the run, since the heavy-tailed schedule's benefit likely depends on how multimodal the landscape is.
  • The control mechanism should transfer to other LLM-based code optimization tasks, because it only requires a measurable edit distance; using a semantic diff instead of a line-based diff would be a stricter test of whether the model changes the intended functionality.
  • The negative result with GPT-3.5-turbo suggests a threshold effect: below some model capability, prompt-based mutation control is ineffective, so practitioners may need to enforce the mutation budget outside the model, for example by rejecting children whose measured code difference exceeds the requested rate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper augments the LLaMEA framework with dynamic mutation prompts: instead of the default mutation instruction, a requested mutation rate is sampled from a power-law distribution and inserted into the prompt. The authors test five manually constructed and six automatically generated prompts on GPT-3.5-turbo and GPT-4o, measuring how close the delivered code change is to the requested rate. They then run LLaMEA with the best prompt per model plus dynamic rates and compare convergence against raw baseline data from [18]. They report that GPT-4o follows the rate instructions much better than GPT-3.5-turbo, and that with GPT-4o the dynamic-rate version converges faster than the baseline.

Significance. The prompt-adherence study is a useful, falsifiable empirical contribution: it shows a clear qualitative difference between two GPT models and that longer, more explicit prompts improve rate adherence for GPT-4o. The second claim—that power-law dynamic rates improve convergence of LLaMEA—is potentially important for automated algorithm design, but in its current form it rests on an unverified external baseline and on very few runs without statistical tests. The theoretical grounding in fast mutation operators is appropriate, and no fitted parameters are used beyond beta=1.5, which is a strength. Reproducibility would be stronger with a released code/data artifact, which the manuscript does not mention.

major comments (4)
  1. [§5.2, Fig. 5] The convergence improvement attributed to dynamic mutation rates is obtained by comparing new runs against 'raw data directly from [18]' (Fig. 5 caption). Because the baseline was not re-run under the exact pipeline of this paper, the comparison is confounded: the model version (GPT-4o-2024-08-06 vs. whatever [18] used), the mutation prompt (default vs. explicit numeric rate), the BBOB instances, the number of runs, and random seeds are not shown to be identical. Given that Section 5.1 demonstrates that model capability changes mutation behavior substantially, this confound must be removed by re-running the default LLaMEA baseline under identical conditions or by providing a direct equivalence argument.
  2. [§5.2, Fig. 5] The claim that dynamic rates 'improve the convergence speed' with GPT-4o is based on five runs per condition with no significance test, confidence interval, or effect size. The paper mentions 'not significantly' only for GPT-3.5-turbo; the GPT-4o difference is reported without any uncertainty quantification. Please add an appropriate statistical comparison (e.g., Mann-Whitney U or bootstrap over independent runs) and report the number of runs and seeds.
  3. [§3, Eq. (2)] The central metric 'percentage of code modification' is never defined. It is used in Eq. (2) to compute MSE, in Eq. (3) for prompt selection, and in Figs. 4 and 6 to claim that GPT-4o adheres to requested rates. The authors should specify exactly how code difference is measured (e.g., line-based diff, token-level Levenshtein, AST edit distance) and how percentages are computed, since different definitions can change the prompt ranking.
  4. [§3, Eq. (1) and Eq. (3)] The TDW-score in Eq. (3) weights the five requested rates by P r(x_i), but Eq. (1) defines P r over α/n, where n is the number of lines of the parent code. The paper never states how a requested percentage x is mapped to α/n (e.g., α = floor(n*x/100)). Without this mapping, the TDW-score and the resulting choice of 'best prompt' are not reproducible. Please make the mapping explicit and justify using percentages 2, 5, 10, 20, and 40 as a discretization.
minor comments (5)
  1. [§2] Section 2 says LLaMEA uses an evolutionary strategy '(1, 1) or (1 + 1)', but Section 4.1 describes only the (1 + 1) strategy; please clarify which strategy was actually used in the experiments.
  2. [§3, Eq. (2)] In Eq. (2), the term log(x_i / x_i) is identically zero, so the first equality contains a redundant term; the notation should be simplified and checked.
  3. [Fig. 4] The caption of Fig. 4 does not explain the visual encoding of the columns (e.g., which prompt and mutation rate each column corresponds to) or how the green dotted-dashed line is computed; the reader should be able to interpret the panels without relying on the body text.
  4. [§4.1, experimental setup] The bullet list in Section 4.1 contains formatting issues, such as 'Code generation budget100 code instances' and 'the experiment is repeated 3 times for each combination'; please use consistent spacing and specify that these are independent runs.
  5. [§6] The phrase 'advanced automated cue engineering' in Section 6 should be 'prompt engineering', and in Section 2 'easily expend on' should be 'easily extend'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dynamic mutation rates come from external theory and the prompt-adherence metric is independent of the convergence comparison.

full rationale

The paper's central mechanism is not definitionally tied to its outcome. The heavy-tailed power-law distribution (Equation 1 with beta=1.5) is taken from Doerr et al. [7], an external theoretical source, not fitted to the LLaMEA convergence data. Prompt adherence is quantified by a separate MSE/TDW metric (Equations 2-3) that measures code-difference ratios, and this metric is used only to select prompts, not to construct the convergence curves. The final comparison in Fig. 5 uses 'raw data directly from [18]' as baseline; [18] is prior work by the present authors, but it is published benchmark data from an open-source framework, not a parameter fitted in this paper, so the comparison does not reduce to this paper's inputs. A possible confound, such as GPT-4o-2024-08-06 vs the model version used in [18], is a validity threat rather than circularity. No equation is equated to its own output, and no prediction is a renamed fit. Hence score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper relies on the analogy between bit-string mutation rates and LLM code-diff percentages, an undefined code-difference metric, and an externally sourced baseline. These are assumptions rather than fitted parameters, with beta=1.5 and the discrete calibration rates being the only hand-chosen numbers.

free parameters (2)
  • beta = 1.5
    Shape parameter of the power-law distribution in Eq. 1, taken from Doerr et al. [7] and fixed a priori; it governs the sampled mutation rates but is not fitted to the observed data.
  • requested mutation rates for prompt calibration = 2%, 5%, 10%, 20%, 40%
    Discrete rates hand-picked to represent the dynamic range of the power-law; they are used only for the reliability experiment.
assumptions (3)
  • domain assumption The power-law distribution over bit-string mutation rates (Eq. 1) carries over to LLM code mutations when the code length n is the number of lines and the mutation rate is the percentage of code changed.
    Section 3 states 'We set the number of lines of the parent code to n in Equation 1', assuming equivalence between theoretical mutation rate and code-diff percentage.
  • domain assumption The 'percentage of code modification' is measurable with a diff tool and is a faithful proxy for algorithmic mutation strength.
    Equation 2 depends on this percentage, but the paper never specifies the diff algorithm or normalization used to compute it.
  • domain assumption The raw baseline from [18] is comparable without re-running under identical conditions.
    Fig. 5 caption states 'Baselines are raw data directly from [18]' with no verification of matched setups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Controlling the Mutation in Large Language Models for the Efficient Evolution of Algorithms." pith.science (2026). https://pith.science/paper/X3GIL7II

@misc{pith2026241203250,
  author       = {Pith},
  title        = {Pith review of: Controlling the Mutation in Large Language Models for the Efficient Evolution of Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3GIL7II}},
  note         = {Machine review of arXiv:2412.03250}
}
read the original abstract

The integration of Large Language Models (LLMs) with evolutionary computation (EC) has introduced a promising paradigm for automating the design of metaheuristic algorithms. However, existing frameworks, such as the Large Language Model Evolutionary Algorithm (LLaMEA), often lack precise control over mutation mechanisms, leading to inefficiencies in solution space exploration and potentially suboptimal convergence. This paper introduces a novel approach to mutation control within LLM-driven evolutionary frameworks, inspired by theory of genetic algorithms. Specifically, we propose dynamic mutation prompts that adaptively regulate mutation rates, leveraging a heavy-tailed power-law distribution to balance exploration and exploitation. Experiments using GPT-3.5-turbo and GPT-4o models demonstrate that GPT-3.5-turbo fails to adhere to the specific mutation instructions, while GPT-4o is able to adapt its mutation based on the prompt engineered dynamic prompts. Further experiments show that the introduction of these dynamic rates can improve the convergence speed and adaptability of LLaMEA, when using GPT-4o. This work sets the starting point for better controlled LLM-based mutations in code optimization tasks, paving the way for further advancements in automated metaheuristic design.

Figures

Figures reproduced from arXiv: 2412.03250 by the authors.

Figure 1
Figure 1. The distribution of Equation 1. The area under the curves are all 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. In each iteration, LLaMEA mutates the parent metaheuristic algorithm [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. In Figs. 4a and 4b, the red dashed line represents the requested mutation rate, and the green dotted-dashed line indicates that the delivered code difference of the points on this line is equal to the requested mutation rate. Each column of data contains mutated codes from the 100 codes generated in 3 independent runs using LLaMEA and the requested mutation rate in each prompt. From Fig. 4a, we can see that progress… view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: MSE and TDW-scores of prompts. For both figures, the [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]
Figure 4
Figure 4. Figure 4: The distribution of code difference of the codes generated with different [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Mean convergence curves (best-so-far algorithm scores) over the 5 different [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Example code difference of 1 run out of 5 runs for each selected prompt [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design

    cs.AI 2025-01 conditional novelty 7.0 of 10

    MCTS-AHD organizes LLM-generated heuristics in a Monte Carlo tree and reports better or comparable heuristics than population-based baselines on several NP-hard problems and a Bayesian optimization task.

  2. How to Guide LLM Generation: Dual-Surrogate Guided Search for Automated Heuristic Design

    cs.NE 2026-07 conditional novelty 6.0 of 10

    Dual-Surrogate Guided Search selects operator-parent actions for LLM-based heuristic generation by predicting the child's latent distribution and its expected per-instance utility, improving sample efficiency on sever...

  3. BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics

    cs.SE 2025-04 conditional novelty 6.0 of 10

    BLADE is an open-source, modular benchmark suite for evaluating LLM-driven automated algorithm discovery on continuous black-box optimization problems.

  4. An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems

    cs.CL 2024-12 unverdicted novelty 3.0 of 10

    A survey and position paper that reviews LLM prompting, RAG, and RL techniques and argues they could support open-ended implementation generation, without presenting new results.

Reference graph

Works this paper leans on

24 extracted references · 13 canonical work pages · cited by 4 Pith papers

  1. [18]

    IEEE Transactions on Evolutionary Computation pp

    van Stein, N., Bäck, T.: Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Transactions on Evolutionary Computation pp. 1–1 (2024).https://doi.org/10.1109/TEVC.2024.3497793

    arXiv 2024

  2. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [2]

    In: International Conference on Parallel Problem Solving from Nature

    Antipov, D., Doerr, B.: Runtime analysis of a heavy-tailed genetic algorithm on jump functions. In: International Conference on Parallel Problem Solving from Nature. pp. 545–559. Springer (2020)

    work page 2020

  4. [3]

    Evolutionary computation1(1), 1–23 (1993)

    Bäck, T., Schwefel, H.P.: An overview of evolutionary algorithms for parameter optimization. Evolutionary computation1(1), 1–23 (1993)

    work page 1993

  5. [4]

    Natural computing 1, 3–52 (2002)

    Beyer, H.G., Schwefel, H.P.: Evolution strategies–a comprehensive introduction. Natural computing 1, 3–52 (2002)

    work page 2002

  6. [5]

    In: Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Kraków, Poland, September 11-15, 2010, Proceed- ings, Part I 11

    Böttcher, S., Doerr, B., Neumann, F.: Optimal fixed and adaptive mutation rates for the leadingones problem. In: Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Kraków, Poland, September 11-15, 2010, Proceed- ings, Part I 11. pp. 1–10. Springer (2010)

    work page 2010

  7. [6]

    ACM Transactions on Intelligent Systems and Technology15(3), 1–45 (2024)

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology15(3), 1–45 (2024)

    2024

  8. [7]

    In: Proceedings of the genetic and evolutionary computation conference

    Doerr, B., Le, H.P., Makhmara, R., Nguyen, T.D.: Fast genetic algorithms. In: Proceedings of the genetic and evolutionary computation conference. pp. 777–784 (2017)

    work page 2017

Show all 24 references
  1. [8]

    Hansen, N., Finck, S., Ros, R., Auger, A.: Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Ph.D. thesis, INRIA (2009)

    2009

  2. [9]

    Evolutionary computation9(2), 159–195 (2001)

    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary computation9(2), 159–195 (2001)

    2001

  3. [10]

    MIT press (1992)

    Holland, J.H.: Adaptation in natural and artificial systems: an introductory anal- ysis with applications to biology, control, and artificial intelligence. MIT press (1992)

    1992

  4. [11]

    Journal of Discrete Algorithms4(1), 181–199 (2006)

    Jansen, T., Wegener, I.: On the analysis of a dynamic evolutionary algorithm. Journal of Discrete Algorithms4(1), 181–199 (2006)

    2006

  5. [12]

    In: ProceedingsoftheGeneticandEvolutionaryComputationConferenceCompanion

    Lange, R., Tian, Y., Tang, Y.: Large language models as evolution strategies. In: ProceedingsoftheGeneticandEvolutionaryComputationConferenceCompanion. pp. 579–582 (2024)

    2024

  6. [13]

    arXiv preprint arXiv:2311.15249 (2023) Controlling Mutation in LLMs 15

    Liu, F., Tong, X., Yuan, M., Zhang, Q.: Algorithm evolution using large language model. arXiv preprint arXiv:2311.15249 (2023) Controlling Mutation in LLMs 15

    2023 arXiv

  7. [14]

    In: Forty-first International Conference on Machine Learning (2024)

    Liu, F., Xialiang, T., Yuan, M., Lin, X., Luo, F., Wang, Z., Lu, Z., Zhang, Q.: Evo- lution of heuristics: Towards efficient automatic algorithm design using large lan- guage model. In: Forty-first International Conference on Machine Learning (2024)

    2024

  8. [15]

    arXiv e- prints:2111.04077 (Nov 2021),https://arxiv.org/abs/2111.04077

    de Nobel, J., Ye, F., Vermetten, D., Wang, H., Doerr, C., Bäck, T.: IOHexper- imenter: Benchmarking Platform for Iterative Optimization Heuristics. arXiv e- prints:2111.04077 (Nov 2021),https://arxiv.org/abs/2111.04077

    2021 arXiv

  9. [16]

    Engineering Applications of Artificial Intelligence 60, 97–116 (2017)

    Ojha, V.K., Abraham, A., Snášel, V.: Metaheuristic design of feedforward neu- ral networks: A review of two decades of research. Engineering Applications of Artificial Intelligence 60, 97–116 (2017)

    2017

  10. [17]

    Procedia Computer Science51, 630–639 (2015)

    Penas, D.R., González, P., Egea, J.A., Banga, J.R., Doallo, R.: Parallel meta- heuristics in computational biology: An asynchronous cooperative enhanced scatter search method. Procedia Computer Science51, 630–639 (2015)

    2015

  11. [19]

    arXiv preprint arXiv:2410.16309 (2024)

    van Stein, N., Vermetten, D., Bäck, T.: In-the-loop hyper-parameter optimiza- tion for llm-based automated design of heuristics. arXiv preprint arXiv:2410.16309 (2024)

    2024 arXiv

  12. [20]

    arXiv preprint arXiv:2401.17842 (2024)

    vanStein,N.,Vermetten,D.,Kononova,A.V.,Bäck,T.:Explainablebenchmarking for iterative optimization heuristics. arXiv preprint arXiv:2401.17842 (2024)

    2024 arXiv

  13. [21]

    arXiv preprint arXiv:2401.10034 (2024)

    Wu, X., Wu, S.h., Wu, J., Feng, L., Tan, K.C.: Evolutionary computation in the era of large language model: Survey and roadmap. arXiv preprint arXiv:2401.10034 (2024)

    2024 arXiv

  14. [22]

    Metaheuris- tics in water, geotechnical and transport engineering1, 23 (2013)

    Yang, X.S.: Optimization and metaheuristic algorithms in engineering. Metaheuris- tics in water, geotechnical and transport engineering1, 23 (2013)

    2013

  15. [23]

    arXiv preprint arXiv:2303.18223 (2023)

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  16. [24]

    In: NeurIPS 2022 Foundation Models for Decision Making Workshop (2022),https://openreview.net/forum? id=YdqwNaCLCx

    Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., Ba, J.: Large language models are human-level prompt engineers. In: NeurIPS 2022 Foundation Models for Decision Making Workshop (2022),https://openreview.net/forum? id=YdqwNaCLCx

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.