{"id":"98d7ed5d-89f0-44ba-845d-a04c26ac2c92","arxiv_id":"2412.08072","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM-driven evolutionary search, LLM-PSO, finds near-optimal airfoil and Stokes-flow body shapes on two benchmarks, generally converging faster than classical optimizers.","lead":"Researchers used a large language model as the optimizer for engineering shape design, letting Claude 3.5 Sonnet propose new airfoil and underwater-body shapes based on past simulation results. The method matched known optimal shapes on two benchmark flow problems and often converged faster than genetic or reinforcement-learning optimizers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No leakage control leaves open that Claude 3.5 Sonnet is recalling known optimal airfoil and Stokes profiles rather than optimizing from the provided records; this threatens the generality claim, though the paper's nF=4 failure offers partial counter-evidence.","rationale":"The paper's empirical results are plausible, but the paper's goal is to demonstrate that an LLM can perform parametric shape optimization via in-context learning. The strongest threat to that demonstration is that the two benchmark problems are well-known, and the closed-source model may have memorized their solutions. The paper offers no control to rule this out, and the only hint against memorization—the nF=4 failure—is double-edged because it also undercuts the abstract's 'successfully identifies optimal shapes in both cases' claim. A leakage test would settle the matter: if the method works on a scrambled or novel problem, the records are doing the work; if not, the conclusions must be sharply qualified. The reader's conditional verdict already accounts for this unresolved concern and for the missing code/data; my read does not change that. I therefore recommend no change to the verdict, while insisting that the requested control be part of any revision.","tokens_in":28473,"tokens_out":8754,"duration_ms":82246,"concrete_test":"Run LLM-PSO on the Stokes problem with a fixed random permutation applied to the coefficient vector (i.e., present the records with A_k relabeled by a secret random bijection unknown to the LLM), so any memorized optimal coefficient vector is invalid. If the algorithm still converges to the true minimum drag (matching the theoretical value), the result is not attributable to memorization; if it fails, the claim should be restricted to settings where the LLM has relevant prior knowledge. Alternatively, use a non-canonical objective such as drag minimization with a non-standard volume constraint that has a known but unpublished optimum.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LLM-PSO 'successfully identifies optimal shapes'—would be unremarkable if the LLM's pretraining already contains the solutions. The airfoil benchmark follows Viquerat et al. (2021) and the Stokes benchmarks are classic theoretical results (Bourot 1974; Montenegro-Johnson & Lauga 2015), all likely present in Claude 3.5 Sonnet's internet-scale training data. The paper provides no leakage test: no scrambled parametrization, no withheld benchmark, and no comparison against a prompt variant that omits records. The only indirect evidence against memorization is the nF=4 airfoil failure, which shows the LLM did not simply recall the known optimum; however, that failure also weakens the 'success' claim. For the Stokes case, the Legendre coefficients of the theoretical optimum are publicly known, and the LLM could propose near-optimal coefficients even without extracting signal from the records. Without a control, the paper's conclusion that in-context records drive the optimization is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLM-PSO, an evolutionary-strategy-style optimizer in which an LLM (Claude 3.5 Sonnet) receives a prompt containing selected historical design-performance records and returns the mean of the next generation's Gaussian-sampled design vectors. The method is applied to two fluid-dynamics shape optimization problems: a 2D Bézier-parametrized airfoil at Re=100 maximizing lift-to-drag ratio, and a 3D axisymmetric body in Stokes flow parameterized by Legendre coefficients, minimizing drag under fixed area or volume. The authors report that LLM-PSO reproduces benchmark optimal shapes in both settings and generally converges faster than the RL baseline of Viquerat et al. and a genetic algorithm, while acknowledging that the nF=4 airfoil case fails to reach the benchmark optimum.","tokens_in":28696,"tokens_out":4032,"duration_ms":45779,"significance":"If the claims hold, the paper is a useful demonstration that a prompt-driven, zero-retraining LLM can act as a search operator for low-dimensional parametric shape optimization, with a simple interface and competitive convergence. The Stokes-flow results are the strongest part: they include five-run statistics, comparison against a GA with matched population size, and agreement with externally known theoretical optima for K=2..6. The airfoil experiments are also informative, and the explicit acknowledgment of the nF=4 failure is a sign of careful reporting. However, the central claim that in-context records drive the optimization is not yet established, because the two benchmark optima may be present in the LLM's pretraining data and the paper contains no leakage-control experiment. The significance is therefore conditional on additional control tests.","major_comments":[{"comment":"The paper contains no control for pretraining contamination. The airfoil benchmark follows Viquerat et al. [32] and the Stokes problems are standard published results ([72], [73]); a closed, internet-scale LLM such as Claude 3.5 Sonnet may have seen the optimal airfoil profiles and the known Legendre coefficients during training. Agreement with these benchmark optima therefore does not by itself show that the LLM is extracting signal from the records in the prompt. The authors should add at least one leakage control, for example: (i) a scrambled or rotated parameterization whose optimum shape is identical but whose numerical coordinates differ from any public record; (ii) a held-out benchmark whose optimum is not in the training data; or (iii) an ablation in which the prompt contains random records or records from a different problem. The nF=4 failure is partial counter-evidence but not sufficient, since it only shows the LLM does not always recall the exact optimum. This is load-bearing because the abstract's claim that LLM-PSO 'successfully identifies optimal shapes' collapses if the LLM is recalling known solutions rather than optimizing from the provided records.","section":"§IV.A.3, §IV.B.3"},{"comment":"The convergence-speed claim relative to RL is not statistically supported for the airfoil case. Fig. 3(c) shows single trajectories without error bars or multiple-run statistics, and the text does not state how many RL runs were used. Given that the abstract claims LLM-PSO 'generally converges faster than other classical optimization algorithms,' the airfoil comparison should either include repeated-run statistics with mean/min-max bands (as done for the Stokes case in Fig. 5) or be explicitly qualified as a single-trial observation. This is especially important because the reported nF=4 behavior, verified only through 'private communications' with the authors of [32], is not independently checkable and directly weakens the 'success' claim for that case.","section":"§IV.A.3, Fig. 3(c)"},{"comment":"The manuscript does not report the numerical values of the key hyperparameters, so the experiments are not reproducible from the text. The population size N, the standard deviation sigma^2 of the Gaussian sampler, the number of random initial generations nini, the prompt-selection counts T, R, M, the integer encoding resolution, and the full prompt text used for each of the two benchmark problems are either omitted or only illustrated by a generic example (Fig. 2). Since the method's behavior depends directly on these choices, they should be listed in the Methods or in a table. The authors' statement that code 'will be open-sourced upon acceptance' does not help the reviewer verify the current claims. I would treat this as a major issue for a methods-oriented paper.","section":"§III.A.1, §III.A.2, §IV.A.2, §IV.B.2"}],"minor_comments":[{"comment":"The sentence 'invoking us to contempt whether LLMs possess similar potential' should read 'prompting us to consider whether', or similar.","section":"§II.D"},{"comment":"In the caption description for Fig. 5, 'K = −5' appears to be a sign error; it should presumably read 'K = 5'.","section":"§IV.B.3"},{"comment":"The sentence 'This problem [36] was previously addressed using RL' cites [36], but the surrounding text and the benchmark reference suggest the intended citation is [32]. Please verify all citation numbers after the final compilation.","section":"§IV.A.1"},{"comment":"For the Reynolds-number-100 airfoil flow, the text states that the flow is unsteady and periodic and that the time-averaged lift-to-drag ratio is used, but it does not specify the time-averaging interval, the number of shedding periods, or the mesh-convergence criterion. A brief statement would improve confidence in the reported objective values.","section":"§IV.A.2"},{"comment":"The Stokes solver description reports domain size, element count, and boundary conditions, but not the corresponding verification of mesh independence; adding one sentence on this would align the level of detail with the strength of the Stokes claims.","section":"§IV.B.2"}],"recommendation":"major_revision","confidential_remarks":"The omission of a leakage control is the main editorial risk: for LLM-based optimization papers, reviewers will expect either an explicit leakage test or a scrupulously hedged claim. The authors' Stokes validation is a genuine strength and should be credited; the requested controls and statistics are within the scope of a revision rather than grounds for rejection. I would also encourage the editor to treat the 'private communication' about the RL failure with caution, as it is not verifiable from the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First and most important: this is the first paper I know of that runs an LLM as the optimizer (not as a surrogate, not as a design generator) directly on parametric shape optimization with PDE-evaluated objectives. The paper explicitly builds on Lange et al.'s \"LLMs as evolution strategies\" [52] and adapts that recipe to Bézier airfoils and Legendre-parametrized Stokes bodies. That is a real domain transfer, and the two benchmarks are nontrivial.\n\nThe Stokes-flow results are the strongest part. For K=2..6, the converged drag values match the theoretical Bourot and Montenegro-Johnson/Lauga profiles, they report five-run mean and min-max ranges, and the comparison with GA uses the same population size. This is credible evidence that in-context records plus Gaussian resampling can drive a low-dimensional PDE-constrained optimization. The airfoil results are thinner: only single trajectories in Fig. 3(c), visual agreement with Viquerat et al. for nF=1..3, and the paper itself acknowledges that nF=4 fails to reproduce the reference optimum. That failure is honest, but it also directly contradicts the abstract's blanket claim that LLM-PSO \"successfully identifies optimal shapes\" in both cases. The authors say the RL baseline also fails on nF=4, but that is backed by \"private communications\" rather than data, which is weak.\n\nThe biggest soft spot is the absence of any leakage control. Claude 3.5 Sonnet is a closed model trained on internet-scale data, and both benchmarks are classic. A reviewer should require either a scrambled parametrization, a withheld benchmark, or a prompt ablation without records. The nF=4 failure is partial counter-evidence against pure memorization, but it is not enough: a model can recall the well-known airfoil optimum for nF=1..3 and still fail on an obscure corner of the same family. The second soft spot is reproducibility: code is promised after acceptance, not shipped; \"available on reasonable request\" for data is not adequate for a computational methods paper. Hyperparameters T, R, M, sigma^2, nini are not given beyond the prompt example, though the paper does test nini in Fig. 6. Minor: there is a typo \"K = -5\" in the Fig. 5 caption, and the conclusions section overstates generality. Citations are appropriate: [52] is credited as the direct inspiration, and the two benchmark problems are properly attributed.\n\nNone of this makes me doubt the core idea. The framework is simple, the Stokes-flow results are solid, and the limitations are stated in the text. The paper deserves a serious referee, and it should go back for revision rather than be rejected. Before acceptance, I would want the leakage control, the nF=4 discussion backed by actual RL results, and code/data release. The right audience is the LLM-for-optimization community and CFD engineers who want a low-code, gradient-free optimizer. I would cite it as the first LLM-for-shape-optimization paper, but I would not treat it as a solid baseline until the leakage control is done.","headline":"First plausible LLM-driven shape optimization on PDE benchmarks; the Stokes evidence is solid, the airfoil evidence is thinner, and the missing leakage control is the main open question.","tokens_in":29272,"tokens_out":5362,"would_cite":true,"duration_ms":50123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A prompt-driven LLM optimizer matches benchmark optima in two flow shape problems.","keywords":["parametric shape optimization","large language models","in-context learning","evolutionary strategies","airfoil optimization","Stokes flow drag minimization","Bézier parametrization","Legendre polynomials"],"falsifier":"Re-run both benchmark optimizations after applying a fixed random permutation to the integer-coded design parameters (or renaming the variables and axes), so any memorized profile is scrambled; if LLM-PSO still converges to the physical optimum, the result is reasoning, while a collapse to near-random search would show the earlier success came from recall.","tokens_in":28247,"feed_emoji":"📐","tokens_out":5149,"duration_ms":50467,"temperature":0.7,"pith_summary":"The paper proposes LLM-PSO, a parametric shape optimizer that steers a population of designs by asking a large language model to read the best-performing records from past generations and propose the mean of the next generation's sampling distribution. It applies this scheme to two flow benchmarks: a two-dimensional airfoil in laminar flow, where the aim is to maximize the relative lift-to-drag ratio, and a three-dimensional axisymmetric body in Stokes flow, where the aim is to minimize drag under a fixed volume or surface area. In both settings the recovered shapes agree with published optimal profiles, and on the tested cases the LLM-driven search generally converges faster than reinforcement learning and genetic algorithm baselines. The intended significance is that a frozen, prompt-only model can act as an evolutionary search operator for engineering design without retraining or gradients.","feed_headline":"Prompted LLM matches benchmark optimal shapes in two flow problems","feed_subtitle":"Zero-retraining optimizer finds drag-minimizing airfoils and Stokes bodies, often faster than RL or genetic algorithms.","key_machinery":"The load-bearing mechanism is the evolutionary-strategy loop: the LLM is asked to act as an optimizer, sees a few-shot prompt of the top M designs from the best T and most recent R generations, and outputs an integer-coded mean design vector that seeds the next Gaussian population. The design vector is scaled to integers between 0 and 1000 to sidestep the model's weakness with floating-point text, and the temperature is set to zero so the model's answer is deterministic. All candidate designs are evaluated by a numerical flow solver, and the evaluated design-objective pairs are appended to a record buffer that feeds the next prompt. The loop requires no gradient information and no model retraining, so the LLM's only role is to translate observed records into the next search direction.","core_discovery":"The central claim is that in-context learning alone is enough to make a large language model a competitive evolutionary optimizer for low-dimensional parametric shape problems. Treating the LLM as an evolutionary strategist, the method feeds it a text prompt listing top-performing design vectors and objective values from selected generations, and the LLM responds with a proposed mean vector; new designs are then sampled from a Gaussian centered at that mean and evaluated with a PDE solver. On the airfoil task with three to nine free degrees of freedom, LLM-PSO reaches the reinforcement-learning benchmark optimum and needs fewer iterations in most cases; on the Stokes task with two to six Legendre coefficients, it recovers the theoretically optimal drag-minimizing profiles for both area and volume constraints. The paper also reports the boundary of the claim: with four free Bézier control points the airfoil optimum is not reproduced, and the authors treat higher-dimensional problems as open future work.","pith_inferences":["An unstated but testable implication is that the LLM's prior knowledge of classical aerodynamic and Stokes-drag results could be contributing to the fast convergence; permuting the parameter encoding before prompting would separate in-context reasoning from memorized profiles.","Since the paper uses a closed commercial model with no reproducibility guarantees, an open-weight model with the same prompt would be the natural control experiment to see whether the optimizer's skill is tied to the specific model.","The integer coding and Gaussian resampling suggest a direct bridge to discrete and mixed-variable evolutionary search, so LLM-PSO could be applied to combinatorial design spaces where continuous gradients do not exist.","A cheap extension would be to let the LLM propose a full covariance or a shortlist of candidate means instead of a single mean, potentially recovering lost performance at higher degrees of freedom."],"forward_implications":["If the central claim holds, a frozen LLM can act as a drop-in evolutionary operator for parametric shape optimization, removing the need to train a surrogate or policy.","The faster convergence observed on the two benchmarks implies fewer expensive CFD evaluations to reach the same objective, which matters when each evaluation is a nonlinear or Stokes flow solve.","For the airfoil case, matching the reinforcement-learning benchmark at three, six, and nine degrees of freedom suggests prompt-based search can be competitive where gradient information is unavailable or chaotic.","The reported failure at twelve degrees of freedom marks a clear scalability boundary: the method, as configured, does not yet handle higher-dimensional shape spaces.","Because the framework only needs a prompt and a record buffer, the same code path extends to any parametrization whose candidate vectors can be evaluated by a black-box objective."],"supporting_citations":[{"why":"Supplies the reinforcement-learning airfoil benchmark, the objective function R, and the convergence curves LLM-PSO is compared against.","marker":"[32]"},{"why":"Provides the evolutionary-strategy prompting approach the framework is built on, treating the LLM as the population-mean proposer.","marker":"[52]"},{"why":"Supplies the finite-element solver used to evaluate airfoil lift and drag at each generation.","marker":"[71]"},{"why":"Gives the theoretical optimal Stokes drag profile for fixed volume that validates the volume-constrained results.","marker":"[72]"},{"why":"Gives the theoretical optimal Stokes drag profile for fixed surface area that validates the area-constrained results.","marker":"[73]"}],"fun_headline_variants":["LLM-PSO matches benchmark airfoil and Stokes optima","Zero-retraining LLM optimizer beats RL on flow shapes","In-context LLM finds drag-minimizing shapes in fewer steps","Prompted LLM matches classic optimizers on 2D airfoil and 3D body"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison's validity rests on the assumption that the reported benchmark matches do not simply reflect the language model's prior exposure to the known optimal airfoil and Stokes profiles, since the paper runs no leakage or scrambling control.","fun_headline_variants_meta":{"raw":{"variants":["LLM-PSO matches benchmark airfoil and Stokes optima","Zero-retraining LLM optimizer beats RL on flow shapes","In-context LLM finds drag-minimizing shapes in fewer steps","Prompted LLM matches classic optimizers on 2D airfoil and 3D body"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1575,"prompt_tokens":926,"completion_tokens":649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":568}},"tokens_in":542,"tokens_out":649,"duration_ms":6419,"temperature":1.0,"reasoning_tokens":568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:14:40.095353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both benchmark optimizations after applying a fixed random permutation to the integer-coded design parameters (or renaming the variables and axes), so any memorized profile is scrambled; if LLM-PSO still converges to the physical optimum, the result is reasoning, while a collapse to near-random search would show the earlier success came from recall.","supporting_citations":[{"cited_title":"Shape optimization of a fan- shaped hole to enhance film-cooling effectiveness,","cited_arxiv_id":null,"evidence_quote":"Supplies the reinforcement-learning airfoil benchmark, the objective function R, and the convergence curves LLM-PSO is compared against."},{"cited_title":"Direct shape optimization through deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the evolutionary-strategy prompting approach the framework is built on, treating the LLM as the population-mean proposer."},{"cited_title":"Large language models asevolutionstrategies,","cited_arxiv_id":null,"evidence_quote":"Gives the theoretical optimal Stokes drag profile for fixed volume that validates the volume-constrained results."}],"review_version":1}