{"id":"f0d885da-1639-4412-bb95-cc2b7d4348dd","arxiv_id":"2504.19636","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-assisted algorithm search landscapes are multimodal, rugged, sparsely connected, and task/LLM-dependent, and existing code-similarity metrics correlate only weakly with performance.","lead":"This paper builds graphs from LLM-driven algorithm search runs and analyzes their fitness landscapes, finding them multimodal, rugged, sparsely connected, and different across tasks and models. It also shows that four code-similarity metrics are weakly correlated with algorithm performance differences.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Landscape conclusions rest on a single search-trajectory graph per task/LLM; without a null-model or repeated-run baseline, the reported ruggedness and graph metrics may be sampling artifacts.","rationale":"The reader's weakest_assumption correctly identifies that the graph built from the search process may not represent the underlying fitness landscape. My stress-test agrees with this and sharpens it into a concrete, falsifiable concern: the graph metrics and multimodality claims in Sections IV.B and IV.C are measured on the search trajectory graph, which is confounded by the search algorithm's selection pressure, the LLM's conditional generation, and the limited, non-random sampling of nodes. The reader's conditional verdict already requires repeated runs and a non-search baseline, which is exactly what would resolve this concern. I therefore do not see a reason to change the verdict: the paper's claims are plausible but conditional on the trajectory graph being representative, and the requested experiments would settle it. No independent data or code is released to support the trajectory graph, and the text contains corrupted portions (e.g., Figure 8 and the surrounding similarity section), further motivating the need for public artifacts and replication. The concern is not about disagreeing with consensus; it is about the internal validity of using a search-generated graph to infer static landscape properties.","tokens_in":18264,"tokens_out":1779,"duration_ms":19463,"concrete_test":"Run the same EoH configuration on TSP with DeepSeek-V3 for at least 10 independent random seeds, and for each run compute the three graph metrics (density, average degree, clustering coefficient) plus the fitness-value distribution. Then construct two baselines: (a) a random-sampling baseline that generates 2,000 algorithms by prompting the LLM without evolutionary selection or fitness feedback, evaluates them, and builds a graph with the same number of nodes and random edges; and (b) a configuration-model null graph that preserves the observed degree sequence but randomizes edges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LAS landscapes are 'highly multimodal and rugged' with 'distinct structural variations across tasks and LLMs' depends on treating the graph constructed in Section III as representative of the underlying fitness landscape. The neighborhood relation is defined as ai in NS(aj) if ai is generated from aj, and nodes are only algorithms produced during the search. This graph is therefore a Search Trajectory Network, not a sample of the algorithm space. Section IV.B and IV.C then interpret graph-level metrics (density, average degree, clustering coefficient) and visual node distributions as properties of the landscape itself. But those metrics are heavily influenced by the search process: the EoH update step selects which offspring survive, the initial population is generated by prompts, and the LLM's generation distribution conditions on parents. Under this construction, 'low connectivity' may simply mean the search revisited few algorithms and made few transitions, not that the landscape has sparse connections. Similarly, the observed multimodal fitness distributions may reflect the selection pressure and the limited number of evaluated candidates (up to 2,000) rather than intrinsic landscape modality. The paper reports no repeated runs and no statistical comparison to a null model, so it is possible that a random generation of 2,000 algorithms without evolutionary feedback would produce similar fitness distributions and graph metrics. If so, the headline conclusions would not distinguish landscape properties from sampling artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a graph-based approach to characterize the fitness landscape of LLM-assisted Algorithm Search (LAS). In this representation, nodes are candidate algorithms generated during an evolutionary search (specifically EoH), and edges are parent-offspring transitions induced by LLM-based operators. The authors run experiments across six algorithm design tasks and six LLMs, visualize the resulting graphs in 2D and 3D, compute graph metrics (density, average degree, clustering coefficient), and plot fitness distributions. They conclude that LAS landscapes are highly multimodal and rugged, lack a funnel structure, and exhibit low connectivity, with structural variation across tasks and LLMs. They additionally evaluate four code-based similarity metrics (BLEU, weighted BLEU, AST Match, Dataflow Match) and report correlations with performance deltas and operator type. The paper's central structural claims rest on interpreting these search-trajectory graphs as faithful samples of the underlying algorithm space.","tokens_in":18420,"tokens_out":4989,"duration_ms":49721,"significance":"If supported, the paper would provide a useful first characterization of LLM-driven algorithm search landscapes and inform the design of better search operators. The empirical scope is broad: six tasks spanning combinatorial optimization, reinforcement learning, and symbolic regression, together with six LLMs, and the authors position their work relative to existing code-evolution-graph studies. The similarity-metric analysis is a reasonable, though preliminary, contribution. However, the central claims about multimodality, ruggedness, absence of funnels, and low connectivity are methodologically undermined because the graph is built from a single search trajectory, not from an independent or representative sample of the algorithm space. Without repeated runs, null-model comparisons, or quantitative landscape measures, the main conclusions are not established. The paper does not provide machine-checked proofs or a reproducible code repository in the text, but it does build on the open-source LLM4AD platform, which is a strength.","major_comments":[{"comment":"The graph-based representation defines the neighborhood as ai in NS(aj) if ai is generated from aj during the search, and nodes are only algorithms that the search happened to generate. This constructs a Search Trajectory Network, not a fitness landscape graph over the solution space S. As a result, the graph properties measured in Sections IV.B and IV.C are properties of the search process and the LLM's conditional generation distribution, not properties of the static algorithm space. To support claims about the underlying landscape, the authors must either reframe the paper as an analysis of search trajectory networks or supplement the construction with random or systematic sampling of the algorithm space and a null model that accounts for the search process.","section":"Section III"},{"comment":"The claims of 'low connectivity,' 'absence of funnel structure,' and 'multimodality' rest on visual inspection of 2D/3D layouts and on graph metrics that are directly shaped by the evolutionary search dynamics. For example, the EoH update step keeps a population of 20, and the run evaluates up to 2,000 candidates, so the average degree and density are bounded by the number of parents retained and the frequency of repeated transitions, not by the connectivity of the algorithm space itself. Without a null model (e.g., random generation of the same number of algorithms without selection, or a random graph with the same node set) or quantitative landscape measures (e.g., fitness-distance correlation, autocorrelation, local optima networks), the paper's central structural conclusions do not follow.","section":"Sections IV.B and IV.C (Table I)"},{"comment":"The experimental section describes a single run per task and LLM combination; no random seeds, repetitions, or statistical aggregation are mentioned. Single-run results cannot support statements such as 'the fitness landscape of LLM-assisted algorithm search is usually multi-modal with distinct patterns and characteristics across different algorithm design tasks and LLMs' (Section I). Repeated runs with different seeds are necessary to distinguish robust landscape properties from run-specific variability, and the paper currently provides no evidence that the observed structural differences across tasks and LLMs are reproducible.","section":"Section IV.A (Experimental Settings)"},{"comment":"The 2D and 3D 'landscape' visualizations appear to be graph layouts (the layout algorithm is not specified), whose coordinates have no defined relationship to any metric on the algorithm space. Visual observations such as 'absence of funnel structure' and the 'bimodal distribution' of the Car task are therefore not evidence about landscape geometry. Even if the visualizations are intended only as illustrations, the paper uses them as primary evidence for structural claims. Quantitative measures computed on a properly sampled space, with error bars across runs, are needed before making claims about multimodality or the presence or absence of funnels.","section":"Section IV.B, Figs. 3-7"}],"minor_comments":[{"comment":"The sentence 'Since their are no well-defined distance metrics or neighbourhood structure' contains a typo ('their' should be 'there'), and the preceding claim that 'the neighbourhood is instead characterized by correlations between algorithms' does not match the generation-based definition of NS(aj) given immediately below.","section":"Section III, last paragraph"},{"comment":"The table reports density, average degree, and clustering coefficient, but not the number of nodes or edges for each LLM. Without these counts, the density values (e.g., 0.00069 vs. 0.00223) are hard to interpret; please add the graph sizes.","section":"Table I"},{"comment":"The bullet stating that 'approximately 80% of the algorithms fall within the lowest 20% of fitness values' is ambiguous in a minimization setting, where lower values are better; please clarify whether 'lowest' refers to the best or the worst fitness values.","section":"Section IV.B"},{"comment":"The limitation paragraph says the analysis reveals 'small correlation coefficients to performance delta,' which appears to contradict the earlier statement in Section IV.F.a that Dataflow Match and AST Match yield 'the highest correlations with performance delta' for TSP. Please reconcile these two statements by reporting the actual coefficient values.","section":"Section IV.F.e"},{"comment":"The 5x5 matrix of subplots is very dense and difficult to read in the printed version; consider enlarging the figure or moving it to supplementary material, and explicitly state what the regression lines and shaded areas represent in the caption.","section":"Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely topic, but the gap between search trajectory networks and fitness landscape analysis is fundamental. I would encourage the editor to require the authors to either (a) add multiple runs, a null-model baseline, and quantitative landscape measures that are not dependent on the search trajectory, or (b) substantially reframe the paper as an analysis of search trajectory characteristics rather than landscape structure. The related work [20] already introduces Code Evolution Graphs; the novelty of this submission as a landscape study would be strengthened by demonstrating what additional insight beyond CEGs is provided. If the authors cannot implement the methodological improvements within a revision, the paper's central claims should be correspondingly softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: worth reading as a first map of what LLM-assisted algorithm search looks like on a few tasks, and the similarity-metric comparison is a useful empirical contribution. But the central landscape characterization is not as solid as the abstract and conclusion say. The graph nodes are algorithms the search happened to generate; edges are parent-offspring transitions. That is a Search Trajectory Network, not an independent sample of the algorithm space. So low density, small clustering, and the visual ruggedness partly encode the search process and the LLM's generation distribution, not the intrinsic fitness landscape. The paper reports single runs and no null model; a random set of 2,000 generated algorithms could plausibly show similar distributions. That concern lands.\n\nWhat is genuinely new: this is the first dedicated fitness-landscape analysis of LAS, adapting STNs and CEGs to this setting, and covering six tasks and six LLMs. The distribution plots in Fig. 4 are informative, and the observation that Bact is hard and Car is bimodal is concrete. The similarity study with four metrics is honest: they find weak correlations and openly say code similarity metrics are inadequate. That is a real gap worth pointing at.\n\nSoft spots, in proportion. First, the graph-based landscape claims need repeated runs with error bars and a null baseline, e.g., random algorithm generation or a different LLM proposal distribution. Without that, 'low connectivity' and 'multimodal' are just descriptions of one EoH run. Second, the paper never formally defines the algorithm space S; it says source code, but the neighborhood is defined operationally. That makes the fitness-landscape terminology loose, though not wrong. Third, there is no code/data release, and parts of the similarity section (Figs. 8 and 11) are corrupted in the extracted text, with /uni00000013/ strings. That must be fixed for any archival version. Fourth, the interpretation that task differences correspond to intrinsic landscape differences is undercut because each task uses different instance sets and evaluation protocols; per-task difficulty alone could explain the distribution shapes.\n\nNone of this is disqualifying. The paper is a reasonable exploratory study, the similarity-metric findings are reproducible in principle, and the limitations are stated. It deserves a serious referee, but the referee should ask for stronger evidence for the headline claims. Who it's for: researchers working on LLM-based algorithm design who want a first sense of what the search space looks like and what similarity metrics do not capture. I'd encourage engagement, conditional on revisions that add repeated runs, a null model, and clean code/data artifacts.","headline":"A useful first map of what LLM-based algorithm search actually walks, but the headline claims about ruggedness and multimodality outrun the evidence because the graph is built from the search itself.","tokens_in":19010,"tokens_out":1608,"would_cite":false,"duration_ms":16308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-assisted algorithm search operates on rugged, multimodal fitness landscapes that vary by task and model.","keywords":["fitness landscape","large language models","automated algorithm design","evolutionary search","graph-based landscape analysis","algorithm similarity","combinatorial optimization","search trajectory networks"],"falsifier":"Re-run the same LAS search but also generate a control set of algorithms by prompting the LLM independently (without a parent), then compare graph density, average degree, and clustering of the trajectory graph against a control graph built from random algorithm pairs; if the control shows similar low connectivity and multimodal fitness values, the landscape claims are artifacts of sampling rather than properties of the space.","tokens_in":1673,"feed_emoji":"🗺️","tokens_out":1990,"duration_ms":78113,"temperature":0.7,"pith_summary":"This paper sets out to characterize the fitness landscape faced by LLM-assisted automated algorithm search (LAS): the space of candidate algorithms that an LLM can generate and evolve. It claims that these landscapes are highly multimodal and rugged, particularly for combinatorial optimization heuristic design, with many local optima of similar quality, no funnel toward a single best region, and sparse connectivity between solutions. It further claims that the landscape structure differs across the six tested algorithm-design tasks and six LLMs, and that code-similarity measures (BLEU, weighted BLEU, AST match, dataflow match) correlate only weakly with performance differences. A reader should care because these properties determine whether and how LLM-based search can succeed: if the landscape is a field of disconnected peaks, then operators and models that escape local optima matter more than those that refine a single basin.","feed_headline":"LLM algorithm search maps to rugged, multimodal landscapes","feed_subtitle":"Graph analysis across six tasks shows structure shifts with task and model, and no funnel to a best region.","key_machinery":"The central object is the graph-based fitness landscape $G=(N,E)$, where each node is a candidate algorithm generated during one search run and each undirected edge records that one algorithm was generated from another via an LLM operator; edges are weighted by transition frequency. Because there is no natural distance in code space, the neighborhood relation is defined procedurally ($a_i$ is a neighbor of $a_j$ if $a_i = \\mathrm{LLM}(\\mathrm{Prompt}(a_j))$), and landscape properties are read off graph metrics, such as density, average degree, and clustering coefficient, along with fitness distributions. The argument also relies on four code-similarity measures (BLEU, weighted BLEU, AST match, dataflow match) used to relate code distance to performance delta and to operator behavior, with dataflow match treated as the most behaviorally meaningful.","core_discovery":"The paper's central claim is that the fitness landscape of LAS, built from a graph whose nodes are candidate algorithms produced during search and whose edges are parent-offspring transitions, is highly multimodal and rugged, especially in combinatorial optimization tasks, and that its structure varies across tasks and LLMs. Formally, the landscape is the triplet $(S, NS, f)$ with $f$ the mean performance over instances, and the neighborhood relation $NS$ defined by $a_i \\in NS(a_j)$ iff $a_i$ is generated from $a_j$ by an LLM operator. Measured on that graph, the paper reports multiple approximate local optima with similar fitness, no funnel structure, and low average degree and clustering coefficient, together with fitness distributions in which roughly 80% of heuristic-design algorithms sit in the lowest 20% of fitness, the Mountain Car task is bimodal, and symbolic regression leaves 98% of algorithms far from optimal. Across LLMs on TSP, graph density, average degree, and clustering coefficient differ markedly, showing different exploration-exploitation balances. Finally, the four code-based similarity metrics correlate with each other and with performance delta, with AST and dataflow matching correlating most strongly on TSP, but the correlations are small and variable, so the paper concludes that current code-similarity metrics are not adequate for LAS and that specialized ones are needed.","pith_inferences":["If the reported ruggedness is intrinsic, an immediately testable design rule is to favor mutation operators that make large syntactic jumps (low dataflow match) early in search, and to reserve semantically preserving operators for late refinement.","The graph metrics themselves could serve as a cheap diagnostic: an early run's density and clustering might predict whether a given LLM-task pair will get trapped, enabling adaptive operator or model selection mid-search.","The similarity-metric results suggest that an execution-trace-based or learned functional embedding of algorithms would outperform all four tested metrics at predicting performance delta, and such a metric could be validated directly on the paper's recorded trajectories.","A null-model comparison, randomly generated algorithms with the same number of nodes, would sharpen whether 'low connectivity' is a property of the algorithm space or an artifact of the trajectory sampling; that comparison is not in the paper but is directly computable from its data."],"forward_implications":["If the landscape has no funnel, then search methods that only refine around the current best will plateau; escaping many equal-quality local optima becomes the central challenge.","Different LLMs produce measurably different landscape connectivity, so model choice is a search-strategy choice: dense, clustered landscapes favor local refinement, while sparse ones favor global exploration.","Task differences imply that the same LAS method should not be expected to behave uniformly; combinatorial heuristic tasks concentrate most algorithms in low-fitness regions, while symbolic regression leaves almost all algorithms far from optimum.","Because code similarity correlates with performance delta only weakly, distance-based mechanisms such as diversity maintenance or niching cannot rely on current metrics; LAS-specific similarity measures are needed.","Larger population sizes broaden exploration and reveal more landscape complexity, so population size is a practical lever for balancing exploitation and exploration."],"supporting_citations":[{"why":"Provides the evolutionary-search framework and its four LLM-based operators used to generate all candidate algorithms and trajectories.","marker":"[1]"},{"why":"Prior graph-based study of LLM-driven algorithm design, which this paper extends by focusing on fitness landscape rather than structural features.","marker":"[20]"},{"why":"Supplies the formal fitness-landscape triplet (S, NS, f) that the graph representation instantiates.","marker":"[32]"},{"why":"Prior trajectory-network models motivate using search-generated nodes and transition edges when local optima are not well defined.","marker":"[34]"},{"why":"Supplies the TSP heuristic-design task and an early version of the evolutionary search algorithm used in experiments.","marker":"[35]"},{"why":"The platform on which all six tasks were evaluated and all algorithm-parent relationships were recorded.","marker":"[36]"},{"why":"Defines BLEU, one of the four code-similarity metrics used to measure algorithm distance.","marker":"[41]"},{"why":"Defines weighted BLEU, AST match, and dataflow match, the other three similarity metrics.","marker":"[42]"}],"fun_headline_variants":["LLM search landscapes: rugged, multimodal, no funnel","Fitness landscapes of LLM algorithm search are rugged and multimodal","LLM-designed algorithms: landscape structure varies by task and model","No funnel: LLM algorithm search landscapes are multimodal and rugged","LLM search terrain: multimodal, rugged, and task-dependent"],"cache_read_input_tokens":21120,"weakest_assumption_plain":"The paper treats the graph of algorithms actually generated during one search run, and the parent-offspring transitions between them, as an adequate representation of the underlying fitness landscape; if that trajectory sample is not representative of the algorithm space, the reported ruggedness and connectivity findings do not follow.","fun_headline_variants_meta":{"raw":{"variants":["LLM search landscapes: rugged, multimodal, no funnel","Fitness landscapes of LLM algorithm search are rugged and multimodal","LLM-designed algorithms: landscape structure varies by task and model","No funnel: LLM algorithm search landscapes are multimodal and rugged","LLM search terrain: multimodal, rugged, and task-dependent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1218,"prompt_tokens":961,"completion_tokens":257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":172}},"tokens_in":577,"tokens_out":257,"duration_ms":2686,"temperature":1.0,"reasoning_tokens":172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:46:12.436855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same LAS search but also generate a control set of algorithms by prompting the LLM independently (without a parent), then compare graph density, average degree, and clustering of the trajectory graph against a control graph built from random algorithm pairs; if the control shows similar low connectivity and multimodal fitness values, the landscape claims are artifacts of sampling rather than properties of the space.","supporting_citations":[{"cited_title":"Evolution of heuristics: Towards efficient automatic algo- rithm design using large language model,","cited_arxiv_id":null,"evidence_quote":"Provides the evolutionary-search framework and its four LLM-based operators used to generate all candidate algorithms and trajectories."},{"cited_title":"Code evolution graphs: Understanding large language model driven design of algo- rithms,","cited_arxiv_id":null,"evidence_quote":"Prior graph-based study of LLM-driven algorithm design, which this paper extends by focusing on fitness landscape rather than structural features."},{"cited_title":"Fitness landscapes,","cited_arxiv_id":null,"evidence_quote":"Supplies the formal fitness-landscape triplet (S, NS, f) that the graph representation instantiates."},{"cited_title":"Search trajectory networks of population-based algorithms in continuous spaces,","cited_arxiv_id":null,"evidence_quote":"Prior trajectory-network models motivate using search-generated nodes and transition edges when local optima are not well defined."}],"review_version":1}