{"id":"98c4c878-5787-42f4-bace-6ef6a5432eb8","arxiv_id":"2608.07395","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PACE decouples reusable code components from host algorithms in LLM-based evolution, preserving them across generations, and reports stronger results than prior AAD methods on four tasks.","lead":"This paper introduces PACE, a method for automated algorithm design that saves useful code snippets, called Executable Algorithmic Primitives, in a persistent set separate from the full algorithms built on them. Later algorithms can reuse those snippets, and PACE uses Thompson sampling to pick which snippets to promote, reporting competitive results on four benchmark tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. 6-7 rewards confound EAP injection with uncontrolled LLM edits and P2's removal target, so the Thompson posterior may not measure transferable value.","rationale":"PACE is a well-specified system: the operator contracts (Eqs. 10-13) and persistence invariant (Eq. 5) are clear, and the ablations show each component contributes. The four-task results are directionally consistent, and Table 11's token accounting is an honest check on the claim of no extra evaluation budget. However, the central explanatory mechanism, that Thompson sampling selects EAPs by their true transfer value, rests on the validity of the parent-relative reward. The reader's weakest assumption identifies the same region: the Limitation section admits independence of primitives is assumed. My stress-test sharpens this: the reward is confounded not only by primitive-primitive coupling but also by any LLM edit outside the focus EAP and by the removal of e− in P2. The AST verifier cannot detect these confounds. Because the Related Work section explicitly claims causal isolation, and because the without-TS ablation shows the selection mechanism matters, a biased posterior would undermine the stated explanation even if the final scores stand. The proposed diagnostic, re-estimating posteriors on minimal-edit trials, would settle the question without introducing new assumptions. I therefore concur with the CONDITIONAL verdict: accept pending a demonstration that the reward signal tracks EAP transferability, or a revision that weakens the causal claim to a predictive one.","tokens_in":22720,"tokens_out":13944,"duration_ms":131119,"concrete_test":"From the logged P1/P2 trials (or a rerun with logging), compute an AST diff between each child and its parent, and split the change into the focus-EAP call plus its direct wiring versus all other edits. Recompute the Eq. 8 posterior using only trials whose non-EAP edit distance is zero (or minimal), then rerun Thompson selection on this re-estimated posterior. If the EAP ranking or the final algorithm score changes materially relative to the published results, Eqs. 6-7 are confounded and the causal attribution is unsupported; if the ranking is stable, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Thompson-sampled EAP transfer (Eqs. 8-9) identifies and reuses valuable algorithmic components. The Related Work section asserts that PACE \"isolate[s] an EAP's causal impact from host program noise\" via parent-relative rewards. The trial design does not achieve this isolation. P1 (Eq. 10) and P2 (Eq. 11) constrain only the call set; the LLM is explicitly allowed to reorganize, refine, or adapt all non-EAP code. P2 additionally removes a target EAP e−, so r_e = I[J(A') > J(A)] conflates the benefit of e+ with the benefit of deleting e− and with any other edits the LLM makes. The AST verifier checks only call-set membership, not code equivalence outside the calls. Consequently, Eq. 7's ρ_e is not a property of e alone; it is a joint function of e, the specific parent, and the LLM's unconstrained modifications. The Beta update (Eq. 8) then treats these confounded outcomes as i.i.d. draws from a fixed Bernoulli rate, although parents come from an evolving population whose baseline quality rises over time, making success probabilities non-stationary. The paper's own Limitation section concedes that coupling between primitives can break independent evaluation, but the confound is broader: even additively independent EAPs receive biased credit whenever the LLM changes unrelated code. Since the ablation without Thompson sampling shows selection matters, a biased posterior could mean PACE succeeds despite, rather than because of, its credit-assignment mechanism, undercutting the stated explanation of why primitive transfer helps.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PACE, a component-level evolutionary framework for LLM-based automated algorithm design. PACE decouples local algorithmic logic from complete programs by maintaining a persistent library of Executable Algorithmic Primitives (EAPs), each a callable function with a fixed implementation. Four primitive-aware operators (insert, replace, refine, crossover) constrain EAP call sets in generated offspring, and an AST verifier enforces these contracts while allowing structurally invalid offspring to be repaired. EAP selection is guided by Thompson sampling over Beta posteriors updated by parent-relative binary rewards (Eqs. 6-8). The paper reports experiments on Racing Car, Bipedal Walker, TSP-Construct, and TSP-ACO, with ablations on the selection mechanism, operators, EAP discovery modules, the parameter k, and the LLM backbone.","tokens_in":23070,"tokens_out":8317,"duration_ms":68344,"significance":"If the central claims hold, PACE addresses a real limitation of whole-program evolution: reusable local logic is lost when a host program is discarded. The formal state definitions (Eqs. 1-5) and operator contracts (Eqs. 10-13) are internally consistent, and the AST-verifier-based rejection of structurally invalid offspring is a concrete engineering safeguard. The paper also reports k-sensitivity and token consumption, and its reward signal comes from external evaluators rather than from fitting the test data, so there is no circularity. The main risk is that the credit-assignment mechanism--the load-bearing part that supposedly identifies which components are valuable--has not been shown to measure what it claims, and the empirical comparisons lack inferential support.","major_comments":[{"comment":"The parent-relative reward r_e = I[J(A')>J(A)] does not isolate the injected EAP's causal contribution. P1 permits the LLM to reorganize, refine, and adapt all non-EAP code (Eq. 10), and P2 simultaneously removes a target EAP e^- while injecting e^+ (Eq. 11), so a successful trial can be caused by the deletion of e^- or by unrelated LLM edits rather than by e^+. The AST verifier checks only call-set membership, not code equivalence outside those calls. The Related Work claim that PACE 'isolate[s] an EAP's causal impact from host program noise' is therefore unsupported. I request either a revised claim or a controlled test: log non-EAP code changes per trial and re-estimate the posterior conditioned on trials where non-EAP code is unchanged, or run an ablation in which P1/P2 forbid non-EAP edits.","section":"Related Work / Primitive-Aware Operators, Eqs. (6)-(8), (10)-(11)"},{"comment":"Equation (7) defines ρ_e as a fixed Bernoulli success rate, but the conditioning event is incomplete: the probability depends on the specific parent A and on the unconstrained LLM modifications. Because the parent population's baseline quality rises over time, P[J(A')>J(A)] for a fixed EAP is non-stationary, so the Beta-Bernoulli update in Eq. (8) treats observations that are not i.i.d. draws from a common rate. The Thompson sampling draw in Eq. (9) then responds to a posterior that can drift for reasons unrelated to the EAP's transferable value. Reporting time-binned success rates per EAP or using a drift-aware model would make the selection mechanism's validity testable.","section":"Adaptive EAP Selection, Eqs. (7)-(9)"},{"comment":"Test scores and out-of-domain results are reported as point estimates without error bars, confidence intervals, or significance tests. Several headline advantages are within one standard deviation of a baseline (Racing Car: PACE 92.40±7.69 vs HSEvo 84.17±5.57; Bipedal Walker: PACE 67.06±72.63; TSP-ACO in-domain: PACE 5.795 vs ReEvo 5.774), and the TSP tables have no variance information at all. The conclusion that PACE 'effectively discovers competitive algorithms' needs per-run test scores and a paired comparison across the three runs (e.g., Wilcoxon signed-rank or bootstrap) before it is supported.","section":"Tables 1-3 and Figure 3"},{"comment":"The stated limitation--that primitives may not be independently evaluable under strong coupling--is narrower than the actual confound. Even with additively independent EAPs, Eq. (6) assigns credit to e^+ whenever the whole child improves, including improvements caused by arbitrary LLM edits in P1 or by the deletion in P2. Because the ablation in Table 4 shows that removing Thompson sampling causes a severe drop, the empirical case depends on the posterior being informative; a biased posterior could mean PACE succeeds despite, rather than because of, its credit-assignment mechanism. The authors should address this by showing that the learned posterior correlates with out-of-sample EAP transfer success, for example by holding out a subset of EAPs and comparing Thompson-selected vs randomly selected transfers.","section":"Limitation and future work"}],"minor_comments":[{"comment":"The statement that 'the only parameter k=3' is an overstatement: Table 8 lists population size, operator-selection probabilities, the one-proposal-per-generation rule, and the three repair attempts as fixed settings; a sentence acknowledging these as fixed defaults would be more accurate.","section":"Experimental Setup / Table 8"},{"comment":"The text says the w/o EAP Generation difference of 0.009 'falls within natural noise' yet concludes 'confirms the overall necessity'; this should be rephrased to avoid claiming confirmation from a within-noise difference.","section":"Table 4"},{"comment":"Figure 3 axis labels and some table formatting are corrupted in the manuscript PDF (e.g., 'NumberofEvaluations'); please ensure the camera-ready version renders them correctly.","section":"Figure 3"},{"comment":"Equation (7) conditions on e∈C(A') but not on the parent A; since the reward is parent-relative, the notation (or an accompanying sentence) should make it explicit that ρ_e is averaged over the parent distribution.","section":"Eq. (7)"},{"comment":"Table 11 compares token consumption, but the budget definition in Algorithm 1 counts 'complete-program evaluations'; clarify whether the 1,000-evaluation budget includes the three repair attempts per offspring and whether the EAP discovery LLM calls are charged to any comparable budget for baselines.","section":"Algorithm 1 / Token Usage"},{"comment":"Figure 1 caption 'abstains a better score' should be 'attains a better score'; also 'summary ara shown' in Detailed Methodology should be 'summaries are shown'.","section":"Figure 1 caption and Detailed Methodology"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is within scope and the core idea is interesting. The main risk is the credit-assignment confound in Eqs. (6)-(8); I would not accept before the authors either soften the causal claims or provide a controlled validation that the posterior tracks transferable EAP value. The empirical section also needs inferential support. The use of MCTS-AHD as a baseline is standard practice and is not a concern, but the discrepancy between the paper's 'only parameter k=3' claim and the additional fixed settings in Table 8 should be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PACE is a serious entry into LLM-based automated algorithm design, and if I were an editor I'd send it to review. The new idea is to treat reusable code snippets as persistent first-class citizens—EAPs—with structural contracts verified by AST, and to guide their selection by Thompson sampling on parent-relative performance. The paper is unusually complete: formal definitions, operator contracts, prompts, ablation studies, and even token consumption. The four-task evidence (Racing Car, Bipedal Walker, TSP-Construct, TSP-ACO) is directionally strong, with PACE beating the AAD baselines at equal budget.\n\nThe main soft spot is in the credit assignment. The paper says it 'isolates an EAP's causal impact from host program noise,' but Eqs. 6–7 don't actually do that. The reward r_e is a binary improvement of the child over the parent; the child is produced by an LLM that is explicitly allowed to reorganize, refine, or adapt all non-EAP code. So the reward conflates the injected EAP with whatever else the LLM changed. In the replacement operator P2, the reward also conflates the benefit of e+ with the benefit of deleting e−. The AST verifier checks call-set membership, not code equivalence outside calls. And the Beta posterior treats these rewards as i.i.d., while the parent population's baseline rises over time, so the success probabilities are non-stationary. The paper's limitation section flags primitive coupling, but the confound is broader: even independent primitives get biased credit whenever the LLM edits unrelated code.\n\nThat said, this is not fatal. The method is well-specified, the ablation with vs. without Thompson sampling shows selection matters, and even biased credit can work as a heuristic. The empirical results stand as evidence that the whole system works, but they don't support the causal interpretation. The authors should either weaken the causal language or add a controlled analysis, e.g., fixing the non-EAP code during insertion trials or measuring the impact of LLM edit rate.\n\nOther, smaller soft spots: test columns in Tables 1–3 are point estimates with no error bars or significance tests, and Bipedal Walker's training std is huge (67.06 ± 72.63). The closest reuse-based baselines (G-LNS, EoH-S, EvoLattice, BEAM) are discussed but not benchmarked. No code artifact is linked in the preprint, which will matter for reproducibility. The citation pattern is standard, including the self-citation to MCTS-AHD, which is a legitimate baseline.\n\nThe paper deserves a serious referee. The core mechanism is new enough within AAD, the experimental documentation is unusually thorough, and the credit-assignment question makes it a good discussion paper. I'd bring it to reading group and recommend sending it to peer review.","headline":"PACE is a well-specified and well-documented AAD method; the Thompson sampling credit assignment is more confounded than the paper claims, but the empirical evidence is strong enough to deserve peer review.","tokens_in":23607,"tokens_out":3826,"would_cite":true,"duration_ms":33617,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PACE treats reusable code components as persistent units, so useful local logic survives discarded algorithms and component-level transfer beats whole-program LLM evolution under the same evaluation budget.","keywords":["LLM-based automated algorithm design","executable algorithmic primitives","code evolution","Thompson sampling","program search","genetic programming","heuristic design","credit assignment"],"falsifier":"Take a control task engineered so that two subroutines are useless alone and highly effective together, such as one subroutine that normalizes a sensor stream that only the other subroutine consumes, and run PACE under a fixed budget against whole-program evolution; if the Thompson posterior promotes individually weak components and the final algorithm is no better than whole-program search, the central persistence claim fails.","tokens_in":22542,"feed_emoji":"🧩","tokens_out":6737,"duration_ms":53164,"temperature":0.7,"pith_summary":"This paper argues that LLM-based automated algorithm design fails when it evolves algorithms as indivisible programs: useful local logic is discarded with its host, and the same snippets must be rediscovered. To fix this, the authors propose PACE, which extracts callable functions called Executable Algorithmic Primitives (EAPs), keeps them in a persistent library, and transfers them across algorithm generations. Four primitive-aware operators constrain how an LLM may insert, replace, refine, or recombine EAPs, and Thompson sampling decides which EAP to expose based on whether the child beat its parent. On continuous control and TSP tasks, PACE finds competitive or better algorithms than whole-program baselines under the same 1,000-evaluation budget, in some cases with large zero-shot gains when the evolved program is applied to larger problem instances. The sympathetic reader takes away that modular persistence, not just prompt engineering, can carry algorithmic search.","feed_headline":"Reusable code snippets outperform whole-program algorithm evolution","feed_subtitle":"The result: stronger controllers and TSP heuristics under the same 1,000-evaluation budget.","key_machinery":"The central object is the Executable Algorithmic Primitive (EAP), a callable function whose implementation stays fixed while it moves between host programs. The machinery that carries the argument has three parts: the persistent EAP set with its retention invariant, four primitive-aware variation operators whose call-set contracts are verified by an AST checker before evaluation, and a Thompson-sampling bandit over EAPs whose posterior is updated by the parent-relative binary reward. Together they turn a whole-program evolutionary loop into a two-level search: complete algorithms are still evaluated by the task evaluator, but the components that survive are chosen by accumulated transfer evidence rather than by the fate of their host program.","core_discovery":"PACE's central claim is that the unit of evolution in LLM-based algorithm design should be a persistent Executable Algorithmic Primitive, not the complete program. An EAP is a callable function with a fixed implementation and a stable identity; once admitted to the EAP set it is never removed ($e \\in E_t$ implies $e \\in E_{t+1}$), so it can be called by algorithms generated after its source program is gone. The paper operationalizes this with four structural contracts: insertion adds one focus EAP while preserving all exposed parent EAPs, replacement swaps the weakest EAP for a Thompson-sampled one, refinement rewires the host around a fixed EAP set, and crossover recombines the union of two parents' EAP sets. Each insertion or replacement trial is scored as a binary parent-relative improvement, which updates a Beta posterior for the EAP's transferability; this is what lets PACE credit individual components without any extra evaluation data. The reported results support the claim that this persistence-plus-selection mechanism discovers competitive algorithms on Racing Car, Bipedal Walker, TSP-Construct, and TSP-ACO.","pith_inferences":["If PACE's claim holds, the same two-level structure could transfer to other LLM program-search settings where local logic matters, such as code optimization, symbolic regression, or theorem-proving tactic selection, wherever a component can be given a stable function signature.","The independence assumption suggests a stress test: on a task engineered so that two primitives only pay off when called together, the parent-relative reward should misassign credit, and adding pairwise coupling terms to the reward would be the natural fix.","EAPs are named functions with textual descriptions, so the library itself is interpretable; one could inspect which EAPs accumulate high posterior means and treat them as discovered design patterns for the task, or seed them into a fresh search.","Parent-relative reward favors EAPs that improve a randomly selected parent, so an EAP with high mean gain on weak parents may be preferred over one with moderate gain on strong parents; a quality-normalized reward would be a testable variant."],"forward_implications":["Because EAPs persist by construction, useful code is no longer lost when a low-scoring program is removed, so the search budget is not spent rediscovering the same local logic.","The parent-relative reward turns each transfer into a free credit-assignment experiment, so EAP selection improves without any auxiliary validation set or extra evaluation calls.","The structural contracts, enforced by an AST verifier, mean that an LLM's failure to actually call the designated EAPs cannot corrupt the Thompson-sampling posterior.","Under a fixed budget of 1,000 evaluations, PACE matches or outperforms whole-program LLM-based automated algorithm design baselines on four tasks and generalizes zero-shot to larger TSP instances.","Because EAP discovery is decoupled from host generation, upgrading the LLM used only for EAP discovery (about 1.6% of tokens) yields disproportionate performance gains."],"supporting_citations":[{"why":"Defines programs as composed from reusable primitives, the conceptual starting point for EAPs.","marker":"Koza 1990"},{"why":"Introduces automatically defined functions, the precursor idea of evolved subprograms that PACE makes persistent and transferable.","marker":"Koza 1994"},{"why":"DreamCoder's library learning motivates persistent reusable components, though PACE differs by not requiring pass/fail tests.","marker":"Ellis et al. 2021"},{"why":"EoH is the whole-program evolution baseline that PACE extends and the main comparison for full-program search.","marker":"Liu et al. 2024"},{"why":"ReEvo, a reflective-evolution baseline, represents the alternative context-only search memory that PACE contrasts with structural persistence.","marker":"Ye et al. 2024"},{"why":"MCTS-AHD is a tree-search baseline and the source of the TSP-ACO experimental setup.","marker":"Zheng et al. 2025"},{"why":"Provides the Thompson sampling method used for EAP selection.","marker":"Daniel et al. 2018"},{"why":"MLES supplies the control-task benchmarks and cold-start protocol that PACE compares against.","marker":"Hu et al. 2025"}],"fun_headline_variants":["Persistent code primitives improve algorithm search","Evolution keeps useful snippets, not whole programs","PACE: evolve code bits, not whole algorithms","Reusable algorithmic parts beat whole-program evolution","Transferable code units enhance algorithm design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a primitive's value can be judged from isolated transfer trials—that a code component either helps or hurts on its own—whereas in strongly coupled algorithms a primitive may only pay off in the presence of a specific host.","fun_headline_variants_meta":{"raw":{"variants":["Persistent code primitives improve algorithm search","Evolution keeps useful snippets, not whole programs","PACE: evolve code bits, not whole algorithms","Reusable algorithmic parts beat whole-program evolution","Transferable code units enhance algorithm design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4269,"prompt_tokens":947,"completion_tokens":3322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3255}},"tokens_in":563,"tokens_out":3322,"duration_ms":23306,"temperature":1.0,"reasoning_tokens":3255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:26:51.469220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a control task engineered so that two subroutines are useless alone and highly effective together, such as one subroutine that normalizes a sensor stream that only the other subroutine consumes, and run PACE under a fixed budget against whole-program evolution; if the Thompson posterior promotes individually weak components and the final algorithm is no better than whole-program search, the central persistence claim fails.","supporting_citations":[],"review_version":2}