{"id":"9b8820d6-2e67-4419-8081-0e00bdc50e3a","arxiv_id":"2501.18784","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM-generated heuristics, compiled into greedy best-first search over Rust successor-generator tasks, achieve competitive coverage on numeric IPC benchmarks and handle non-PDDL tasks.","lead":"This paper uses large language models to write scoring functions that guide a search algorithm through planning problems described as Rust code. The approach beats standard planners on many numeric benchmarks and solves tasks, like a maze with ghosts and a twin-prime puzzle, that are hard to write in traditional planning languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on selecting the best of 18 model/fallback configurations on the same benchmark set; the manual PDDL-to-Rust translation is VAL-verified and is therefore less threatening than the reader's weakest assumption.","rationale":"The reader's verdict is CONDITIONAL, and this stress-test confirms that conditionality is warranted, so no verdict change is needed. The reader's stated weakest assumption was the manual PDDL-to-Rust translation, but that assumption is not the most load-bearing threat to the central claim. Because every PDDL-derived solution is verified with VAL against the original PDDL, a semantically incorrect Rust translation cannot manufacture valid solutions; it can only suppress them. The translation step is therefore a fairness and automation caveat, not a source of inflated coverage. The more serious threat is that the headline 'state-of-the-art' is obtained by selecting the best-performing LLM and fallback configuration after evaluating on the same benchmarks that define the comparison. This is a form of test-set overfitting: with 18 configurations and stochastic heuristic generation, the maximum observed coverage can substantially exceed the expected coverage of any fixed configuration. The paper's own treatment of the P(3h||3n) portfolio as an unfair post-hoc baseline makes the asymmetry explicit. The paper does provide partial safeguards: VAL verification, a representation-control experiment with h_md, and a Monte-Carlo variance analysis of heuristic ordering, all of which are real evidence. However, those safeguards do not address cross-model/cross-configuration selection. A held-out domain split, or a pre-registered configuration, would directly test whether the advantage persists when the configuration is chosen independently of the evaluation set. Until such a test is run, the conditional verdict should remain, with the additional condition that the SOTA claim be demonstrated under a selection-free evaluation protocol.","tokens_in":14308,"tokens_out":6206,"duration_ms":68459,"concrete_test":"Perform a domain-level holdout evaluation. Randomly partition the 18 numeric domains into a selection set (e.g., 14 domains) and a held-out test set (4 domains). On the selection set, choose the single best (model, fallback) configuration using the paper's own protocol. Then evaluate that exact configuration on the held-out domains under the same 10-minute/8GB limits, comparing against NFD and MFF. Repeat over at least 100 random partitions. If the mean held-out coverage of the selected configuration does not exceed the best individual planner's coverage, the state-of-the-art claim is an artifact of in-sample selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM-generated heuristics achieve state-of-the-art performance is supported by Table 1, which reports coverage for six LLMs under three fallback strategies (FC, US, SP-10), i.e., 18 configurations, on the same 18-domain/360-instance numeric benchmark set. The best SP-10 result (247 for GPT-4.1 or Claude) exceeds the best individual planner (MFF at 217, NFD at 202), but that winner is selected post hoc after observing performance on the evaluation set. The paper explicitly dismisses the P(3h||3n) portfolio as 'a post-hoc optimization and not a fair comparison,' yet the same reasoning applies to choosing the best model and fallback after seeing the same benchmarks. The variance analysis in Figure 4 only randomizes heuristic order within a single model/configuration; it does not account for selection across models and fallback policies. In contrast, the reader's weakest assumption about manual PDDL-to-Rust translation is largely neutralized by the verification protocol: for all PDDL-specified instances, solutions are checked with VAL against the original PDDL. An overly permissive Rust translation would yield plans that fail VAL, and an overly restrictive translation would only reduce coverage. Translation errors therefore bias against the method, not in its favor. The load-bearing threat is thus in-sample configuration selection, which can inflate the reported coverage of 'our approach' relative to the expected performance of any single pre-registered configuration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an approach to satisficing numeric planning in which a planning task is represented as an 'Explicit Successor Generator' in Rust (successor function, goal test, initial state), and an LLM is prompted to generate a Rust heuristic function that is compiled into greedy best-first search. The authors evaluate six LLMs under three fallback strategies (FirstCompilation, UntilSuccess, and SelfPortfolio-N) on the IPC-2023 numeric benchmarks, which are manually translated from PDDL to Rust, plus two non-PDDL domains (Twin Prime and Pacman). Table 1 reports that the best configuration shown there, GPT-4.1 with SelfPortfolio-10, solves 247 of the 360 PDDL instances, above the individual numeric planners NFD (202) and MFF (217), but below the P(3h||3n) portfolio (295), which the authors set aside as post-hoc. The paper also presents ablations of instance-specificity, model/fallback choice, and reasoning effort.","tokens_in":14465,"tokens_out":8136,"duration_ms":88967,"significance":"The idea of using LLMs to synthesize problem-specific heuristic functions is timely and potentially useful: it avoids repeated LLM calls during search, yields inspectable code, and the two non-PDDL domains illustrate representational flexibility. The evaluation has genuine strengths, including external IPC-2023 numeric benchmarks, plan validation with VAL against the original PDDL, multiple LLMs, fallback strategies, and a Monte-Carlo variance analysis of heuristic order. However, the headline state-of-the-art claim is not yet supported because the best model and fallback configuration are selected after observing performance on the same benchmark set, and because the winning configuration is effectively a portfolio of sampled heuristics. With a pre-registered configuration, a held-out evaluation, or an appropriately corrected comparison, the contribution would be significant.","major_comments":[{"comment":"The central claim that LLM-generated heuristics achieve state-of-the-art performance rests on selecting the best of many evaluated configurations after seeing the benchmark results. Table 1 reports three models under three fallback strategies, and the ablation section states that six models and SP-5 were also tested, so the best result (GPT-4.1 with SP-10, 247 on the 360 PDDL instances) is a post-hoc selection. Footnote 4 dismisses the P(3h||3n) portfolio as 'a post-hoc optimization and not a fair comparison,' but the same criticism applies to choosing the best model and fallback after observing performance on the same 360-instance set; for example, FirstCompilation with GPT-4.1 solves only 144 instances. Figure 4 randomizes heuristic order within a single model and fallback policy, so it does not account for selection across models and fallbacks. Please report a pre-registered configuration, a held-out domain split, or a multiplicity-corrected summary over all configurations.","section":"Table 1; Discussion"},{"comment":"The comparison in Table 1 is asymmetric: SelfPortfolio-10 allocates ten time slices and may generate and run up to ten distinct heuristics with restarts, whereas the baselines NFD and MFF are single-configuration planners. The paper's own non-portfolio LLM configurations, such as FirstCompilation with GPT-4.1 (144 on the 360 PDDL instances), are far below NFD (202) and MFF (217), so the reported advantage is largely attributable to the portfolio/retry mechanism rather than to an individual LLM-generated heuristic. Please add a fair single-heuristic comparison, or apply the same multi-run portfolio treatment to the baseline planners.","section":"Table 1; Empirical Evaluation"},{"comment":"The abstract and Discussion claim that the approach 'enables the solution of problems that cannot be adequately expressed in any existing PDDL dialect,' but the Twin Prime and Pacman domains are manually encoded in Rust by the authors, and the paper gives no formal expressiveness argument for why these domains cannot be encoded in PDDL or its extensions. The manual encoding step is also acknowledged in the Methodology as requiring human intervention. As a proof-of-concept the two domains are interesting, but the claim should be softened to say that the approach handles tasks for which the authors did not find a practical PDDL encoding, with the human modeling step stated as part of the method.","section":"Discussion; Expressiveness"}],"minor_comments":[{"comment":"The sentence 'the best setting of our method is to use a non-reasoning model to generate non-IS heuristics and run them as SelfPortfolio-10 (276 with GPT-4.1)' conflicts with Table 1 (247 on the 360 PDDL instances) and Table 2 (281 on the 400 total instances); please clarify which number is intended and why.","section":"Discussion"},{"comment":"The caption says results are from GPT-4.1, GPT-5.1, and Claude Sonnet 4.5, while the text states that six LLMs were tested; please clarify which models appear in the main table and where the full six-model results are reported.","section":"Table 1 caption"},{"comment":"The phrase 'a broad range of established planning benchmarks' overstates the scope of the PDDL evaluation, which is limited to the IPC-2023 numeric track; please qualify the claim as applying to numeric satisficing planning.","section":"Abstract"},{"comment":"The head-to-head plot in Figure 2 is computed only on instances that both planners solve, which can make the two approaches look more similar than they are across the full benchmark; please state this selection effect explicitly or show a plot that includes unsolved instances.","section":"Figure 2"},{"comment":"The statement that instance-specificity 'slightly decreased generation time' is supported only by a p-value (P=2e-5) without describing the statistical test, sample size, or multiple-comparison correction; please provide these details.","section":"Impact of Instance-Specificity"},{"comment":"The paper does not provide a link to the code, the Rust translations of the IPC domains, or the full set of generated heuristics; releasing these would substantially improve reproducibility of the SP-10 results.","section":"Reproducibility"},{"comment":"There are several typos and spacing errors, including 'it’s mini variant' (should be 'its'), 'Pacmanand' (missing space), and 'the qualitative measure ... can be seen in Figure 2' which should refer to 'comparisons'; a light copyedit would help.","section":"Typos"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is a useful proof-of-concept with a sound experimental backbone, but the headline state-of-the-art claim is overstated because the best configuration is selected on the test set and the winning variant is a portfolio. The manual PDDL-to-Rust translation is a lesser concern than the configuration-selection issue, since VAL verification and the direction of possible translation bias mitigate it. I would be willing to reconsider after a revision that reports a pre-registered configuration or a held-out/aggregate analysis, and that compares SP-10 to portfolios on equal terms. The absence of a code/data link also matters for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid empirical contribution. It extends the LLM-heuristic idea from classical PDDL (Corrêa et al.) to numeric IPC-2023 benchmarks and to two non-PDDL domains (Twin Prime, Pacman), using a successor-generator representation in Rust. The evaluation is careful: external benchmarks, VAL verification of all PDDL plans, multiple models, three fallback strategies, ablations, and a real attempt to measure variance. The manual PDDL-to-Rust translation looks like a threat, but the stress-test note is right: VAL-checking means translation errors bias against the method, not for it. That concern is largely neutralized.\n\nThe real soft spot is configuration selection. Table 1 reports the best SP-10 score (247) against individual planners, but that winner is chosen after seeing the same 360 instances. The variance analysis in Figure 4 randomizes only heuristic order within one fixed model and fallback; it does not account for selection across six models and three fallback policies. The paper criticizes the P(3h||3n) portfolio as post-hoc, yet the same logic applies to its own headline number. The authors are transparent about full results in the appendix, so it is not deceptive, but the abstract's \"state-of-the-art\" is stronger than what any single pre-registered configuration would deliver.\n\nMinor issues: no code release, no error bars on Table 1, and the appendix prompts were not in the text I reviewed. Reproducibility is limited.\n\nDespite that, the core phenomenon—LLM-generated heuristics can beat strong individual numeric planners on many domains—is plausible and supported. The paper deserves a serious referee. The revision should state clearly that the SOTA claim is for the best of N configurations, ideally with a correction for selection or a pre-registered configuration, and should release code and prompts. I would bring this to a reading group and would cite it.\n\nRecommendation: accept for peer review, with revisions focused on the selection issue and artifact release.","headline":"Useful empirical study of LLM-generated heuristics for numeric planning, but the state-of-the-art claim hinges on picking the best of 18 configurations on the same benchmark set.","tokens_in":15133,"tokens_out":2390,"would_cite":true,"duration_ms":28480,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated heuristics can match the best numeric planners and reach problems PDDL cannot express.","keywords":["automated planning","heuristic search","LLM-generated heuristics","successor generator","numeric planning","greedy best-first search","code generation","PDDL"],"falsifier":"Have independent researchers re-translate the eighteen IPC-2023 numeric domains from PDDL into Rust, run the same LLM-generation pipeline, and validate every returned plan with VAL; if coverage changes materially, the hand-written Rust encoding, not the heuristic synthesis, is responsible for the reported results.","tokens_in":13990,"feed_emoji":"🤖","tokens_out":9677,"duration_ms":87392,"temperature":0.7,"pith_summary":"Planners need heuristics to guide search, and building them by hand is the bottleneck that this paper removes using large language models. The task is described as a successor generator, a goal test, and an initial state in Rust; the LLM writes a heuristic function in the same language, which is then compiled into greedy best-first search. The paper reports that this automatic pipeline is competitive with, and in several domains better than, leading numeric planners on the IPC-2023 benchmark set, and that it solves two problems (Twin Prime and Deterministic Pacman) that cannot be adequately expressed in PDDL. The point of the work is that LLMs, which cannot yet plan reliably, can still write the search guidance that makes planning work.","feed_headline":"LLM heuristics match state-of-the-art numeric planners","feed_subtitle":"Greedy search guided by AI-written code solves problems that cannot be expressed in PDDL.","key_machinery":"The central object is an Explicit Successor Generator (ESG): a planning task expressed as a Rust successor generator, a goal-testing function, and a JSON file giving the initial state and goal. The LLM turns this ESG plus a prompt into a heuristic function in Rust, which is compiled and dropped into a standard greedy best-first search, so search remains sound and requires no further model queries. Because the heuristic is ordinary code it can contain arithmetic over numeric fluents, custom distance estimates, or subroutines, which is what lets the method handle infinite state spaces and non-PDDL dynamics. Fallback strategies (FirstCompilation, UntilSuccess, SelfPortfolio-N) cope with the fact that generated heuristics may fail to compile or may be weak; SelfPortfolio-N runs N candidate heuristics in fixed time slices and resamples on failure.","core_discovery":"The central discovery is that LLM-generated heuristics can replace domain-independent heuristic derivation: given only the executable code of a task's successor generator and goal test, a language model produces a heuristic function that guides greedy best-first search to solutions. The authors show this by encoding eighteen IPC-2023 numeric domains in Rust, prompting several GPT and Claude models for heuristics, and comparing coverage against ENHSP, Metric-FF, and Numeric-FD under identical time and memory limits. They find the best configurations match or exceed the strongest domain-independent planners, doing especially well on Sailing and Zenotravel, and that the approach solves Twin Prime and Deterministic Pacman, tasks they argue no PDDL dialect can express. Because the heuristic is generated once and then compiled, the search itself is sound and requires no further LLM calls.","pith_inferences":["If LLM code synthesis continues to improve, planning may move away from PDDL-derived heuristics toward reading executable state-transition functions directly; the paper's explicit dependence on hand-written Rust encodings is the main obstacle, and automating that translation is an obvious next step.","The ablation showing that instance-specific prompts hurt most domains suggests the current models use the instance information to overfit or mis-tune coefficients; a stronger reasoning model, or a prompt asking for parameterized heuristics, might reverse this and make each instance's heuristic more precise.","Combining an LLM heuristic with a domain-independent heuristic in a parallel search could combine strengths: the paper's own representation control shows the Rust encoding alone is not what drives the gains, pointing to heuristic complementarity rather than simple replacement."],"forward_implications":["Handcrafted domain knowledge is no longer required: a problem stated as code plus a goal test suffices to obtain a usable heuristic.","The representation extends planning to transition systems with unbounded numeric variables, conditional effects, and custom dynamics that PDDL dialects cannot express.","Heuristics can be generated once per domain and reused across instances, since instance-specific prompts did not help for the tested non-reasoning models and increased cost.","On the IPC-2023 numeric benchmarks the approach is competitive with, and in several domains superior to, the leading domain-independent numeric planners."],"supporting_citations":[{"why":"Showed LLMs can generate executable search components, the direct predecessor this work extends to heuristic search.","marker":"Katz et al. 2024"},{"why":"Supplies the h_md heuristic and the ENHSP novelty/portfolio baselines that the paper's coverage is compared against.","marker":"Chen and Thiébaux 2024"},{"why":"Provides the Metric-FF baseline, a leading numeric planner using interval-relaxed heuristics.","marker":"Hoffmann 2003"},{"why":"Provides the Numeric-FD baseline, the IPC-2023 numeric track winner using LM-cut style heuristics.","marker":"Aldinger and Nebel 2017"},{"why":"Defines the IPC-2023 numeric competition and its benchmark domains used in the evaluation.","marker":"Taitler et al. 2024"},{"why":"Provides VAL, the plan validator used to check solutions to PDDL-encoded benchmark problems.","marker":"Howey, Long, and Fox 2004"},{"why":"The closest related work, generating heuristics from PDDL, whose representational limits motivate the ESG approach.","marker":"Corrêa, Pereira, and Seipp 2025"},{"why":"Documents LLMs' unreliability at direct planning, motivating the component-generation strategy.","marker":"Valmeekam et al. 2022"}],"fun_headline_variants":["LLM heuristics match classic planners on numeric domains","AI-written heuristics solve planning tasks PDDL can't express","GPT heuristics match top numeric planners via code","From code to heuristics: LLMs guide greedy search","LLM heuristics enable planning beyond PDDL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manually written Rust encodings of the IPC-2023 numeric PDDL benchmarks faithfully capture the original PDDL semantics; if a translation subtly changes preconditions, numeric conditions, or effects, the coverage comparison is not against the intended benchmark.","fun_headline_variants_meta":{"raw":{"variants":["LLM heuristics match classic planners on numeric domains","AI-written heuristics solve planning tasks PDDL can't express","GPT heuristics match top numeric planners via code","From code to heuristics: LLMs guide greedy search","LLM heuristics enable planning beyond PDDL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001071,"raw_usage":{"total_tokens":4451,"prompt_tokens":877,"completion_tokens":3574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":3495}},"tokens_in":493,"tokens_out":3574,"duration_ms":25954,"temperature":1.0,"reasoning_tokens":3495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:28:37.153389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent researchers re-translate the eighteen IPC-2023 numeric domains from PDDL into Rust, run the same LLM-generation pipeline, and validate every returned plan with VAL; if coverage changes materially, the hand-written Rust encoding, not the heuristic synthesis, is responsible for the reported results.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Showed LLMs can generate executable search components, the direct predecessor this work extends to heuristic search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Metric-FF baseline, a leading numeric planner using interval-relaxed heuristics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Numeric-FD baseline, the IPC-2023 numeric track winner using LM-cut style heuristics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the IPC-2023 numeric competition and its benchmark domains used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides VAL, the plan validator used to check solutions to PDDL-encoded benchmark problems."}],"review_version":1}