Pith. sign in

REVIEW 5 major objections 4 minor 41 references

On the Limit of Language Models as Planning Formalizers

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Language models solve more planning problems by writing PDDL than by planning

desk verdict Useful full-PDDL benchmark, but the formalizer-vs-planner comparison runs against a weak zero-shot baseline, so the headline advantage is overstated. read the letter →

arxiv 2412.09879 v4 pith:7MWMIUYA submitted 2024-12-13 cs.CL

Cassie Huang , Li Zhang This is my paper · ORCID
classification cs.CL
keywords LLM-as-formalizerPDDLgenerationclassicalplanningnaturallanguagedescriptionsBlocksWorldMysteryLLM-as-plannerneuro-symbolic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a language model is better used as a formalizer than as a planner. As a formalizer, the model translates a natural-language description of a planning environment into a complete PDDL domain file and problem file, which a deterministic planner then solves. The authors benchmark this on four IPC planning domains with descriptions at three levels of naturalness. They find that models large enough to write PDDL often get more correct plans this way than by generating plans directly: gpt-4o as formalizer solves 60 of 100 blocks problems versus 33 of 100 as planner. Performance drops as descriptions become more natural and omit implied facts, and error analysis traces failures to syntax errors in weaker models and semantic errors in the domain or problem file in stronger ones.

What carries the argument

The central machinery is the PDDL pair (domain file plus problem file) written by the LLM and solved by an external planner. The domain file defines predicates, action preconditions, and effects; the problem file defines objects, initial state, and goal. Solvability is tested with the dual-bfws-ffparser planner and correctness with VAL against ground-truth files. To probe how descriptions matter, the paper builds three naturalness levels—Heavily Templated, Moderately Templated, and Natural—and adds Mystery BlocksWorld, a renamed version of BlocksWorld used as a wug test to reveal reliance on memorized names.

What would settle it

Run the same 100 BlocksWorld problems with a planner loop that lets gpt-4o propose a plan, checks it with VAL, returns the error message, and lets the model revise for a few rounds; if that loop reaches or exceeds the formalizer's 60/100 correctness, the paper's claim that formalization is the better methodology would be undercut.

Watch

Extended reading notes

Core claim

The central claim is that LLM-as-formalizer—generating the complete PDDL domain file and problem file from a natural-language description, then handing those files to a deterministic planner—is a viable and often better methodology than asking the LLM to produce the plan itself. The evidence: on BlocksWorld-100, gpt-4o as formalizer attains 60/100 correct plans on Natural descriptions and 89/100 on Heavily Templated ones, versus 33/100 and 29/100 as planner; formalization also handles the renamed predicates and actions of Mystery BlocksWorld, where direct planning finds nothing. But the advantage is not universal: o3-mini and DeepSeek-R1 are as good or better planners on some settings, and on the harder Barman domain nearly everything fails. The paper also claims naturalness is the main limit: human-like descriptions that omit implied preconditions such as 'clear' reduce solvability and correctness across models.

Load-bearing premise

The headline comparison assumes that the LLM-as-planner baseline is a single zero-shot pass with no checking or retries; if a planner baseline is allowed to validate its plans and revise, the formalizer's advantage may shrink or vanish.

Editorial extensions

If this is right

  • Formalizer gains are largest on simple, fully observed domains, which is exactly where direct planners fail most; users should prefer generating a complete model over generating a plan in such settings.
  • Templated descriptions overestimate ability, so evaluations that use only templated inputs will look better than real deployments where users describe environments loosely.
  • Models that cannot write low-resource formal languages, such as Llama up to 405B, cannot use this methodology at all, pointing to syntax-level code generation as a bottleneck.
  • Semantic errors persist even in strong models, especially in the problem file's initial and goal states, so future work should target semantic verification rather than only syntactic correctness.
  • The formalizer advantage is not universal: reasoning-oriented models such as o3-mini and DeepSeek-R1 can plan as well or better, so the choice between formalizer and planner should be made per model, per domain, and per description naturalness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the planner baseline is strengthened with a feedback loop that validates candidate plans and lets the model retry, the formalizer's advantage probably shrinks; the paper itself names this as a likely stronger baseline.
  • A commonsense-recovery pre-pass that makes implied facts such as 'clear' explicit before PDDL generation could close much of the natural-description gap documented in the error analysis.
  • Because full PDDL generation turns the initial state into a prediction target, previous partial-PDDL results likely overstated readiness; deployment decisions should concentrate on problem-file errors.
  • A natural extension is to test the same formalizer-versus-planner comparison on partially observable or longer-horizon environments, where the deterministic solver's guarantees may interact differently with description naturalness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper evaluates whether LLMs can be used as 'formalizers' that translate natural-language descriptions of planning environments into complete PDDL domain and problem files, which are then solved by a deterministic planner. The authors construct four IPC-style benchmarks (BlocksWorld, Mystery BlocksWorld, Logistics, Barman) with three levels of description naturalness (Heavily Templated, Moderately Templated, Natural), and compare LLM-as-formalizer against a zero-shot LLM-as-planner baseline across several open and closed models. The main claims are that (i) sufficiently large closed models can generate solvable and correct PDDL, (ii) formalizing often outperforms direct plan generation, (iii) more natural descriptions decrease performance, and (iv) the formalizer approach is robust to lexical perturbations. The paper also provides a manual error analysis of syntax and semantic errors. The empirical results are grounded in VAL-validated plans and the code/data are released, but the headline comparison rests on a deliberately weak planner baseline and on single-run measurements without confidence intervals.

Significance. If the results hold, the paper is a useful systematic contribution to the emergent LLM-as-formalizer paradigm. Its strengths are that it moves beyond partial PDDL generation to whole domain and problem files, introduces and validates a naturalness continuum for descriptions, uses deterministic VAL-based correctness checking rather than only string/plan matching, and includes a detailed error taxonomy with hand-annotated examples. The finding that natural descriptions degrade formalization quality, and that models exhibit different error profiles, is actionable for future work. The public release of code and data supports reproducibility. However, the comparative superiority claim is not yet fully established because the planner baseline is minimal and the error analysis is based on small samples; the abstract's generalization to 'most large enough models' overstates what the data show.

major comments (5)
  1. [Section 3 and Section 8, Tables 2–5] The central comparative claim—that LLM-as-formalizer often outperforms LLM-as-planner—is measured against a zero-shot, single-attempt planner baseline that receives no validation or correction loop. The formalizer pipeline, in contrast, benefits from a deterministic planner and, in principle, the ability to detect unsolvable PDDL. The paper's own Limitation section concedes that a Kambhampati et al. (2024) style LLM-Modulo baseline, which adds symbolic validation to the planner, would 'likely lead to a stronger baseline.' This is load-bearing: the observed advantage could reflect the presence of an external solver/validator rather than the value of formalization as such. The authors should either add such a validated-planner control (e.g., allowing the planner to propose multiple candidates and checking them with VAL) or substantially soften the 'formalizing is often superior' conclusion to 'formalizing is superior to unverified zero-shot plan generation.'
  2. [Section 5 and Tables 2–5] All results are reported from a single run per model/dataset combination, with no confidence intervals, seeds, or repeated sampling. This makes small differences unreliable; for example, on Heavily Templated BlocksWorld-100, o3-mini as planner achieves 96/100 while its formalizer correctness is 94/100, and on Barman-100, several planner scores exceed formalizer scores (e.g., DeepSeek-R1 planner 22 vs. formalizer 2). The paper's conclusion that formalizing is 'often' superior is based on point estimates that could change with reruns. At minimum, the authors should state that measurements are single-run and provide, for a few representative conditions, repeated-run estimates or bootstrap intervals to quantify variance.
  3. [Abstract and Section 5.1] The abstract states that 'most large enough models can effectively formalize descriptions as PDDL, outperforming those directly generating plans.' The data in Tables 2–4 show that this holds only for a few closed models (gpt-4o, o3-mini, DeepSeek-R1, and partially gpt-4o-mini). Llama-3.1-405B and all DeepSeek-R1-Distill variants achieve zero solvability on BlocksWorld and Logistics, and Gemma models are highly variable (near-perfect on Mystery BlocksWorld but poor on BlocksWorld-100). The generalization 'most large enough models' is not supported. The authors should restrict the claim to the specific models that performed well, or rephrase to 'some large closed models.'
  4. [Section 4.1 and footnote 2] The paper repeatedly describes the task as generating 'the complete representation' or 'entire PDDL,' but footnote 2 states that 'it is minimally necessary to provide the action space, the identifiers and parameters of the actions in DF.' The model therefore receives the action schemas (names and parameter lists) and only needs to fill in preconditions/effects and the problem file. This is a meaningful reduction of the formalization burden and should be stated clearly in the abstract and main text, not only in a footnote. The claim of 'complete representation' is accurate with respect to the full DF/PF content, but readers may infer that the model invents action names and parameters from scratch, which is not the case.
  5. [Section 4.1 (Natural dataset generation)] The Natural descriptions were generated with gpt-4o assistance and then gpt-4o is later evaluated on those descriptions. Although the authors manually verified correctness and naturalness, this creates a potential confound: gpt-4o may perform better on its own paraphrases than on independently written natural text, which could inflate the gpt-4o results and the naturalness-degradation finding. The authors should add a control using human-authored natural descriptions (or descriptions generated by a different model) for at least one dataset, or explicitly discuss why this confound is unlikely to affect the main conclusions.
minor comments (4)
  1. [Appendix B] The paper claims zero-shot prompting, but the planner prompts in Listings 13–16 include a concrete example plan in the instruction ('Here is what the output should look like: (PICK-UP A)...'). This is a form of one-shot formatting example. The authors should clarify whether the formalizer prompts also contain such examples, and discuss whether the example output influenced behaviors such as the DeepSeek-R1 pattern matching noted in Section 5.6.
  2. [Section 5.5 and Tables 8–9] The error analysis is based on a random sample of only 20 errors per model/dataset, which is small for drawing conclusions about common error types. The counts are presented as deterministic values without noting sampling uncertainty; a sentence acknowledging the small sample size and the lack of inter-annotator agreement (if only one annotator) would be appropriate.
  3. [Figure 3 and Figure 7] Barman-100 results are relegated to the appendix (Figure 7) with only a sentence in Section 5.1; since Barman is the most complex domain and shows a reverse trend (planner outperforming formalizer for several models), it deserves at least a brief discussion in the main text rather than a dismissal as 'close-to-zero performance for all models.'
  4. [Throughout] There are a few typographical and formatting issues: 'Mystery BlocksWorld' is sometimes written as 'MysteryBlocksWorld' (e.g., Tables 3, 8); the text in Figure 1 is not fully legible; and Table 1's 'This work' row could be more explicit that the action-space prompt is given. These do not affect the scientific content.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the formalizer-versus-planner comparison is an empirical benchmark study whose claims are validated against ground-truth PDDL, not derived from its own inputs.

full rationale

This paper is an empirical evaluation, not a derivation, so the circularity patterns of fitted equations or definitional identities do not apply. The central claims—that LLM-as-formalizer often outperforms LLM-as-planner, that natural descriptions are harder, and that formalization is robust to lexical perturbation—are supported by measured correctness scores computed with VAL against ground-truth PDDL domain and problem files, using an independent deterministic planner for solvability. The LLM-as-formalizer output is not compared to its own input text; it is validated against externally supplied ground-truth PDDL, and the LLM-as-planner baseline is validated with the same ground-truth files. The natural-sounding descriptions were generated with gpt-4o assistance and then manually verified for correctness, and the same descriptions were given to both the formalizer and planner baselines; this creates a possible data-generation concern but not a circular reduction, because the evaluated quantity is plan correctness in a grounded environment, not agreement with the generated descriptions. The paper cites prior work by one of its authors (e.g., Lyu et al. 2023, Zhang et al. 2024a-c), but those citations are used for positioning and related-work context, not as the evidence for the paper's headline results. The acknowledged limitation that a validated LLM-Modulo planner 'will likely lead to a stronger baseline' is a legitimate concern about baseline strength and experimental fairness, but it is not a circular argument: the comparison is between two well-defined pipelines, and the absence of a stronger baseline does not make either pipeline's output equivalent to its input by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The paper is therefore self-contained against external benchmarks, and the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

This is an empirical benchmark, so the ledger records experimental design assumptions rather than fitted constants. The main non-verifiable inputs are the hand-verified natural descriptions, planner completeness, and VAL correctness.

free parameters (2)
  • Error analysis sample size = 20 instances per model-dataset
    Chosen by hand in Section 5.5; error rates in Tables 8-9 are estimates on 20 random failures, so fine-grained error proportions have wide uncertainty.
  • Instance complexity ranges = BlocksWorld 2-15 blocks; Barman 1-9 shot glasses, ingredients, cocktails; 100 problems per dataset
    Chosen to vary difficulty (Appendix A); the uniform distribution is a design decision that shapes the difficulty mix and could affect aggregate scores.
assumptions (5)
  • domain assumption Ground-truth PDDL domain and problem files correctly model the four environments.
    Correctness of predicted plans is determined by executing them against these files with VAL; any error in ground truth would propagate into all scores. Sources: standard IPC domains, Valmeekam et al. (2024a), and PDDL generators.
  • domain assumption The dual-bfws planner returns a plan whenever one exists within its limits.
    Solvability counts a problem as solvable only if this planner returns a plan; completeness of the planner is assumed, not proven.
  • domain assumption VAL correctly determines plan executability and goal satisfaction.
    Correctness is defined as VAL reporting no errors against ground-truth files, so VAL's correctness is load-bearing.
  • domain assumption The Natural descriptions are correct, complete, and representative of human descriptions.
    Natural DDs and PDs were generated by gpt-4o and manually verified by the authors; unverified omissions or ambiguities would bias the naturalness comparison. Section 4.1.
  • domain assumption A single run of each stochastic LLM adequately represents its performance.
    No repeated sampling or confidence intervals are reported; API models with temperature settings can vary across runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Limit of Language Models as Planning Formalizers." pith.science (2026). https://pith.science/paper/7MWMIUYA

@misc{pith2026241209879,
  author       = {Pith},
  title        = {Pith review of: On the Limit of Language Models as Planning Formalizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MWMIUYA}},
  note         = {Machine review of arXiv:2412.09879}
}
read the original abstract

Large Language Models have been found to create plans that are neither executable nor verifiable in grounded environments. An emerging line of work demonstrates success in using the LLM as a formalizer to generate a formal representation of the planning domain in some language, such as Planning Domain Definition Language (PDDL). This formal representation can be deterministically solved to find a plan. We systematically evaluate this methodology while bridging some major gaps. While previous work only generates a partial PDDL representation, given templated, and therefore unrealistic environment descriptions, we generate the complete representation given descriptions of various naturalness levels. Among an array of observations critical to improve LLMs' formal planning abilities, we note that most large enough models can effectively formalize descriptions as PDDL, outperforming those directly generating plans, while being robust to lexical perturbation. As the descriptions become more natural-sounding, we observe a decrease in performance and provide detailed error analysis.

Figures

Figures reproduced from arXiv: 2412.09879 by the authors.

Figure 1
Figure 1. LLM-as-formalizer uses natural language descriptions to generate the Domain and Problem File in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Methodologies for using LLMs in planning. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance of both usages of LLMs on Heavily Templated [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of LLM-as-planner (as-P) and LLM-as-formalizer (as-F) across different naturalness level of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Number of syntax errors (solver cannot parse [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Number of fine-grained DF errors in 40 man￾ually annotated, randomly sampled errors in Natural BlocksWorld-100 and Logistics-100. See full re￾sults in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Performance for Barman-100. D Detailed Results Beyond the visualizations above, we show the de￾tailed results of all models on all simulations of all naturalness levels. D.1 Results for BlocksWorld-100 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 11 canonical work pages

  1. [1]

    Jean Berko. 1958. The child's learning of english morphology. Word, 14(2-3):150--177

    work page 1958

  2. [2]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227

    arXiv 2022

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  4. [4]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  5. [5]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764--10799. PMLR

    2023

  6. [6]

    Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, 36:79081--79094

    work page 2023

  7. [7]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  8. [8]

    Howey, D

    R. Howey, D. Long, and M. Fox. 2004. https://doi.org/10.1109/ICTAI.2004.120 Val: automatic plan validation, continuous effects and mixed initiative planning using pddl . In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294--301

    doi:10.1109/ictai.2004.120 2004

Show all 41 references
  1. [9]

    IPC. 1998. International planning competition. https://www.icaps-conference.org/competitions

    1998

  2. [10]

    IPC. 2000. International planning competition. https://www.icaps-conference.org/competitions

    2000

  3. [11]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515

    2024 arXiv

  4. [12]

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. 2024. Llms can't plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817

    2024 arXiv

  5. [13]

    Yash Kumar Lal, Li Zhang, Faeze Brahman, Bodhisattwa Prasad Majumder, Peter Clark, and Niket Tandon. 2024. https://doi.org/10.18653/v1/2024.findings-acl.921 Tailoring with targeted precision: Edit-based agents for open-domain procedure customization . In Findings of the Associ...

    2024 doi

  6. [14]

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. 2024. Embodied agent interface: Benchmarking llms for embodied decision making. arXiv preprint arXiv:2410.07166

    2024 arXiv

  7. [15]

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023 a . Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477

    2023 arXiv

  8. [16]

    Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023 b . Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439

    2023 arXiv

  9. [17]

    Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.20 Faithful chain-of-thought reasoning . In Proceedings of the 13th International Joint Conference on Natura...

    2023 doi

  10. [18]

    Qing Lyu, Li Zhang, and Chris Callison-Burch. 2021. https://doi.org/10.18653/v1/2021.inlg-1.19 Goal-oriented script construction . In Proceedings of the 14th International Conference on Natural Language Generation, pages 184--200, Aberdeen, Scotland, UK. Association for Comput...

    2021 doi

  11. [19]

    Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.90 Language models of code are few-shot commonsense learners . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages...

    2022 doi

  12. [20]

    Christian Muise. 2016. Planning.Domains . In The 26th International Conference on Automated Planning and Scheduling - Demonstrations

    2016

  13. [21]

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.248 Logic- LM : Empowering large language models with symbolic solvers for faithful logical reasoning . In Findings of the Association for Computational Linguistics...

    2023 doi

  14. [22]

    Jendrik Seipp, \'A lvaro Torralba, and J \"o rg Hoffmann. 2022. PDDL generators. https://doi.org/10.5281/zenodo.6382173

    2022 doi

  15. [23]

    Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. 2024. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20256--20264

    2024

  16. [24]

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. Chain of thoughtlessness: An analysis of cot in planning. arXiv preprint arXiv:2405.04776

    2024 arXiv

  17. [25]

    Hao Tang, Darren Key, and Kevin Ellis. 2024. https://arxiv.org/abs/2402.12275 Worldcoder, a model-based llm agent: Building world models by writing code and interacting with the environment . Preprint, arXiv:2402.12275

    2024 arXiv

  18. [26]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  19. [27]

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2024. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36

    2024

  20. [28]

    Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Subbarao Kambhampati. 2023. https://arxiv.org/abs/2302.06706 On the planning abilities of large language models (a critical investigation with a proposed benchmark) . Preprint, arXiv:2302.06706

    2023 arXiv

  21. [29]

    Daniel S Weld. 1999. Recent advances in ai planning. AI magazine, 20(2):93--93

    1999

  22. [30]

    Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S Siegel, Jiahai Feng, Noa Korneev, Joshua B Tenenbaum, and Jacob Andreas. 2023. Learning adaptive planning representations with natural language guidance. arXiv preprint arXiv:2312.08566

    2023 arXiv

  23. [31]

    Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128

    2023 arXiv

  24. [32]

    Hongming Zhang, Muhao Chen, Haoyu Wang, Yangqiu Song, and Dan Roth. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.119 Analogous process structure induction for sub-event sequence prediction . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pr...

    2020 doi

  25. [33]

    Li Zhang, Peter Jansen, Tianyi Zhang, Peter Clark, Chris Callison-Burch, and Niket Tandon. 2024 a . https://doi.org/10.18653/v1/2024.starsem-1.17 PDDLEGO : Iterative planning in textual environments . In Proceedings of the 13th Joint Conference on Lexical and Computational Sem...

    2024 doi

  26. [34]

    Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, and Niket Tandon. 2024 b . https://aclanthology.org/2024.eacl-long.10 O pen PI 2.0: An improved dataset for entity tracking in texts . In Proceedings of the 18th Conference of the European Chapter of the Association f...

    2024

  27. [35]

    Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison-Burch. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.31 Causal reasoning of entities and events in procedural texts . In Findings of the Association for Computational Linguistics: EA...

    2023 doi

  28. [36]

    Tianyi Zhang, Li Zhang, Zhaoyi Hou, Ziyu Wang, Yuling Gu, Peter Clark, Chris Callison-Burch, and Niket Tandon. 2024 c . https://aclanthology.org/2024.nlrse-1.2 PROC 2 PDDL : Open-domain planning representations from texts . In Proceedings of the 2nd Workshop on Natural Languag...

    2024

  29. [37]

    Andrew Zhu, Liam Dugan, Alyssa Hwang, and Chris Callison-Burch. 2023. https://doi.org/10.18653/v1/2023.nlposs-1.8 Kani: A lightweight and highly hackable framework for building language model applications . In Proceedings of the 3rd Workshop for Natural Language Processing Ope...

    2023 doi

  30. [38]

    Wang Zhu, Ishika Singh, Robin Jia, and Jesse Thomason. 2024. Language models can infer action semantics for classical planners from environment feedback. arXiv preprint arXiv:2406.02791

    2024 arXiv

  31. [39]

    Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L Littman, and Stephen H Bach. 2024. Planetarium: A rigorous benchmark for translating text to structured planning languages. arXiv preprint arXiv:2407.03321

    2024

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.