Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Even the best LLM planners solve only 63.4% of benchmark tasks, while Fast Downward solves 97.85%.

desk verdict A useful, modest benchmark snapshot whose headline executability numbers rest on an undocumented validation pipeline; the paper deserves review but needs a described validator, run counts, and data release. read the letter →

arxiv 2507.23589 v1 pith:IUGFP2LJ submitted 2025-07-31 cs.RO cs.AI

Kai Goebel , Patrik Zips This is my paper
classification cs.ROcs.AI
keywords LLMplanningPDDLexecutionfidelityFastDownwardrobottaskreasoningmodelshybridbenchmarkstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to settle a practical question: can today's LLM reasoning models replace classical planners for robot task planning when both are given the same Planning Domain Definition Language (PDDL) domain and problem files? Across five benchmark domains (barman, blocks, elevator, satellite, tidybot), it compares nine LLMs against the Fast Downward planner, measuring not just whether a plan reaches the goal but how many of its steps are actually executable. The central finding is a large gap in reliability: the best LLMs (GPT-o1 and Claude Sonnet 3.7 Thinking) succeed on 63.4% of problems, versus 97.85% for Fast Downward, and execution fidelity can be as low as 13.9%. The paper argues this gap shows that pure LLM planning is not yet safe for real-world robotics, and points toward hybrid systems that combine LLM flexibility with symbolic verification.

What carries the argument

The load-bearing object is the execution-fidelity metric, defined as the ratio of executed actions to planned actions: a plan's steps are counted as executed only if they can be applied according to the domain's preconditions, so a plan that looks goal-directed but violates constraints mid-way is penalized. The benchmark harness itself is the second component: nine LLMs are prompted with the raw PDDL domain and problem files and asked to return a JSON plan with a reasoning section, a per-action justification, and a confirm_reasoning statement, and all are compared against Fast Downward's seq-sat-lama-2011 configuration as the classical baseline. Execution fidelity is what carries the paper's conclusion, because the success-rate gap alone could be blamed on goal selection; the fidelity numbers show that even within single plans, models lose track of state.

What would settle it

Re-run the same PDDL prompts on the five domains and replay every generated plan through a public PDDL plan validator, comparing the executed-action counts to Table 3; if the counts cannot be reproduced, the central fidelity gap is an artifact of the paper's unstated validation procedure.

Watch

Extended reading notes

Core claim

The paper's central claim is that current LLM reasoning models are not reliable planners even when prompted directly with the formal PDDL representation of a task. The best models, GPT-o1 and Claude Sonnet 3.7 Thinking, each solve 63.4% of the 93 benchmark problems, while Fast Downward solves 97.85%; and a successful-looking plan is not the same as an executable one. GPT-o1 achieves the highest execution fidelity at 73.4%, meaning about a quarter of its planned actions cannot be validly performed, while Llama 405B Instruct executes only 13.9% of its actions before violating domain constraints. Although reasoning models beat their non-reasoning counterparts, the paper concludes that LLMs still struggle exactly where robotics matters: precise resource management, consistent state tracking, and strict constraint compliance over long horizons.

Load-bearing premise

The headline numbers rest on an unstated plan-validation mechanism: the paper never specifies which parser, validator, or simulator decides that an action is executable, so if that validator accepts invalid actions or mislabels malformed JSON, every success rate and fidelity figure changes.

Editorial extensions

If this is right

  • If the benchmark generalizes, LLM planning should not be trusted as a standalone backend for robots: the top models leave 36.6% of tasks unsolved and a substantial fraction of their steps unexecutable.
  • The boost of reasoning models over their base versions (Claude Sonnet 3.7 Thinking 63.4% vs 58.1%; Gemini 2 Flash Thinking 35.5% vs 23.7%) is a real but insufficient improvement, so scaling alone is not the evident fix.
  • The high LLM scores in blocks and elevator likely reflect pretraining exposure, which means past success on familiar domains cannot be used as evidence of planning competence.
  • The practical path suggested by the results is hybrid: use LLMs for flexibility or heuristics, but let a symbolic planner or validator guarantee executability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates a single direct-prompt setup; it does not test iterative self-correction or validator feedback loops, so the measured gap is a lower bound on what prompt engineering could achieve, not necessarily the ceiling for LLM planning.
  • Because the plan-validation procedure is not specified, the reproduced numbers depend on choices the reader cannot audit; a natural extension is to publish the validator and raw plan transcripts alongside the benchmark.
  • The five domains are all classical IPC benchmarks; a testable extension would add partially observable or continuous-state robot domains, where the LLM's commonsense knowledge might plausibly help more and the fidelity gap could be smaller.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a benchmark comparison of nine large language models (Claude, Gemini, Llama, GPT variants) against the classical planner Fast Downward on five PDDL IPC domains (barman, blocks, elevator, satellite, tidybot). The LLMs are prompted with PDDL domain and problem files and asked to output JSON plans; the evaluation reports success rate, plan length, executed actions, execution fidelity, and planning time. The central finding is that even the best reasoning model (GPT-o1 or Claude Sonnet 3.7 Thinking, both 63.4% success) is far behind Fast Downward (97.85% success), and that successful-looking LLM plans often violate domain constraints during execution, with execution fidelity as low as 13.9% for Llama 405B Instruct. The authors conclude that LLMs cannot yet replace classical planners and recommend hybrid approaches.

Significance. The paper addresses a timely and practically important question for robotic task planning. Its main contribution is a systematic, current snapshot of nine state-of-the-art LLMs, including reasoning-optimized models, with an execution-fidelity metric that goes beyond simple success rates; the inclusion of Fast Downward as an external baseline is a strength, and Table 3's data are internally consistent (Ac ≤ PL, and FD has Ac = PL). If the results are reproducible, they usefully quantify the gap between generative and symbolic planning and motivate hybrid systems. The main limitations are the unspecified plan-validation pipeline, which makes all executability numbers unverifiable, and the absence of repeated runs and statistical analysis, which weakens the model-ranking conclusions. The qualitative takeaway is nonetheless consistent with prior work by Valmeekam, Kambhampati, and others.

major comments (3)
  1. [§3.5, Table 3, Figure 1] The executability metrics (Ac, Execution Fidelity) are defined only at the level of 'how many actions from a generated plan can be validly executed according to the domain's constraints,' but the manuscript never describes the validation pipeline: there is no specification of the JSON parser, the grounding of action names and parameters to PDDL schemas, the state-transition simulator, the treatment of malformed JSON, unknown objects, invalid parameter types, or actions with unsatisfied preconditions, nor the rule for scoring partial execution. It also does not state how the goal check is performed for SR, nor how instances with no generated plan (e.g., GPT-o1's 'no solution' responses) enter the PL, Ac, and fidelity averages. Because every headline number in Table 3 and Figure 1 depends on this pipeline, the central claim that LLM plans are not robustly executable (e.g., 13.9% fidelity for Llama 405B Instruct) cannot be independently audited. The authors must specify the validator in detail (ideally, release the code and the raw model outputs) and state how unsolved and empty-plan cases are handled.
  2. [§4.1, Table 3] The paper reports single-point estimates for SR, PL, and Ac without any repeated runs, error bars, confidence intervals, or significance tests. LLM API outputs are stochastic; the paper does not state the sampling temperature or number of runs per problem. As a result, differences that drive the model ranking (e.g., Claude Sonnet 3.5 SR 41.94% vs. Claude Sonnet 3.7 SR 58.06% vs. GPT-o1 SR 63.44%, or the 10% gap between Gemini 2 Flash and Gemini 2 Flash Thinking) may be within run-to-run noise. The qualitative conclusion that all LLMs are far behind Fast Downward is robust to this issue, but the comparative claims about reasoning models in Section 4.1 are not. At minimum, the authors should report the number of runs and variance, and preferably test whether the observed differences are statistically significant given the small per-domain problem counts (14-20 instances).
  3. [§3.1, Table 3] The Fast Downward baseline is not described at the level needed for reproduction. While the planner configuration is given (seq-sat-lama-2011, 600 s), the manuscript does not specify the exact benchmark problem files used (source, version, any preprocessing), nor how successful solution status is verified for FD, nor how plan lengths are averaged when FD fails on some instances (e.g., tidybot SR 89.5%, but PL 39.5 is reported). Because FD's PL and Ac columns are identical by definition, this does not affect the FD-vs-LLM fidelity gap, but it is necessary for a benchmark study to be reproducible.
minor comments (5)
  1. [§4.2] The text states that execution fidelity is 'followed by Llama DeepSeek R1 at 55.9% and Claude Sonnet 3.5 at 51.1%', but Table 3 yields Ac/PL for Claude Sonnet 3.5 = 11.34/16.69 ≈ 67.9%; the 51.1% value belongs to Claude Sonnet 3.7 Thinking (19.92/38.98). Figure 1's caption is correct, so the text should be corrected for consistency.
  2. [§2.1.1] The related-work text contains several citation errors: 'SayCan Hazra et al. [2024]' should cite Ahn et al. [2022] for SayCan and Hazra et al. [2024] for SayCanPay; 'Hee et al. Song et al. [2023]' appears to be a typo for Song et al. [2023]. Please correct these and check the remaining citations for accuracy.
  3. [§3.4] The prompt description gives the JSON output template but not the full prompt text, system prompt, or instructions on how to handle 'confirm_reasoning'. To make the experiment reproducible, the complete prompt (including any few-shot examples) should be included in an appendix or supplementary material.
  4. [Figure 1 caption] There is a typo in the caption ('whileGemini 2 Flash') and the caption repeats most of Section 4.2 in prose; please shorten the caption and let the figure show the data.
  5. [§3.3] The domain descriptions do not specify which exact IPC problem instances were used (e.g., which files from IPC 2000/2004/2011/2014). Please list the problem sources or release the problem files to allow exact replication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark results are measured against an external classical planner, with no fitted parameter, no equation reducing a prediction to an input, and no load-bearing self-citation chain.

full rationale

This paper is a purely empirical benchmark study. It prompts nine LLMs with PDDL domain and problem files, compares their outputs against Fast Downward, and reports measured metrics: Success Rate, Plan Length, Executed Actions, Execution Fidelity, and Planning Time. There is no derivation in which a target quantity is defined in terms of the result it is supposed to predict. In particular, Execution Fidelity is explicitly defined in Section 3.5 as the ratio of executed actions to planned actions, and the numbers in Table 3 are measurements of that ratio, not fits. The headline claim that LLMs do not achieve robust planning is supported by direct comparison with an external baseline (Fast Downward), not by any assumption supplied by the authors. The paper contains no equations that reduce a conclusion to an input, and no cited result by the same authors is invoked to forbid alternative interpretations; indeed, Goebel and Zips do not appear to cite their own prior work at all. The reviewer's concern that the plan-validation mechanism is unspecified is a legitimate reproducibility and auditability issue, since the paper does not describe the parser, validator, or simulator used to compute Executed Actions. However, an unspecified validation pipeline is not itself a circular step: it is missing detail about measurement, not a definitional equivalence between input and output. For these reasons, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new entities or fitted parameters. The paper is purely an empirical comparison; its conclusions depend on benchmark representativeness, reliable plan parsing and validation, and treating single samples as stable.

assumptions (3)
  • domain assumption The five PDDL domains and their 93 problems are representative of robotic planning complexity.
    Section 3.3 selects these IPC domains as proxies for real-world manipulation and scheduling; the generality of the conclusions depends on this choice.
  • domain assumption LLM outputs can be reliably parsed into the requested JSON structure and checked against the PDDL domain.
    Section 3.4.3 requires a strict JSON format; no parse-failure rate is reported, so the robustness of the pipeline is assumed.
  • domain assumption A single run per problem per model is sufficient to estimate success rate.
    Table 3 reports one value per model-domain cell; stochastic LLM sampling implies variance that is never reported, so single-shot results are treated as representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study." pith.science (2026). https://pith.science/paper/IUGFP2LJ

@misc{pith2026250723589,
  author       = {Pith},
  title        = {Pith review of: Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUGFP2LJ}},
  note         = {Machine review of arXiv:2507.23589}
}
read the original abstract

Recent advancements in Large Language Models have sparked interest in their potential for robotic task planning. While these models demonstrate strong generative capabilities, their effectiveness in producing structured and executable plans remains uncertain. This paper presents a systematic evaluation of a broad spectrum of current state of the art language models, each directly prompted using Planning Domain Definition Language domain and problem files, and compares their planning performance with the Fast Downward planner across a variety of benchmarks. In addition to measuring success rates, we assess how faithfully the generated plans translate into sequences of actions that can actually be executed, identifying both strengths and limitations of using these models in this setting. Our findings show that while the models perform well on simpler planning tasks, they continue to struggle with more complex scenarios that require precise resource management, consistent state tracking, and strict constraint compliance. These results underscore fundamental challenges in applying language models to robotic planning in real world environments. By outlining the gaps that emerge during execution, we aim to guide future research toward combined approaches that integrate language models with classical planners in order to enhance the reliability and scalability of planning in autonomous robotics.

Figures

Figures reproduced from arXiv: 2507.23589 by the authors.

Figure 1
Figure 1. This scatter plot illustrates the relationship between average plan length (x-axis) and average executed steps [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the planning capabilities of nine different LLMs against the classical planner [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Per-domain success and failure outcomes of the evaluated LLMs compared to the baseline classical planner [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 20 canonical work pages

  1. [1]

    The fast downward planning system

    Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 0 191--246, 2006. doi:10.1613/jair.1705

    doi:10.1613/jair.1705 2006

  2. [2]

    PDDL —the planning domain definition language

    Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. PDDL —the planning domain definition language. Technical Report CVC TR-98-003/DCS TR-1165, Yale Center for Computational Vision and Control, 1998

    work page 1998

  3. [3]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models . In Advances in Neural Information Processing Systems, volume 35, pages 24824--24837, 2022

    work page 2022

  4. [4]

    ReAct : Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023

    work page 2023

  5. [5]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models . In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi:10.1109/ICCV.2023.2998

    work page arXiv 2023

  6. [6]

    Generating Executable Action Plans with Environmentally-Aware Language Models

    Maitrey Gramopadhye and Daniel Szafir. Generating Executable Action Plans with Environmentally-Aware Language Models . In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3568--3575, 2023. doi:10.1109/IROS56739.2023.10045678

    work page arXiv 2023

  7. [7]

    Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, and et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances . In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 3--14, 2022. doi:10.48550/arXiv.2204.01691

    arXiv doi:10.48550/arxiv.2204.01691 2022

  8. [8]

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B. Murthy. Position: LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks . In Proceedings of the 41st International Conference on Machine Learning, 2024

    work page 2024

Show all 38 references
  1. [9]

    Can Large Language Models Reason and Plan? Annals of the New York Academy of Sciences, 2024

    Subbarao Kambhampati. Can Large Language Models Reason and Plan? Annals of the New York Academy of Sciences, 2024. doi:10.1111/nyas.15125

    2024 doi

  2. [10]

    Leveraging environment interaction for automated pddl translation and planning with large language models

    Sadegh Mahdavi, Raquel Aoki, Keyi Tang, and Yanshuai Cao. Leveraging environment interaction for automated pddl translation and planning with large language models. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  3. [11]

    LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: Empowering Large Language Models with Optimal Planning Proficiency . arXiv preprint arXiv:2304.11477, 2023

    2023 arXiv

  4. [12]

    Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents . In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning R...

    2022

  5. [13]

    Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example

    Yanan Chen, Ali Pesaranghader, Tanmana Sadhu, and Dong Hoon Yi. Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example . arXiv preprint arXiv:2408.06318, 2024

    2024 arXiv

  6. [14]

    Nathan Cloos, Meagan Jens, Michelangelo Naim, Yen-Ling Kuo, Ignacio Cases, Andrei Barbu, and Christopher J. Cueva. Baba Is AI: Break the Rules to Beat the Benchmark . In Proceedings of the ICML 2024 Workshop on LLMs and Cognition, 2024

    2024

  7. [15]

    AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation

    Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation . arXiv preprint arXiv:2408.00764, 2025. Accepted by KDD 2025...

    2025 arXiv

  8. [16]

    On the planning abilities of large language models: A critical investigation

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models: A critical investigation. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS), 2023 a

    2023

  9. [17]

    A Framework for Neurosymbolic Robot Action Planning using Large Language Models

    Alessio Capitanelli and Fulvio Mastrogiovanni. A Framework for Neurosymbolic Robot Action Planning using Large Language Models . Frontiers in Neurorobotics, 18, 2024. doi:10.3389/fnbot.2024.1342786

    2024

  10. [18]

    Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning

    Katharina Stein, Daniel Fi s er, J \"o rg Hoffmann, and Alexander Koller. Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning . In Proceedings of the ICAPS 2024 Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL), 202...

    2024 arXiv

  11. [19]

    Tenenbaum, Leslie Pack Kaelbling, and Michael Katz

    Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized Planning in PDDL Domains with Pretrained Large Language Models . Proceedings of the AAAI Conference on Artificial Intelligence, 38 0 (18), 2024. doi:10.1609/aaai....

    2024 doi

  12. [21]

    Minseo Kwon, Yaesol Kim, and Young J. Kim. Fast and Accurate Task Planning using Neuro-Symbolic Language Models and Multi-level Goal Decomposition . arXiv preprint arXiv:2409.19250, 2024

    2024 arXiv

  13. [22]

    CoPAL: Corrective Planning of Robot Actions with Large Language Models

    Frank Joublin, Antonello Ceravola, Pavel Smirnov, Felix Ocker, Joerg Deigmoeller, Anna Belardinelli, Chao Wang, Stephan Hasler, Daniel Tanneberg, and Michael Gienger. CoPAL: Corrective Planning of Robot Actions with Large Language Models . In 2024 IEEE International Conference...

    2024

  14. [23]

    Saycanpay: Heuristic planning with large language models using learnable domain knowledge

    Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20123--20133, 2024. doi:10.1609/aaai.v38i18.29991

    2024 doi

  15. [24]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, and et al. PaLM-E: An Embodied Multimodal Language Model . arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  16. [25]

    PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change . Advances in Neural Information Processing Systems, 36: 0 38975--38987, 2023 b

    2023

  17. [26]

    LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench

    Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench . arXiv preprint arXiv:2409.13373, 2024

    2024 arXiv

  18. [27]

    Littman, and Stephen H

    Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L. Littman, and Stephen H. Bach. Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages . arXiv preprint arXiv:2407.03321, 2024

    2024

  19. [28]

    NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptions

    Elliot Gestrin, Marco Kuhlmann, and Jendrik Seipp. NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptions . arXiv preprint arXiv:2405.04215, 2024

    2024

  20. [29]

    Le, Ed H

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, and Denny Zhou. NATURAL PLAN: Benchmarking LLMs on Natural Language Planning . arXiv preprint arXiv:2406.04520, 2024

    2024 arXiv

  21. [30]

    Open Grounded Planning: Challenges and Benchmark Construction

    Shiguang Guo, Ziliang Deng, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Open Grounded Planning: Challenges and Benchmark Construction . arXiv preprint arXiv:2406.02903, 2024

    2024 arXiv

  22. [31]

    CaT-Bench: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans

    Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, and Ray Mooney. CaT-Bench: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...

    2024 doi

  23. [32]

    The production ai platform built for developers, 2024

    Fireworks AI . The production ai platform built for developers, 2024. URL https://fireworks.ai/. Accessed: 2025-02-27

    2024

  24. [33]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023

    2023

  25. [34]

    Deepseek vs

    DataCamp AI Research Team. Deepseek vs. openai: Comparing the new ai titans. DataCamp Blog, 2025. URL https://www.datacamp.com/blog/deepseek-vs-openai. Accessed: 2025-02-27

    2025

  26. [35]

    Meta releases new llama 3.1 models, including highly anticipated 405b parameter variant

    IBM AI Research. Meta releases new llama 3.1 models, including highly anticipated 405b parameter variant. IBM Newsroom, 2025. URL https://www.ibm.com/think/news/meta-releases-llama-3-1-models-405b-parameter-variant. Accessed: 2025-02-27

    2025

  27. [36]

    Gemini 2.0 flash thinking experimental: A guide with examples

    Alex Olteanu. Gemini 2.0 flash thinking experimental: A guide with examples. DataCamp Blog, February 2025. URL https://www.datacamp.com/blog/gemini-2-0-flash-experimental. Accessed: 2025-02-27

    2025

  28. [37]

    Comparing claude 3.7 sonnet, claude 3.5 sonnet, openai o3-mini, deepseek r1, and grok 3 beta

    Swarit Sharma. Comparing claude 3.7 sonnet, claude 3.5 sonnet, openai o3-mini, deepseek r1, and grok 3 beta. Passionfruit Blog, February 2025. URL https://www.getpassionfruit.com/blog/comparing-claude-3-7-sonnet-claude-3-5-sonnet-openai-o3-mini-deepseek-r1-and-grok-3-beta. Acc...

    2025

  29. [38]

    Grok 3 beta—the age of reasoning agents

    xAI . Grok 3 beta—the age of reasoning agents. xAI Blog, February 2025. URL https://x.ai/blog/grok-3. Accessed: 2025-02-27

    2025

  30. [39]

    Claude 3.7 sonnet and claude code

    Anthropic. Claude 3.7 sonnet and claude code. Anthropic News, February 2025. URL https://www.anthropic.com/news/claude-3-7-sonnet. Accessed: 2025-02-27

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.