REVIEW 9 cited by
LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities. PlanBench, an extensible benchmark we developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating the planning abilities of LLMs. Despite the slew of new private and open source LLMs since GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs--making it a new kind of model: a Large Reasoning Model (LRM). Using this development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new LRMs do on PlanBench. As we shall see, while o1's performance is a quantum improvement on the benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings to the fore questions about accuracy, efficiency, and guarantees which must be considered before deploying such systems.
Forward citations
Cited by 9 Pith papers
-
SymStep: Symbolic Step Verification for Logical Reasoning
SymStep couples atomic LLM deductions to a deterministic constraint propagator with MRV hints, reaching ~97–100% on constraint-dense logic puzzles where CoT scores 0%.
-
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.
-
LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs
LogiPlan introduces a three-task benchmark with controllable graph complexity, showing that although reasoning models excel at plan generation, all models degrade sharply on cycle detection and deep comparison questions.
-
Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation
Nested exception-chain eligibility breaks frontier LLMs in unstable ways; an SMT execution layer makes outcomes deterministic given authored rules.
-
Embark Now: User Demand Oriented Framework for Multi-day Urban Travel Itinerary Planning
UDOIP combines dual-layer LLM preference extraction with clustering-and-substitution GRASP to produce higher-scoring, constraint-feasible multi-day urban itineraries than LLM-only and adapted solver baselines on two C...
-
Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories
Large reasoning models frequently ignore explicitly injected ground-truth solutions mid-trajectory and continue to overthink, often ending on incorrect answers.
-
Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
Reasoning-optimized LLMs reach at most 63.4% success across the five PDDL benchmarks, versus 97.9% for Fast Downward, and frequently produce plans that fail during execution.
-
The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models
Prompted foundation models plus symbolic programs can match or approach trained neuro-symbolic systems on several reasoning benchmarks, but the empirical support is uneven and the position restates known ideas.
-
CoreThink: A Symbolic Reasoning Layer to reason over Long Horizon Tasks with LLMs
The paper claims a symbolic orchestration layer, CoreThink, achieves state-of-the-art results on seven coding and reasoning benchmarks with no training, but provides no verifiable implementation or method details.
Discussion (0). Sign in to comment.