REVIEW 12 cited by
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce NATURAL PLAN, a realistic planning benchmark in natural language containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. We focus our evaluation on the planning capabilities of LLMs with full information on the task, by providing outputs from tools such as Google Flights, Google Maps, and Google Calendar as contexts to the models. This eliminates the need for a tool-use environment for evaluating LLMs on Planning. We observe that NATURAL PLAN is a challenging benchmark for state of the art models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only achieve 31.1% and 34.8% solve rate respectively. We find that model performance drops drastically as the complexity of the problem increases: all models perform below 5% when there are 10 cities, highlighting a significant gap in planning in natural language for SoTA LLMs. We also conduct extensive ablation studies on NATURAL PLAN to further shed light on the (in)effectiveness of approaches such as self-correction, few-shot generalization, and in-context planning with long-contexts on improving LLM planning.
Forward citations
Cited by 12 Pith papers
-
Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Skill entropy, a reference-model-based measure of skill-switching difficulty, calibrates a new cross-skill benchmark and serves as an RL reward, more than doubling small models' scores.
-
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
On 800 jointly constrained trip tasks with a deterministic scorer and achievable gold, the best of 15 LLM agents fully solves only 46.2% of feasible plans, with unstated persona needs as the universal bottleneck.
-
SCHEDBench: A Benchmark for Evaluating LLM Constraint Faithfulness in Natural-Language Combinatorial Scheduling
SCHEDBench shows that large language models are not reliably invariant to semantically equivalent natural-language renderings of the same scheduling problem, with constraint reordering producing the clearest above-noi...
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.
-
Programming over Thinking: Efficient and Robust Multi-Constraint Planning
SCOPE separates query-specific reasoning from reusable solver code and reports state-of-the-art success rates on TravelPlanner and Natural Plan at lower token cost.
-
ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Plans
ISO-Bench is presented as a benchmark for cross-modal causal reasoning, but its positive and negative examples are constructed from temporal position, allowing a non-causal image-text matching shortcut.
-
LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs
LogiPlan introduces a three-task benchmark with controllable graph complexity, showing that although reasoning models excel at plan generation, all models degrade sharply on cycle detection and deep comparison questions.
-
Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents
A new multi-turn benchmark shows LLMs struggle to keep global constraints satisfied when local constraints are added later, and often sacrifice budget to satisfy soft preferences.
-
Make Planning Research Rigorous Again!
A position paper calling for LLM-based planning research to reuse the rigor, benchmarks, and tools of the classical automated planning community.
-
Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
Reasoning-optimized LLMs reach at most 63.4% success across the five PDDL benchmarks, versus 97.9% for Fast Downward, and frequently produce plans that fail during execution.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
Discussion (0). Continue with ORCID to comment.