REVIEW 7 cited by
On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) how good LLMs are by themselves in generating and validating simple plans in commonsense planning tasks (of the type that humans are generally quite good at) and (2) how good LLMs are in being a source of heuristic guidance for other agents--either AI planners or human planners--in their planning tasks. To investigate these questions in a systematic rather than anecdotal manner, we start by developing a benchmark suite based on the kinds of domains employed in the International Planning Competition. On this benchmark, we evaluate LLMs in three modes: autonomous, heuristic and human-in-the-loop. Our results show that LLM's ability to autonomously generate executable plans is quite meager, averaging only about 3% success rate. The heuristic and human-in-the-loop modes show slightly more promise. In addition to these results, we also make our benchmark and evaluation tools available to support investigations by research community.
Forward citations
Cited by 7 Pith papers
-
HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents
HERAKLES couples a language-model planner to a small, continually retrained skill executor and outperforms three baselines on the 17-goal Crafter benchmark, scaling better to reworded and repeated goals.
-
Synthesis by Design: Controlled Data Generation via Structural Guidance
A structural code-intervention method generates new math problems with labeled intermediate steps and a harder benchmark, and fine-tuning on the data mostly improves LLM math performance.
-
SimRPD: Optimizing Recruitment Proactive Dialogue Agents through Simulator-Based Data Evaluation and Selection
SimRPD trains a recruiting dialogue agent on simulator-generated conversations filtered to match real intent-transition patterns, lifting contact-acquisition rate from 3.8% to 4.4% in a live A/B test.
-
Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning
The thesis demonstrates that combining implicit 3D scene representations with LLM-based reasoning, using text as an interface, yields strong performance on robotic perception and spatial language tasks.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
-
From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning
Fine-tuning LLMs on synthetic instructions transfers well to simple spatial tasks but degrades on regular, repetitive layouts when instructions are human-authored.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.