REVIEW 9 cited by
Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Tools
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) struggle to directly generate correct plans for complex multi-constraint planning problems, even with self-verification and self-critique. For example, a U.S. domestic travel planning benchmark TravelPlanner was proposed in Xie et al. (2024), where the best LLM OpenAI o1-preview can only find viable travel plans with a 10% success rate given all needed information. In this work, we tackle this by proposing an LLM-based planning framework that formalizes and solves complex multi-constraint planning problems as constrained satisfiability problems, which are further consumed by sound and complete satisfiability solvers. We start with TravelPlanner as the primary use case and show that our framework achieves a success rate of 93.9% and is effective with diverse paraphrased prompts. More importantly, our framework has strong zero-shot generalizability, successfully handling unseen constraints in our newly created unseen international travel dataset and generalizing well to new fundamentally different domains. Moreover, when user input queries are infeasible, our framework can identify the unsatisfiable core, provide failure reasons, and offers personalized modification suggestions. We show that our framework can modify and solve for an average of 81.6% and 91.7% unsatisfiable queries from two datasets and prove with ablations that all key components of our framework are effective and necessary. Project page: https://sites.google.com/view/llm-rwplanning.
Forward citations
Cited by 9 Pith papers
-
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
On 800 jointly constrained trip tasks with a deterministic scorer and achievable gold, the best of 15 LLM agents fully solves only 46.2% of feasible plans, with unstated persona needs as the universal bottleneck.
-
TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluation
TTPA teaches LLMs correct tool calls by sampling near-tie token choices, scoring them with an error taxonomy, and training with DPO.
-
Improving Large Language Model Planning with Action Sequence Similarity
Selecting in-context examples by action-sequence similarity to a model-generated draft plan improves LLM planning accuracy over random or problem-similarity selection.
-
CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
Training a small 8-billion-parameter model to guide GPT-4o between code and text raises its score on 37 math and logic tasks from 53.3 to 86.4, outpacing o1, o1-preview, and DeepSeek R1 on the aggregate.
-
Evolving Deeper LLM Thinking
Mind Evolution, an LLM-driven evolutionary search guided by a programmatic scorer, solves over 98% of TravelPlanner and Natural Plan instances with Gemini 1.5 Pro, outperforming Best-of-N and sequential revision at co...
-
TripTailor: A Real-World Benchmark for Personalized Travel Planning
A travel-planning benchmark is claimed in the abstract, but the full text is an unrelated supernova spectroscopy paper, leaving the central claim completely unsupported.
-
Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation
Fine-tuning experiments show instruction-following data improves LLM function-calling accuracy and relevance detection, a Decision Token plus synthetic negative examples helps non-relevant cases, and a tailored transl...
-
Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
A narrative synthesis of 27 agent evaluation papers identifies six recurring failure clusters and finds that agent failures compound non-linearly with task length, sub-skills do not compose into end-to-end success, an...
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.