REVIEW 12 cited by
TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.
Forward citations
Cited by 12 Pith papers
-
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
LLM agents struggle to adaptively re-plan under progressively disclosed dual world and user constraints, with the best model at 67.75% accuracy on 307 household tasks.
-
ProEvent: An Event-centric Benchmark for Proactive Agents
ProEvent is a benchmark showing LLM agents keep a user's event timetable from chats poorly, with the best fully-correct score at 27.2%.
-
AlterAtlas: Shifting Travel Planning from AI Generation to Validation via Persona-Driven Simulations
AlterAtlas replaces one-shot AI itinerary generation with an interactive validation loop where persona-driven simulations expose route-level constraints and guide iterative revision.
-
When Do Multi-Agent Systems Help? An Information Bottleneck Perspective
Multi-agent LLM systems help exactly when the context removed by compressed relays outweighs the downstream-relevant information those relays discard.
-
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
MobilityBench is a 100,000-episode benchmark with a replay sandbox for deterministic evaluation of LLM route-planning agents; current models score well on basic tasks but fail preference-constrained routing.
-
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.
-
Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM-based Multi-Agent Systems
MOAT alternately aligns a planning and a grounding LLM agent via perplexity-based DPO and self-generated SFT, reporting small but consistent gains over independently tuned baselines.
-
Can Large Language Models Capture Human Risk Preferences? A Cross-Cultural Study
ChatGPT 4o and o1-mini chose more risk-averse lottery options than real respondents in Sydney, Hong Kong, Dhaka, and Nanjing; o1-mini was closer to humans, and Chinese prompts widened the gap.
-
MasHost Builds It All: Autonomous Multi-Agent System Directed by Reinforcement Learning
MasHost uses reinforcement learning to autonomously construct query-adaptive multi-agent graphs, and its authors report the best average accuracy across six LLM benchmarks.
-
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
A new step-level reward-modeling benchmark for multimodal agents shows current MLLMs reach at most 61.6 percent accuracy, and benchmark score correlates strongly (r=0.981 across five models) with downstream A* search ...
-
Think2Go: Generative Next POI Recommendation with LLM Reasoning
Think2Go couples SFT and RL-based reasoning in one LLM, with KDE- and reward-gap-based advantage calibration, and reports state-of-the-art Acc@1 on NYC, Tokyo, and California check-in data.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.