Pith. sign in

REVIEW 6 cited by

Plancraft: an evaluation dataset for planning with LLM agents

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.21033 v2 pith:CCV7M372 submitted 2024-12-30 cs.CL cs.AI

Gautier Dagan , Frank Keller , Alex Lascarides This is my paper · ORCID
classification cs.CLcs.AI
keywords plancraftagentagentsdatasetevaluateevaluationhandcraftedllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Plancraft, a multi-modal evaluation dataset for LLM agents. Plancraft has both a text-only and multi-modal interface, based on the Minecraft crafting GUI. We include the Minecraft Wiki to evaluate tool use and Retrieval Augmented Generation (RAG), as well as a handcrafted planner and Oracle Retriever, to ablate the different components of a modern agent architecture. To evaluate decision-making, Plancraft also includes a subset of examples that are intentionally unsolvable, providing a realistic challenge that requires the agent not only to complete tasks but also to decide whether they are solvable at all. We benchmark both open-source and closed-source LLMs and compare their performance and efficiency to a handcrafted planner. Overall, we find that LLMs and VLMs struggle with the planning problems that Plancraft introduces, and offer suggestions on how to improve their capabilities.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    The manuscript body introduces PokeGym, a vision-only automated 3D-game benchmark, while the abstract claims a G-EvoMAC method and 60.18% success rate absent from the body.

  2. AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration

    cs.MA 2026-07 accept novelty 6.0 of 10

    Background mention-waiting lets four Claude Code agents share discoveries mid-execution and reach 62.1% on SWE-Atlas QnA versus 32.3% for one agent.

  3. Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Relabeling LLM-agent trajectories with all goals actually achieved, plus action masking and reweighting, yields sample-efficient gains over SFT and DPO on ALFWorld, PlanCraft, and WebShop.

  4. Object-Centric Environment Modeling for Agentic Tasks

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Object-Centric Environment Modeling (OCM) builds an online executable object-and-procedure code model that improves average rank and cuts invalid actions on ScienceWorld, ALFWorld, and PlanCraft.

  5. LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.

  6. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

Pith tools