{"id":"6801c57e-30f6-4b69-86a0-c603c6810461","arxiv_id":"2412.05255","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"TeamCraft presents a large multi-modal, multi-agent Minecraft benchmark and shows that current models generalize poorly to novel goals, scenes, and team sizes.","lead":"TeamCraft is a new Minecraft benchmark with 55,000 task variants for testing whether AI agents can collaborate using both images and text. It finds that current vision-language agents still struggle with such teamwork, especially when goals, scenes, or team sizes are new.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Planner-generated 'expert' demonstrations are unvalidated; low success rates may reflect demonstration bias rather than task difficulty.","rationale":"I agree with the reader's weakest assumption: the unvalidated planner demonstrations are the most load-bearing concern. The benchmark itself is substantial and well documented, and the dataset statistics appear internally consistent. However, the central claim that current models struggle on TeamCraft is only meaningful if the training signal is of expert quality. If the planner's hand-tuned cost function produces idiosyncratic strategies, the VLA models may fail on novel splits because of distribution shift from the planner's policy, not because the tasks require genuinely hard multi-modal collaboration. The grid-world confound (privileged coordinates vs. images) is a real secondary issue, but it only weakens a supporting ablation, not the main benchmark conclusion. A concrete test replacing or perturbing the demonstration source would settle whether the difficulty result is robust. The reader's CONDITIONAL verdict remains appropriate; no change to ACCEPT or REJECT is warranted without this validation.","tokens_in":37871,"tokens_out":5069,"duration_ms":52653,"concrete_test":"Sample 100 task variants spanning all four families. Generate a second demonstration set using either (a) human players completing the same tasks, or (b) a modified planner with randomized weights or a greedy heuristic instead of Eq. (1). Train TeamCraft-VLA-7B-Cen separately on original and alternative demonstrations (same 3-epoch protocol), then evaluate both on the same 950-case test/generalization split. If success rates, especially Farming Crop and Agents splits, differ by more than 10 percentage points, the difficulty conclusions are confounded by demonstration quality. As a lighter check, compute each original demonstration's action count relative to a lower bound (e.g., target block count plus unavoidable furnace steps); if many exceed the bound substantially, the 'expert' label is questionable.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central difficulty claim depends on the planner-generated demonstrations being reliable expert behavior (Section 3.6, Appendix D). The planner minimizes C = w1T + w2 sum(E_i) + w3D + w4 sum(c_ij) + w5U with per-task hand-set weights (Building w2=1.4, Clearing w4=1.8, Smelting w3=1.8; other weights 0.8-1.0). No independent validation shows these trajectories are near-optimal, human-comparable, or even consistent with the partial observations available to the trained agents. Since every TeamCraft-VLA model is trained solely on these demonstrations (Section F, 3 epochs), systematic planner biases—e.g., the rigid one-block-per-agent construction seen in Figures 35-36, or the use of privileged coordinate information—will be learned. The observed poor generalization, especially the 0.00 Farming Crop rate for all models including Grid-World, may then reflect mismatch with the planner's policy rather than intrinsic difficulty of multi-modal multi-agent collaboration. The datasheet only states the data was 'verified by the team via manual inspection,' which is not a quantitative quality check. Until demonstration quality is validated, the conclusion that 'existing models face significant challenges' is not fully supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents TeamCraft, a benchmark for multi-modal multi-agent collaboration in Minecraft. It contains 55,257 procedurally generated demonstrations across four task types (Building, Clearing, Farming, Smelting), specified by multi-modal prompts that combine language instructions with orthographic-view images. The paper introduces a planner that uses privileged information to generate 'expert' demonstrations, then evaluates several baselines: TeamCraft-VLA (7B and 13B, centralized and decentralized), GPT-4o (one-shot), and a text-based Grid-World ablation. The evaluation protocol defines task success rate, subgoal success rate, and redundancy rate. Results show that all models achieve low success rates on the test set and on generalization splits (novel goals, novel scenes, and four-agent teams), with Farming Crop generalization at 0.00 for every model. The authors conclude that current multi-modal multi-agent models face significant challenges in generalization.","tokens_in":38094,"tokens_out":6134,"duration_ms":64412,"significance":"TeamCraft addresses a real gap in the literature: few benchmarks combine multi-modal task specification, vision-based multi-agent control, and diverse generalization splits in a visually rich 3D environment. The released dataset, code, and checkpoints are concrete contributions, and the dataset statistics in Table 2 sum consistently (55,257 demonstrations), with a datasheet reporting 57,207 total instances including 1,000 validation and 950 test cases. The evaluation metrics are clearly defined. If the demonstrations are validated, the benchmark could become a valuable testbed for studying multi-modal multi-agent generalization. However, the central conclusion that 'existing models face significant challenges' depends on the quality and representativeness of the planner-generated demonstrations, and the current evidence for that quality is thin.","major_comments":[{"comment":"The demonstration generation algorithm is underspecified. The cost function C = w1T + w2*sum(E_i) + w3D + w4*sum(c_ij) + w5U is given, but the paper never explains how the planner optimizes this cost (e.g., exact search, greedy assignment, or heuristic scheduling) or how ties are broken. This is load-bearing for two reasons. First, without this algorithmic detail the benchmark cannot be reproduced, even with released code, because the design decisions are not documented. Second, all VLA baselines are trained solely on these trajectories, which are labeled 'expert demonstrations,' yet the only quality evidence is the datasheet statement that the data were 'verified by the team via manual inspection.' There is no quantitative validation that the trajectories are near-optimal, human-comparable, or even consistent with the partial observations available to agents at test time. The observed poor generalization, especially the 0.00 Farming Crop result for all models, could therefore reflect a mismatch between the planner's privileged-information policy and the first-person observations, rather than intrinsic task difficulty. Please provide the optimization algorithm, and add at least one quantitative validation, such as comparing demonstration length against a theoretical lower bound, training a privileged-information oracle on the demonstrations, or reporting a human-collected demonstration baseline.","section":"Section 3.6, Eq. (1), Appendix D"},{"comment":"The test and generalization splits contain only 50 cases per condition, which makes the point estimates very imprecise for binary success metrics. For example, a reported success rate of 0.00 has a 95% confidence interval upper bound of about 0.07, and differences such as 0.02 vs 0.04 (Building Agents, 7B vs 13B) are within sampling noise. The paper draws conclusions like 'scaling up model sizes blindly does not guarantee success' from such comparisons without reporting confidence intervals or significance tests. To make the benchmark results more trustworthy, please report bootstrap confidence intervals or a significance test for the headline results in Figure 5 and Table 8.","section":"Section 4.3, Tables 6-8"}],"minor_comments":[{"comment":"The phrase 'More than 30 target object or resource are used' should be 'More than 30 target objects or resources are used.'","section":"Section 3.5"},{"comment":"The validation set (1,000 instances) appears only in the datasheet; state its split in the main text for completeness, alongside the training (55,257) and test (950) sizes.","section":"Section 3.7, Appendix K"},{"comment":"The datasheet says 'The dataset contain all possible instances'; the verb should be 'contains.'","section":"Appendix K"},{"comment":"The caption contains a stray token 'Object 232233' that should be removed.","section":"Figure 2"},{"comment":"The figure is dense with multiple panels and lines; consider adding a grayscale-readable legend and ensuring y-axis labels are visible in every panel.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially valuable and the authors have shipped a datasheet, code, and checkpoints, which is commendable. My main concern is the unvalidated planner-generated demonstrations; if the authors can provide even a modest quantitative check (e.g., a lower-bound comparison or a privileged-information behavior-cloning oracle), I would be willing to accept a revision. If not, the benchmark's usefulness as a measure of task difficulty is substantially weakened. I also recommend the authors consider adding confidence intervals, as the current small test sets make several comparisons unreliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the benchmark itself is solid and fills a real gap: no prior work combines multi-modal prompts, multi-agent Minecraft control, 55,000 procedural variants, and explicit generalization splits for goals, scenes, and agent counts. The dataset statistics are internally consistent (Table 2 sums to 55,257; the datasheet reports 57,207 total), the evaluation protocol is clearly spelled out, and the code, data, and model checkpoints are released. That is a substantive contribution that the community will likely build on. Second, the main empirical conclusion—that existing VLA models struggle on this benchmark—rests on training data generated by a hand-tuned planner. The planner uses privileged information and a cost function with per-task weights (w2=1.4, w4=1.8, w3=1.8, etc.), and there is no independent check that these trajectories are near-optimal, human-comparable, or even consistent with the partial observations the agents actually receive. The datasheet only mentions manual inspection by the team. That is a real soft spot. The stress-test is half right: the 0.00 Farming Crop rate and other failures could reflect a mismatch with the planner's policy rather than intrinsic task difficulty. But I would not call this load-bearing. The benchmark's value as a testbed does not hinge on the expert demonstrations being perfect, and the failure patterns (ignoring a fourth agent, object mismatches) look like genuine learning problems. Still, the paper's difficulty claim should be softened until demonstration quality is validated. Two smaller issues. No error bars or multiple seeds, so the differences between models are hard to judge. The grid-world ablation conflates modality with privileged coordinates: the text input lists exact block positions, so the comparison is not purely about modality. Both are fixable in revision. Overall: this paper deserves a serious referee. It is a well-documented, reproducible benchmark with clear novelty. The planner-demo concern is worth raising in review, and the authors should add validation or at least a sensitivity analysis of the cost weights. I would bring it to a reading group on embodied multi-agent learning, and I would likely cite it if I worked on Minecraft or multi-modal multi-agent benchmarks.","headline":"A genuinely useful benchmark for multi-modal multi-agent Minecraft tasks, but the 'models struggle' conclusion should be read with a grain of salt until the planner-generated demonstrations are independently validated.","tokens_in":700,"tokens_out":696,"would_cite":true,"duration_ms":27956,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TeamCraft introduces a 55,000-task Minecraft benchmark and shows that current vision-language-action models struggle on unseen goals, scenes, and larger teams.","keywords":["multi-agent systems","multi-modal prompts","benchmark","Minecraft","vision-language-action model","generalization","imitation learning","embodied agents"],"falsifier":"A concrete falsifier would be to retrain TeamCraft-VLA on demonstrations collected from human players or from a planner using different cost weights, and evaluate on the same generalization splits; if novel-goal or four-agent success rates rise substantially, the reported difficulty is an artifact of the expert-demonstration policy rather than an inherent property of multi-modal multi-agent tasks. A second falsifier: give the model the ground-truth goal text instead of the orthographic views on the novel-crop farming split; if success jumps from 0.00 to high values, the bottleneck is visual grounding of the goal rather than multi-agent coordination.","tokens_in":37686,"feed_emoji":"🎮","tokens_out":5218,"duration_ms":52208,"temperature":0.7,"pith_summary":"This paper aims to establish that multi-modal multi-agent collaboration can be systematically measured in a rich 3D environment, and that existing models fail at it. TeamCraft generates over 55,000 procedurally varied Minecraft tasks specified by interleaved language and image prompts, produces expert demonstrations with a privileged planner, and creates held-out splits for novel goals, unseen scenes, and four-agent teams. The central empirical finding is that trained vision-language-action models, as well as a proprietary multimodal model, succeed on familiar configurations but collapse on generalization splits, with a 0.00 task success rate on novel farming crops across all models. The authors argue this reveals a real gap in multi-modal multi-agent planning and coordination that the benchmark is designed to expose.","feed_headline":"AI agents score zero on novel crops in Minecraft benchmark","feed_subtitle":"TeamCraft's 55,000-task benchmark shows poor generalization to unseen goals, scenes, and team sizes.","key_machinery":"The central object is the multi-modal prompt: a language instruction interleaved with three orthographic view images (top, left, front) that specify goal or initial states. Demonstrations are generated by a planner that uses privileged environment information to minimize a cost function $C = w_1T + w_2\\sum_i E_i + w_3D + w_4\\sum_i\\sum_{j\\in A_i} c_{ij} + w_5U$, where weights are hand-tuned per task family. The TeamCraft-VLA model combines a vision encoder, a projector, and a language model to map these prompts, per-agent first-person views, and inventories to high-level skills. Generalization splits (novel shape/material, novel crop, novel goal, novel scene, four agents) provide the evaluation machinery that makes the paper's difficulty claims measurable.","core_discovery":"The core claim is that multi-modal multi-agent generalization can be benchmarked in a visually rich 3D world, and that current models do not generalize. The benchmark covers four task families—building, clearing, farming, and smelting—with more than 55,000 procedurally generated task variants. Agents receive first-person RGB images and inventory information, and emit high-level actions; training uses planner-generated expert demonstrations that minimize a weighted cost over completion time, idle actions, dependencies, action costs, and redundancy. Centralized 7B and 13B vision-language-action models reach task success rates as high as 0.64 on in-distribution clearing tasks, but fall to near zero on novel goals and four-agent settings; the novel-crop farming split is 0.00 for every model tested. The proprietary multimodal model scores near zero everywhere, with failures concentrated in 3D spatial reasoning and object-state recognition.","pith_inferences":["A testable extension would be to add explicit communication channels to decentralized TeamCraft agents; comparing redundancy rates and task success with the current implicit-communication setting would show how much of the gap is due to missing coordination.","Because demonstrations come from a planner with hand-set cost weights, human-collected or differently weighted demonstrations on the same tasks could reveal whether the observed generalization failures are inherent to multimodal multi-agent learning or an artifact of one particular expert policy.","The scene splits change textures, lighting, and biome layouts together, so part of the reported scene generalization may be appearance robustness rather than task-level generalization; disentangling these factors would sharpen the benchmark's diagnostic value.","The 0.00 novel-crop result offers a sharp probe: a model that can read the crop name from the orthographic blueprint and map it to its inventory should at least sow the correct seed, so failure on that single step is a strong test of grounded multimodal understanding."],"forward_implications":["If TeamCraft's results are correct, current multimodal vision-language-action models are not ready for zero-shot or few-shot multi-agent planning in 3D worlds; they need explicit mechanisms for unseen goals and for coordinating teams of unseen size.","The uniform 0.00 on novel farming crops indicates that object-state recognition, not just spatial reasoning, is a bottleneck for embodied multi-agent generalization.","Centralized control outperforms decentralized control and produces far fewer redundant actions, suggesting that models without inter-agent communication or joint inference degrade as team size grows.","The grid-world ablation, where the same tasks are described in text instead of images, achieves substantially higher success rates, implying that a large part of the failure is visual grounding rather than high-level task planning alone.","Scaling training data helps in-distribution and scene splits but does not repair novel-goal or novel-agent generalization, and larger models do not automatically close that gap."],"supporting_citations":[{"why":"supplies the multi-modal prompt formulation of interleaved language and image tokens that TeamCraft extends to multi-agent tasks","marker":"[24]"},{"why":"provides the visual-instruction-tuning architecture with image encoder, projector, and language model that TeamCraft-VLA follows","marker":"[28]"},{"why":"establishes Minecraft as a scalable embodied benchmark and gives the comparison for single-agent internet-scale knowledge","marker":"[11]"},{"why":"is the single-agent visual embodied benchmark used as the contrast for TeamCraft's multi-agent and multi-modal task specification","marker":"[47]"},{"why":"represents the state-based multi-agent benchmark that motivates TeamCraft's move to multi-modal observations","marker":"[37]"},{"why":"defines a recent large-scale multi-agent embodied benchmark against which TeamCraft positions its variant count and generalization targets","marker":"[6]"}],"fun_headline_variants":["Minecraft benchmark: AI agents fail novel tasks","TeamCraft: 55k tasks, but AI can't generalize","Zero score on new crops: AI stumbles in TeamCraft","AI agents struggle with unseen goals in Minecraft","TeamCraft benchmark exposes AI generalization gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the planner-generated demonstrations, produced under a hand-tuned cost function with privileged environment information, are a good enough proxy for expert collaborative behavior that imitation learning on them measures multi-modal multi-agent generalization rather than reproducing one particular planning policy.","fun_headline_variants_meta":{"raw":{"variants":["Minecraft benchmark: AI agents fail novel tasks","TeamCraft: 55k tasks, but AI can't generalize","Zero score on new crops: AI stumbles in TeamCraft","AI agents struggle with unseen goals in Minecraft","TeamCraft benchmark exposes AI generalization gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1332,"prompt_tokens":902,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":518,"tokens_out":430,"duration_ms":4493,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:32.803943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier would be to retrain TeamCraft-VLA on demonstrations collected from human players or from a planner using different cost weights, and evaluate on the same generalization splits; if novel-goal or four-agent success rates rise substantially, the reported difficulty is an artifact of the expert-demonstration policy rather than an inherent property of multi-modal multi-agent tasks. A second falsifier: give the model the ground-truth goal text instead of the orthographic views on the novel-crop farming split; if success jumps from 0.00 to high values, the bottleneck is visual grounding of the goal rather than multi-agent coordination.","supporting_citations":[{"cited_title":"Visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"provides the visual-instruction-tuning architecture with image encoder, projector, and language model that TeamCraft-VLA follows"},{"cited_title":"Mine- Dojo: Building open-ended embodied agents with internet- scale knowledge,","cited_arxiv_id":null,"evidence_quote":"establishes Minecraft as a scalable embodied benchmark and gives the comparison for single-agent internet-scale knowledge"},{"cited_title":"ALFRED: A benchmark for interpreting grounded instructions for every- day tasks,","cited_arxiv_id":null,"evidence_quote":"is the single-agent visual embodied benchmark used as the contrast for TeamCraft's multi-agent and multi-modal task specification"},{"cited_title":"RoCo: Dialectic multi-robot collaboration with large language models,","cited_arxiv_id":null,"evidence_quote":"represents the state-based multi-agent benchmark that motivates TeamCraft's move to multi-modal observations"}],"review_version":1}