{"id":"6587090a-99c7-4477-b9f4-739267ce868b","arxiv_id":"2412.11761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"HIVE turns natural-language commands into executable behavior-tree plans for swarms of up to 2,000 units, and a new five-task benchmark shows current LLMs win only a few of 50 trials per model.","lead":"This paper introduces HIVE, a system in which a human types high-level commands and an LLM converts them into behavior-tree plans that control thousands of units in a real-time strategy game. It also presents a five-task benchmark for LLM-driven swarm control, where current LLMs succeed only in a minority of trials and are sensitive to how prompts are phrased.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark ability tests encode the winning strategy in the listed prompts, so Table I largely measures plan-formatting of human strategy rather than the claimed LLM coordination and strategic-planning abilities.","rationale":"The reader's weakest assumption and my stress-test identify the same load-bearing issue: the benchmark's construct validity. The test prompts in Appendix I embed the winning strategy, and all low-level execution is done by fixed handcrafted behavior trees, so the reported success rates largely measure how faithfully an LLM reformats an operator's strategy into the plan language. The paper's own 'HIVE alone' ablation supports this reading, since generic strategy-free prompts cause most wins to disappear. This does not refute the narrower, well-evidenced claim that HIVE can translate high-level human instructions into executable plans in this game, supported by the reported executions, saved plans, and deterministic re-execution. But it does invalidate the broader interpretation that the benchmark demonstrates autonomous LLM coordination or strategic planning abilities. The paper is still a useful framework and a transparently specified benchmark; the concern is addressable by reinterpreting the ability scores, adding non-LLM baselines, and rerunning the tests with strategy-free prompts. I therefore do not change the reader's conditional verdict.","tokens_in":33322,"tokens_out":6506,"duration_ms":76231,"concrete_test":"Build a deterministic non-LLM baseline that consumes the same map description and the 50 test prompts, using keyword and template extraction for bridges, markers, 'middle row', and forest waypoints, and emitting plans with the same five handcrafted behavior trees. Run it on all five ability tests and compare against Table I. If this baseline wins Follow markers, Strategize points, and Exploit terrain at rates at or above the best LLM, then those tests do not require LLM-level planning and the ability labels are confounded. If the baseline fails where LLMs succeed, the LLM contribution is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.A states that the five ability tests assess 'coordination, exploitation of weaknesses, adherence to spatial markers, terrain utilization, and strategic planning.' The Appendix I prompts, however, supply the strategy. For Strategize points, prompt 0 is 'Split our units on all the bridges to defend them and make sure to send archers and spearmen units on each one.' Coordinate prompts specify covering the middle row and protecting archers; Exploit terrain prompts give an ordered sequence of forest waypoints and say to ignore enemies; Follow markers prompts name markers A-D and the objective. Since Section III-B restricts the LLM to choosing among five handcrafted behavior trees, the model's role is largely to transcribe the given strategy into the plan grammar and assign unit slices and target coordinates. The 'HIVE alone' ablation in Section V-B confirms the dependency: with generic prompts that do not state a strategy, wins mostly disappear (e.g., Coordinate: Sonnet 2/10 with human strategy vs 0/10 alone; Exploit terrain: 4/10 vs 0/10). Thus the claim in Section IV.C that HIVE 'effectively possesses all the proposed abilities' is not supported for coordination or strategic planning. The narrower claim that HIVE translates high-level human instructions into executable plans is supported by the reported executions; the broader benchmark-interpretation claim is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HIVE, a framework in which a large language model converts a high-level player instruction into a structured multi-step plan for controlling units in a real-time strategy game, together with a five-task benchmark intended to measure LLM abilities in coordination, exploitation of unit weaknesses, marker following, terrain use, and strategic planning. The plan assigns each unit a target position and one of five handcrafted behavior trees; execution is deterministic in a JAX-based simulator. Nine LLMs are evaluated with ten prompt variations per task, and additional experiments test scaling to larger unit counts, performance without a human-provided strategy, and text versus image map descriptions. The main findings are that several closed-source models can produce executable plans that win in at least some conditions, that results are highly sensitive to prompt wording, that removing the human-provided strategy sharply reduces win rates, and that textual map descriptions outperform image-based inputs.","tokens_in":33566,"tokens_out":9614,"duration_ms":85336,"significance":"If the paper's narrower claim is taken as the contribution--that an LLM can translate a high-level, human-specified strategy into an executable plan for hundreds to thousands of units--the work is a solid and useful demonstration. The plans are saved and can be re-executed deterministically, the environment and prompts are documented in detail, and the comparison across nine models gives a first empirical signal. The paper also reports failures honestly, including invalid plans from a small open model, sensitivity to prompt wording, and poor performance with image inputs. The broader claim that the five benchmark tasks measure distinct LLM capabilities such as strategic planning is not yet supported, because the test prompts largely dictate the winning strategy and the low-level behaviors are fixed handcrafted trees. With additional baselines and a validation of what each task measures, the benchmark could become a valuable community resource.","major_comments":[{"comment":"The ability tests do not currently measure the named capabilities. Section IV.A defines the five tests as assessing coordination, exploitation of weaknesses, marker following, terrain utilization, and strategic planning, but the prompts in Appendix I supply the winning strategy for most tasks: Coordinate prompts tell the model to cover the central row and protect archers; Exploit weakness prompts instruct splitting into three groups matched to enemy weaknesses; Exploit terrain prompts give ordered forest waypoints and say to ignore enemies; Follow markers prompts name the markers and the objective; Strategize points prompts say to split the army across all bridges with archers plus close-range units. Since Section III-B restricts the LLM to choosing among five handcrafted behavior trees, the LLM's role is largely to reformat the provided strategy into the plan grammar and assign unit slices and coordinates. The 'HIVE alone' ablation in Section V-B confirms this dependency for Coordinate and Exploit terrain, where wins drop to 0/10 for both 4o and Sonnet. The Section IV.C statement that HIVE 'effectively possesses all the proposed abilities' is therefore not supported for the named abilities; what is supported is the narrower claim that HIVE translates human-provided strategy into executable plans.","section":"IV.A, Appendix I, V.B"},{"comment":"The evaluation does not exercise the 'dialog' component of the framework. The Abstract and Section I describe a natural language dialog, and Section III-C presents an outer loop in which the player prompts the LLM and the plan is executed, but the benchmarks and ablations send exactly one pre-written prompt per episode and then execute the parsed plan; the paper explicitly states that HIVE never asks the LLM to update the plan or try again. Consequently, multi-turn clarification, plan revision, and the interactive collaboration that motivates the hybrid approach are not tested. The human-machine collaboration claims should be restricted to one-shot instruction translation, or a dialog-based evaluation should be added.","section":"III-C and IV"},{"comment":"The scaling experiment does not establish the 'up to 2,000 agents' claim as a performance result. The text says the number of units was varied from 200 to 4,000 and that the authors stopped at 4,000, but Fig. 7 shows only 100 to 2,000 units, and the text also states that 'there is no significant conclusion regarding the success rate' due to LLM variance. The scaling study should be reported with the range actually tested, and the claims should be calibrated to what the data support, presenting the result as a capacity demonstration rather than as evidence of maintained performance at scale.","section":"V-A, Fig. 7"},{"comment":"The model-comparison results are presented without uncertainty quantification, and there is no non-LLM baseline. With n=10 binary trials per condition, win-count differences such as 2/10 versus 0/10 (Table I, Coordinate) are within binomial sampling noise, and the continuous metrics in Fig. 5 lack confidence intervals. In addition, because no template-based, scripted, or random-plan baseline is included, the benchmark cannot separate the LLM's planning contribution from the information contained in the prompt and the handcrafted behavior trees. Adding such baselines and reporting exact binomial intervals would substantially strengthen the benchmark's conclusions.","section":"IV.B, Table I, Fig. 5"}],"minor_comments":[{"comment":"There are several typos in the Appendix I prompts ('Judisiously', 'troups', 'loose time'); if these are unintentional they should be corrected, and if they are intentional robustness variations this should be stated.","section":"Appendix I"},{"comment":"The example text says '10 bridges in total (5 on the Northern River and 4 on the Southern River)' and the caption says 'ten bridges instead of nine,' but 5 + 4 = 9; the bridge count should be made consistent.","section":"Fig. 1"},{"comment":"The system prompt says 'one unit can belong to two groups' while Section III-D1 and the later IMPORTANT note require each unit to belong to no more than one group per step; the instruction text should be aligned.","section":"Appendix E1"},{"comment":"Section V-A states that units were varied from 200 to 4,000 and that the authors stopped at 4,000, but Fig. 7's x-axis ends at 2,000; the text and figure should be reconciled.","section":"V-A"},{"comment":"The Related Work section would benefit from a sentence explicitly contrasting HIVE's single-shot plan generation with Cicero's multi-turn negotiation, since the dialog claim in the Introduction is otherwise hard to position.","section":"II"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is a legitimate systems/demonstration contribution, and the documentation of plans, prompts, and deterministic re-execution is a strength. The main risk is overclaiming the benchmark's construct validity; if the authors revise the ability labels and add baselines, the paper would be publishable. The paper seems better suited to an applied AI or multi-agent systems venue than to a pure machine learning theory venue. No concerns about citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a useful framework-and-benchmark paper whose headline ability claims outrun the evidence. What is genuinely new is the HIVE pipeline—natural-language instructions, LLM-generated structured plans, handcrafted behavior trees, and a JAX RTS engine—plus a five-task benchmark with complete prompt and grammar specifications. The paper is also honest in its ablations: the \"HIVE alone\" condition and the vision-versus-text comparison are the right experiments to run. The core result, that an LLM can translate a human's high-level instruction into an executable plan that wins in this environment, is supported by deterministic re-execution and detailed logs.\n\nThe soft spot is central, not cosmetic. The Appendix I prompts for the five ability tests essentially dictate the strategy. Strategize points says \"Split our units on all the bridges to defend them...\"; Exploit terrain gives an ordered list of forest waypoints; Coordinate says to cover the middle row and protect archers; Exploit weakness says to split into three groups against the three enemy battalions. Since the LLM can only choose among five behavior trees and must fill in unit slices and coordinates, Table I largely measures plan-formatting compliance, not the named abilities. The paper's own ablation confirms this: generic prompts without the human strategy collapse wins (e.g., Coordinate Sonnet 2/10 vs 0/10). So the sentence in Section IV.C that HIVE \"effectively possesses all the proposed abilities\" is not supported for coordination or strategic planning. The narrower claim—HIVE translates high-level human instructions into executable plans—holds.\n\nOther issues are more conventional: no non-LLM baseline (a scripted planner would clarify what the LLM adds), n=10 per condition without confidence intervals, and no code or data release despite the project page. The paper acknowledges some of these in the discussion, which is to its credit. The lack of a baseline and CIs is a revision-level fix, not a fatal flaw. The benchmark itself discriminates between models and will likely be reused.\n\nWho this is for: anyone building LLM-driven game agents or human-swarm interfaces, and benchmark designers in the LLM-agent space. It deserves a serious referee—the framework is solid and the benchmark is a real artifact. My recommendation: conditional accept, with the requirement that the authors either reframe the ability claims as \"plan-following under strategy-specified prompts\" or add a non-LLM baseline and a stronger set of generic prompts. Ideally they also release the code, because the deterministic re-execution is the strongest evidence they have.","headline":"A useful framework-plus-benchmark paper whose headline ability claims outrun the evidence: the test prompts encode the winning strategy, so success largely reflects plan-formatting of human strategy rather than autonomous coordination.","tokens_in":34095,"tokens_out":3843,"would_cite":true,"duration_ms":32254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single human can coordinate up to 2,000 game units through natural-language orders that an LLM translates into executable plans.","keywords":["large language models","multi-agent coordination","human-swarm interaction","behavior trees","real-time strategy games","LLM benchmark","natural language planning","hybrid intelligence"],"falsifier":"Re-run the five ability tests with prompts that state only the mission objective instead of the tactical instructions in Appendix I, using the same ten prompt variations per task; if win rates collapse to zero, the measured abilities reside in the human's strategy rather than in the LLM planner. A complementary check is to keep the human prompts but replace the textual map description with the raw map image and verify the reported drop in wins.","tokens_in":33095,"feed_emoji":"🎮","tokens_out":8551,"duration_ms":76456,"temperature":0.7,"pith_summary":"This paper introduces HIVE, a framework in which one human commands a large swarm of game units by chatting with a large language model. The LLM turns high-level instructions into a structured plan that assigns each unit a target position and a handcrafted behavior tree, so low-level movement and combat run without further LLM involvement. Alongside the framework, the paper contributes a five-task real-time strategy benchmark that tests coordination, exploitation of unit weaknesses, marker following, terrain use, and strategic point defense. On this benchmark, nine LLMs are compared; the best-performing model solves every task for at least one prompt variation, while a small open-weight model fails almost entirely. The results support the claim that an LLM can act as an operational planner for short-horizon swarm tasks when a human supplies the strategy, and they expose current weaknesses in visual-spatial map reading, prompt sensitivity, and long-term planning.","feed_headline":"One human plus an LLM can steer 2,000 game units to a win","feed_subtitle":"HIVE turns natural-language orders into executable swarm plans, and a five-task benchmark shows where LLMs still fail.","key_machinery":"The central mechanism is the structured plan: a list of steps, each with prerequisites, an objective, and one or more unit groups, where each group pairs a set of unit IDs with a target position and a handcrafted behavior tree. The LLM writes this plan in a constrained syntax that a parser validates, and the game engine then assigns behavior trees to units and rolls steps forward as objectives are met. Behavior trees are the low-level layer: five predefined trees (long-range attack, close-range attack, attack-and-move, move-to-target, stand) translate local observations into single actions, so the LLM never controls units tick by tick. This separation between LLM-written plans and deterministic behavior-tree execution is what makes control of thousands of units feasible.","core_discovery":"The paper's central claim is that a hybrid human-LLM pipeline can translate high-level instructions into executable control for up to 2,000 units, and that this capability is measurable with a purpose-built benchmark. Concretely, HIVE lets a player describe a tactic in natural language, then asks an LLM to emit a plan in a structured BEGIN PLAN/END PLAN syntax; the plan is organized into steps with prerequisites, objectives, unit groups, target positions, and behavior trees. The benchmark's five ability tests are all solved by at least one prompt variation with the best model, and the paper reports that removing the human's strategic prompt sharply reduces wins, that swapping the textual map description for an image degrades performance, and that slight wording changes in prompts cause large swings in outcomes. The authors conclude that LLMs are useful operational planners but still struggle with visual-spatial reasoning and long-horizon strategy.","pith_inferences":["A direct extension would be to ablate the human prompt, not just remove it: run the same tasks with prompts that state only the mission objective and measure how much of the success is the LLM's planning rather than the human's strategy.","The structured-plan design offers a controlled way to separate planning ability from execution ability, since plan validity is checked by a parser and execution is deterministic, so one could hold the LLM fixed and vary only the human input.","If the framework transfers outside the game, the human-in-the-loop structure of human strategy, LLM-written operational plans, and handcrafted low-level controllers could apply to disaster response or logistics swarms, but only for missions whose horizon is short relative to the LLM's planning depth.","Benchmarking future LLMs with the same harness would provide a direct comparison of coordination ability; a natural test is whether newer models surpass the reported success rates, as the paper's own preliminary run with a newly released model already suggests."],"forward_implications":["A single person can control up to 2,000 units through natural language in this game, and the framework scales to at least 4,000 units before hardware limits are reached.","Closed-source LLMs are currently necessary: the small open-weight model tested rarely returns a valid plan, while the best closed models win at least some of the ten prompt variations on every task.","Removing the human's strategic prompt from the loop reduces performance; in two of the four scenarios both tested models lose all of their wins, indicating that the hybrid setup, not the LLM alone, is what solves these tasks.","Providing the map as an image instead of a precomputed textual description hurts performance in the two spatial tasks, so current LLMs cannot reliably read maps that are unlikely to be well represented in their training data.","Because the benchmark uses ten near-identical prompts for each task, it doubles as a prompt-sensitivity test; all models showed large outcome variation across rewording."],"supporting_citations":[{"why":"Demonstrates that an LLM can reach human-level strategic play in a complex negotiation game, motivating the use of LLMs for high-level planning.","marker":"[3]"},{"why":"Presents the hybrid architecture of LLM-based high-level strategy with non-LLM low-level execution that HIVE builds on.","marker":"[6]"},{"why":"The established RTS benchmark whose tactical focus this paper extends toward strategic planning.","marker":"[9]"},{"why":"The procedurally generated successor that supplies the benchmark lineage and generalization target for multi-agent control.","marker":"[12]"},{"why":"The vectorized execution backend that makes real-time simulation of thousands of units possible in HIVE.","marker":"[25]"},{"why":"The behavior-tree formalism used as the handcrafted low-level action layer between LLM plans and unit actions.","marker":"[28]"},{"why":"The coordinate-scaffolding method used in the ablation comparing visual versus textual map inputs to the LLM.","marker":"[33]"}],"fun_headline_variants":["LLM turns talk into orders for 2,000 game units","HIVE lets one human command a 2,000-agent swarm by chat","Benchmark: LLMs plan tactical moves but fail at long-term strategy","One person, one LLM, 2,000 units: HIVE makes swarm control verbal","LLM swarm commander: tactical wins, strategic losses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's five tasks actually measure the five named capabilities, because the test prompts already spell out the winning tactic and the low-level execution is done by fixed handcrafted controllers, so the measured success could reflect the LLM's ability to reformat a human strategy rather than to generate strategy itself.","fun_headline_variants_meta":{"raw":{"variants":["LLM turns talk into orders for 2,000 game units","HIVE lets one human command a 2,000-agent swarm by chat","Benchmark: LLMs plan tactical moves but fail at long-term strategy","One person, one LLM, 2,000 units: HIVE makes swarm control verbal","LLM swarm commander: tactical wins, strategic losses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001401,"raw_usage":{"total_tokens":5656,"prompt_tokens":932,"completion_tokens":4724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":4627}},"tokens_in":548,"tokens_out":4724,"duration_ms":29163,"temperature":1.0,"reasoning_tokens":4627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:36:31.971214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the five ability tests with prompts that state only the mission objective instead of the tactical instructions in Appendix I, using the same ten prompt variations per task; if win rates collapse to zero, the measured abilities reside in the human's strategy rather than in the LLM planner. A complementary check is to keep the human prompts but replace the textual map description with the raw map image and verify the reported drop in wins.","supporting_citations":[],"review_version":1}