REVIEW 4 major objections 5 minor 122 references
Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Control
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A single human can coordinate up to 2,000 game units through natural-language orders that an LLM translates into executable plans.
desk verdict A useful framework-plus-benchmark paper whose headline ability claims outrun the evidence: the test prompts encode the winning strategy, so success largely reflects plan-formatting of human strategy rather than autonomous coordination. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the structured plan: a list of steps, each with prerequisites, an objective, and one or more unit groups, where each group pairs a set of unit IDs with a target position and a handcrafted behavior tree. The LLM writes this plan in a constrained syntax that a parser validates, and the game engine then assigns behavior trees to units and rolls steps forward as objectives are met. Behavior trees are the low-level layer: five predefined trees (long-range attack, close-range attack, attack-and-move, move-to-target, stand) translate local observations into single actions, so the LLM never controls units tick by tick. This separation between LLM-written plans and deterministic behavior-tree execution is what makes control of thousands of units feasible.
What would settle it
Re-run the five ability tests with prompts that state only the mission objective instead of the tactical instructions in Appendix I, using the same ten prompt variations per task; if win rates collapse to zero, the measured abilities reside in the human's strategy rather than in the LLM planner. A complementary check is to keep the human prompts but replace the textual map description with the raw map image and verify the reported drop in wins.
Extended reading notes
Core claim
The paper's central claim is that a hybrid human-LLM pipeline can translate high-level instructions into executable control for up to 2,000 units, and that this capability is measurable with a purpose-built benchmark. Concretely, HIVE lets a player describe a tactic in natural language, then asks an LLM to emit a plan in a structured BEGIN PLAN/END PLAN syntax; the plan is organized into steps with prerequisites, objectives, unit groups, target positions, and behavior trees. The benchmark's five ability tests are all solved by at least one prompt variation with the best model, and the paper reports that removing the human's strategic prompt sharply reduces wins, that swapping the textual map description for an image degrades performance, and that slight wording changes in prompts cause large swings in outcomes. The authors conclude that LLMs are useful operational planners but still struggle with visual-spatial reasoning and long-horizon strategy.
Load-bearing premise
The load-bearing premise is that the benchmark's five tasks actually measure the five named capabilities, because the test prompts already spell out the winning tactic and the low-level execution is done by fixed handcrafted controllers, so the measured success could reflect the LLM's ability to reformat a human strategy rather than to generate strategy itself.
Editorial extensions
If this is right
- A single person can control up to 2,000 units through natural language in this game, and the framework scales to at least 4,000 units before hardware limits are reached.
- Closed-source LLMs are currently necessary: the small open-weight model tested rarely returns a valid plan, while the best closed models win at least some of the ten prompt variations on every task.
- Removing the human's strategic prompt from the loop reduces performance; in two of the four scenarios both tested models lose all of their wins, indicating that the hybrid setup, not the LLM alone, is what solves these tasks.
- Providing the map as an image instead of a precomputed textual description hurts performance in the two spatial tasks, so current LLMs cannot reliably read maps that are unlikely to be well represented in their training data.
- Because the benchmark uses ten near-identical prompts for each task, it doubles as a prompt-sensitivity test; all models showed large outcome variation across rewording.
Reading between the lines
- A direct extension would be to ablate the human prompt, not just remove it: run the same tasks with prompts that state only the mission objective and measure how much of the success is the LLM's planning rather than the human's strategy.
- The structured-plan design offers a controlled way to separate planning ability from execution ability, since plan validity is checked by a parser and execution is deterministic, so one could hold the LLM fixed and vary only the human input.
- If the framework transfers outside the game, the human-in-the-loop structure of human strategy, LLM-written operational plans, and handcrafted low-level controllers could apply to disaster response or logistics swarms, but only for missions whose horizon is short relative to the LLM's planning depth.
- Benchmarking future LLMs with the same harness would provide a direct comparison of coordination ability; a natural test is whether newer models surpass the reported success rates, as the paper's own preliminary run with a newly released model already suggests.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents HIVE, a framework in which a large language model converts a high-level player instruction into a structured multi-step plan for controlling units in a real-time strategy game, together with a five-task benchmark intended to measure LLM abilities in coordination, exploitation of unit weaknesses, marker following, terrain use, and strategic planning. The plan assigns each unit a target position and one of five handcrafted behavior trees; execution is deterministic in a JAX-based simulator. Nine LLMs are evaluated with ten prompt variations per task, and additional experiments test scaling to larger unit counts, performance without a human-provided strategy, and text versus image map descriptions. The main findings are that several closed-source models can produce executable plans that win in at least some conditions, that results are highly sensitive to prompt wording, that removing the human-provided strategy sharply reduces win rates, and that textual map descriptions outperform image-based inputs.
Significance. If the paper's narrower claim is taken as the contribution--that an LLM can translate a high-level, human-specified strategy into an executable plan for hundreds to thousands of units--the work is a solid and useful demonstration. The plans are saved and can be re-executed deterministically, the environment and prompts are documented in detail, and the comparison across nine models gives a first empirical signal. The paper also reports failures honestly, including invalid plans from a small open model, sensitivity to prompt wording, and poor performance with image inputs. The broader claim that the five benchmark tasks measure distinct LLM capabilities such as strategic planning is not yet supported, because the test prompts largely dictate the winning strategy and the low-level behaviors are fixed handcrafted trees. With additional baselines and a validation of what each task measures, the benchmark could become a valuable community resource.
major comments (4)
- [IV.A, Appendix I, V.B] The ability tests do not currently measure the named capabilities. Section IV.A defines the five tests as assessing coordination, exploitation of weaknesses, marker following, terrain utilization, and strategic planning, but the prompts in Appendix I supply the winning strategy for most tasks: Coordinate prompts tell the model to cover the central row and protect archers; Exploit weakness prompts instruct splitting into three groups matched to enemy weaknesses; Exploit terrain prompts give ordered forest waypoints and say to ignore enemies; Follow markers prompts name the markers and the objective; Strategize points prompts say to split the army across all bridges with archers plus close-range units. Since Section III-B restricts the LLM to choosing among five handcrafted behavior trees, the LLM's role is largely to reformat the provided strategy into the plan grammar and assign unit slices and coordinates. The 'HIVE alone' ablation in Section V-B confirms this dependency for Coordinate and Exploit terrain, where wins drop to 0/10 for both 4o and Sonnet. The Section IV.C statement that HIVE 'effectively possesses all the proposed abilities' is therefore not supported for the named abilities; what is supported is the narrower claim that HIVE translates human-provided strategy into executable plans.
- [III-C and IV] The evaluation does not exercise the 'dialog' component of the framework. The Abstract and Section I describe a natural language dialog, and Section III-C presents an outer loop in which the player prompts the LLM and the plan is executed, but the benchmarks and ablations send exactly one pre-written prompt per episode and then execute the parsed plan; the paper explicitly states that HIVE never asks the LLM to update the plan or try again. Consequently, multi-turn clarification, plan revision, and the interactive collaboration that motivates the hybrid approach are not tested. The human-machine collaboration claims should be restricted to one-shot instruction translation, or a dialog-based evaluation should be added.
- [V-A, Fig. 7] The scaling experiment does not establish the 'up to 2,000 agents' claim as a performance result. The text says the number of units was varied from 200 to 4,000 and that the authors stopped at 4,000, but Fig. 7 shows only 100 to 2,000 units, and the text also states that 'there is no significant conclusion regarding the success rate' due to LLM variance. The scaling study should be reported with the range actually tested, and the claims should be calibrated to what the data support, presenting the result as a capacity demonstration rather than as evidence of maintained performance at scale.
- [IV.B, Table I, Fig. 5] The model-comparison results are presented without uncertainty quantification, and there is no non-LLM baseline. With n=10 binary trials per condition, win-count differences such as 2/10 versus 0/10 (Table I, Coordinate) are within binomial sampling noise, and the continuous metrics in Fig. 5 lack confidence intervals. In addition, because no template-based, scripted, or random-plan baseline is included, the benchmark cannot separate the LLM's planning contribution from the information contained in the prompt and the handcrafted behavior trees. Adding such baselines and reporting exact binomial intervals would substantially strengthen the benchmark's conclusions.
minor comments (5)
- [Appendix I] There are several typos in the Appendix I prompts ('Judisiously', 'troups', 'loose time'); if these are unintentional they should be corrected, and if they are intentional robustness variations this should be stated.
- [Fig. 1] The example text says '10 bridges in total (5 on the Northern River and 4 on the Southern River)' and the caption says 'ten bridges instead of nine,' but 5 + 4 = 9; the bridge count should be made consistent.
- [Appendix E1] The system prompt says 'one unit can belong to two groups' while Section III-D1 and the later IMPORTANT note require each unit to belong to no more than one group per step; the instruction text should be aligned.
- [V-A] Section V-A states that units were varied from 200 to 4,000 and that the authors stopped at 4,000, but Fig. 7's x-axis ends at 2,000; the text and figure should be reconciled.
- [II] The Related Work section would benefit from a sentence explicitly contrasting HIVE's single-shot plan generation with Cicero's multi-turn negotiation, since the dialog claim in the Introduction is otherwise hard to position.
Circularity Check
No circularity: the hybrid HIVE claim is directly tested by plan execution, and the paper explicitly reports the human-dependence of its results.
full rationale
The paper contains no fitted parameters, no predictive equations whose outputs are reinserted as inputs, and no load-bearing self-citations. The central claim is that HIVE translates high-level human instructions into executable plans for large agent swarms; the benchmark tests exactly this, since each ability-test prompt is the human strategic input and the LLM must emit a structured plan that is then executed in a deterministic game. Win/loss outcomes are exogenous results of execution rather than quantities defined by the prompts. The 'HIVE alone' ablation (Sec. V-B) explicitly shows that without the human-strategy prompts, wins drop or disappear (e.g., Coordinate: Sonnet 2/10 with human strategy vs 0/10 alone; Exploit terrain: 4/10 vs 0/10), which is consistent with the paper's hybrid claim rather than a hidden tautology. Appendix I prompts do encode the strategy for tests such as Strategize points, but the paper's own stated finding is 'combining human strategic input with LLM-powered operational planning' (Sec. VI), not autonomous LLM strategy generation; the stronger interpretation that LLMs devise the strategy unaided is a construct-validity caveat, not a circular derivation. No circular step is identifiable in the paper's argument.
Assumptions & free parameters
free parameters (1)
- Per-scenario time limits =
300 or 500 game steps
assumptions (4)
- domain assumption The game simulation and behavior trees faithfully represent the intended RTS coordination task.
- ad hoc to paper The ten prompt variations per ability test are semantically equivalent enough to measure sensitivity to wording.
- ad hoc to paper The five ability tests measure the named capabilities rather than the LLM's ability to follow explicit instructions in the prompt.
- domain assumption Textual map descriptions provide sufficient and unambiguous spatial information for planning.
Cite this review
Pith. "Pith review of Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Control." pith.science (2026). https://pith.science/paper/R5ZL7VRU
@misc{pith2026241211761,
author = {Pith},
title = {Pith review of: Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Control},
year = {2026},
howpublished = {\url{https://pith.science/paper/R5ZL7VRU}},
note = {Machine review of arXiv:2412.11761}
}
read the original abstract
Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. Their potential to facilitate human coordination with many agents is a promising but largely under-explored area. Such capabilities would be helpful in disaster response, urban planning, and real-time strategy scenarios. In this work, we introduce (1) a real-time strategy game benchmark designed to evaluate these abilities and (2) a novel framework we term HIVE. HIVE empowers a single human to coordinate swarms of up to 2,000 agents through a natural language dialog with an LLM. We present promising results on this multi-agent benchmark, with our hybrid approach solving tasks such as coordinating agent movements, exploiting unit weaknesses, leveraging human annotations, and understanding terrain and strategic points. Our findings also highlight critical limitations of current models, including difficulties in processing spatial visual information and challenges in formulating long-term strategic plans. This work sheds light on the potential and limitations of LLMs in human-swarm coordination, paving the way for future research in this area. The HIVE project page, hive.syrkis.com, includes videos of the system in action.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Investigating Agency of LLMs in Human-AI Collaboration Tasks,
A. Sharma, S. Rao, C. Brockett, A. Malhotra, N. Jojic, and B. Dolan, “Investigating Agency of LLMs in Human-AI Collaboration Tasks,” in EACL 2024, 2024
2024
-
[2]
Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning,
A. Bakhtin, D. J. Wu, A. Lerer, J. Gray, A. P. Jacob, G. Farina, A. H. Miller, and N. Brown, “Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning,” Oct. 2022
2022
-
[3]
Human-level play in the game of Diplomacy by combining language models with strategic reasoning,
E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, A. P. Jacob, M. Komeili, K. Konath, M. Kwon, A. Lerer, M. Lewis, A. H. Miller, S. Mitts, A. Renduchintala, S. Roller, D. Rowe, W. Shi, J. Spisak, A. Wei, D. Wu, H. Zhang, and M. Zijlstra, “Human-level play in the game of Diplomacy by combining language models with strategic reasoning,” S...
2022
-
[4]
Strategic behavior of large language models and the role of game structure versus contextual framing,
N. Lor `e and B. Heydari, “Strategic behavior of large language models and the role of game structure versus contextual framing,” Scientific Reports, vol. 14, no. 1, p. 18490, Aug. 2024
2024
-
[5]
LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models,
Y . Zhang, S. Mao, T. Ge, X. Wang, A. de Wynter, Y . Xia, W. Wu, T. Song, M. Lan, and F. Wei, “LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models,” 2024
2024
-
[6]
SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models,
X. Shao, W. Jiang, F. Zuo, and M. Liu, “SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models,” 2024
2024
-
[7]
Hierarchical control of multi-agent reinforcement learning team in real-time strategy (RTS) games,
W. J. Zhou, B. Subagdja, A.-H. Tan, and D. W.-S. Ong, “Hierarchical control of multi-agent reinforcement learning team in real-time strategy (RTS) games,” Expert Systems with Applications , vol. 186, 2021
2021
-
[8]
Robot behavior-tree-based task generation with large language models,
Y . Cao and C. Lee, “Robot behavior-tree-based task generation with large language models,” in CEUR workshop proceedings , vol. 3433. RWTH Aachen University, 2023
2023
Show all 122 references
-
[9]
The StarCraft Multi-Agent Challenge,
M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson, “The StarCraft Multi-Agent Challenge,” Dec. 2019
2019
-
[10]
Monotonic Value Function Factorisation for Deep Multi- Agent Reinforcement Learning,
T. Rashid, M. Samvelyan, C. S. D. Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic Value Function Factorisation for Deep Multi- Agent Reinforcement Learning,” ArXiv, Mar. 2020
2020
-
[11]
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games,
C. Yu, A. Velu, E. Vinitsky, Y . Wang, A. Bayen, and Y . Wu, “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games,” in Neural Information Processing Systems , Mar. 2021
2021
-
[12]
SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning,
B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson, “SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning,” Oct. 2023
2023
-
[13]
JaxMARL: Multi- Agent RL Environments in JAX,
A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvars- son, T. Willi, A. Khan, C. S. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan, M. Jiang, R. T. Lange, S. Whiteson, B. Lacerda, N. Hawes, T. Rocktaschel, C. Lu, and J. N. Foerster, “JaxMARL: Multi- Agent RL En...
2023
-
[14]
Large Language Model based Multi-Agents: A Survey of Progress and Challenges,
T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large Language Model based Multi-Agents: A Survey of Progress and Challenges,” Apr. 2024
2024
-
[15]
AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration,
B. Pan, J. Lu, K. Wang, L. Zheng, Z. Wen, Y . Feng, M. Zhu, and W. Chen, “AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration,” Apr. 2024
2024
-
[16]
LLM Multi-Agent Systems: Challenges and Open Problems,
S. Han, Q. Zhang, Y . Yao, W. Jin, Z. Xu, and C. He, “LLM Multi-Agent Systems: Challenges and Open Problems,” 2024
2024
-
[17]
Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy,
J. Kram ´ar, T. Eccles, I. Gemp, A. Tacchetti, K. R. McKee, M. Ma- linowski, T. Graepel, and Y . Bachrach, “Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy,” Nature Communications, vol. 13, no. 1, p. 7214, Dec. 2022
2022
-
[18]
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games,
D. Paglieri, B. Cupiał, S. Coward, U. Piterbarg, M. Wolczyk, A. Khan, E. Pignatelli, Ł. Kuci ´nski, L. Pinto, R. Fergus, J. N. Foerster, J. Parker- Holder, and T. Rockt ¨aschel, “BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games,” Nov. 2024
2024
-
[19]
Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations,
A. Ruoss, F. Pardo, H. Chan, B. Li, V . Mnih, and T. Genewein, “Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations,” arXiv preprint arXiv:2412.01441 , 2024
2024 arXiv
-
[20]
A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment,
M. G. Mecattaf, B. Slater, M. Te ˇsi´c, J. Prunty, K. V oudouris, and L. G. Cheke, “A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment,” 2024
2024
-
[21]
CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long- Range Reasoning Problems using LLMs,
K. Wang, X. Zhang, H. Liu, S. Han, H. Ma, and T. Hu, “CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long- Range Reasoning Problems using LLMs,” 2024
2024
-
[22]
Under- standing the Weakness of Large Language Model Agents within a Com- plex Android Environment,
M. Xing, R. Zhang, H. Xue, Q. Chen, F. Yang, and Z. Xiao, “Under- standing the Weakness of Large Language Model Agents within a Com- plex Android Environment,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . Barcelona Spain: ACM, Aug. ...
2024
-
[23]
Towards Vision-Language Geo-Foundation Model: A Survey,
Y . Zhou, L. Feng, Y . Ke, X. Jiang, J. Yan, X. Yang, and W. Zhang, “Towards Vision-Language Geo-Foundation Model: A Survey,” 2024
2024
-
[24]
Vision language models are blind,
P. Rahmanzadehgervi, L. Bolton, M. R. Taesiri, and A. T. Nguyen, “Vision language models are blind,” in Computer Vision – ACCV 2024 , M. Cho, I. Laptev, D. Tran, A. Yao, and H. Zha, Eds. Singapore: Springer Nature Singapore, 2025, pp. 293–309
2024
-
[25]
JAX: composable transformations of Python+NumPy pro- grams,
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclau- rin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, “JAX: composable transformations of Python+NumPy pro- grams,” 2018
2018
-
[26]
(2025) Inkarnate – An online map-making platform
Inkarnate. (2025) Inkarnate – An online map-making platform. [Online]. Available: https://inkarnate.com
2025
-
[27]
Theory-based approach for assessing cognitive load during time- critical resource-managing human–computer interactions: An eye- tracking study,
N. Sevcenko, T. Appel, M. Ninaus, K. Moeller, and P. Gerjets, “Theory-based approach for assessing cognitive load during time- critical resource-managing human–computer interactions: An eye- tracking study,” Journal on Multimodal User Interfaces , vol. 17, 2023
2023
-
[28]
Colledanchise and P
M. Colledanchise and P. ¨Ogren, Behavior trees in robotics and AI: An introduction. CRC Press, 2018
2018
-
[29]
Gpt-4 technical report,
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
2023 arXiv
-
[30]
The claude 3 model family: Opus, sonnet, haiku,
Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” 2024
2024
-
[31]
Gemini: a family of highly capable multimodal models,
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al. , “Gemini: a family of highly capable multimodal models,” arXiv:2312.11805, 2023
2023 arXiv
-
[32]
The llama 3 herd of models,
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024
2024 arXiv
-
[33]
Scaffolding coordinates to promote vision-language coordination in large multi-modal models,
X. Lei, Z. Yang, X. Chen, P. Li, and Y . Liu, “Scaffolding coordinates to promote vision-language coordination in large multi-modal models,” arXiv preprint arXiv:2402.12058 , 2024
2024 arXiv
-
[34]
A survey on llm-based code generation for low-resource and domain-specific programming languages,
S. Joel, J. J. Wu, and F. H. Fard, “A survey on llm-based code generation for low-resource and domain-specific programming languages,” arXiv preprint arXiv:2410.03981, 2024
2024
-
[35]
Evaluating Causal Reasoning Ca- pabilities of Large Language Models: A Systematic Analysis Across Three Scenarios,
L. . Wang, Y . . . S. o. S. E. Shen, and L. Guangzhou Intelligence Communications Technology Co., “Evaluating Causal Reasoning Ca- pabilities of Large Language Models: A Systematic Analysis Across Three Scenarios,” p. 4584
-
[36]
Evaluating Mathematical Problem-Solving Abilities of Genera- tive AI Models: Performance Analysis of o1-preview and gpt-4o Using the Korean College Scholastic Ability Test,
S. Oh, “Evaluating Mathematical Problem-Solving Abilities of Genera- tive AI Models: Performance Analysis of o1-preview and gpt-4o Using the Korean College Scholastic Ability Test,” vol. 13, pp. 1227–1235
-
[37]
Review of automatic text summarization techniques & methods,
A. P. Widyassari, S. Rustad, G. F. Shidik, E. Noersasongko, A. Syukur, A. Affandy et al., “Review of automatic text summarization techniques & methods,” Journal of King Saud University-Computer and Information Sciences, vol. 34, no. 4, pp. 1029–1046, 2022
2022
-
[38]
An empirical study of the non-determinism of chatgpt in code generation,
S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” vol. 34, 2025
2025
-
[39]
V oyager: An open-ended embodied agent with large language models,
G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291 , 2023
2023 arXiv
-
[40]
Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc,
H. Liu, X. Zhang, H. Xu, Y . Wanyan, J. Wang, M. Yan, J. Zhang, C. Yuan, C. Xu, W. Hu et al. , “Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc,” arXiv preprint arXiv:2502.14282, 2025
2025 arXiv
-
[41]
Geogpt: An assistant for understanding and processing geospatial tasks,
Y . Zhang, C. Wei, Z. He, and W. Yu, “Geogpt: An assistant for understanding and processing geospatial tasks,” International Journal of Applied Earth Observation and Geoinformation , 2024
2024
-
[42]
Are large vision language models good game players?
X. Wang, B. Zhuang, and Q. Wu, “Are large vision language models good game players?” in The Thirteenth International Conference on Learning Representations, 2025. 11 APPENDIX A. Inference time analyses 4o 4o-minio1-miniSonnetHaiku Gemini 2Gemini-proGemini-flashLlama3-8B 0 50 1...
2025
-
[43]
BEGIN PLAN
Instruction: # Map Instruction You are a game assistant that helps the player in a strategy video game.,→ 12 TABLE II CHARACTERISTICS OF THE THREE TYPES OF UNITS USED IN THIS PAPER . Spearman Archer Cavalry Visual shape square circle triangle Speed (unit distance/step) 1 2 6 M...
-
[44]
The game info: Health and positions of all the units of each team (/ 0means that the unit is dead). ,→ ,→ Allies: Health: [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24...
-
[45]
Coordination with Sonnet: I'll help you create a plan to effectively eliminate the enemies. Looking at the positions, I notice that all enemy units are currently in the Northern Forest (y-coordinates between 135-149), and they will need to move south through the middle of the ...
-
[46]
Western group (at x=25)
-
[47]
Central group (at x=75)
-
[48]
,→ ,→ ,→ ,→ The groups are spread out to cover the entire width of the map, ensuring no enemies can slip through
Eastern group (at x=125) Each group has: - About 167 spearmen in front (at y=75) - About 167 archers behind them (at y=65) The spearmen are positioned 10 units ahead of the archers to provide protection while still allowing the archers to shoot over them at approaching enemies...
-
[49]
,→ ,→ ,→ Enemy forces are organized in three distinct groups:,→
Exploit weaknesses with Sonnet: Let me analyze the situation and the enemy positions.,→ The map is divided into 4 quadrants by two rivers that cross at (46-52, 47-53), with bridges allowing passage between these quadrants. ,→ ,→ ,→ Enemy forces are organized in three distinct ...
-
[50]
Northwest quadrant (around x:5-25, y:75-95): Mainly archers (health=2),→
-
[51]
Northeast quadrant (around x:75-95, y:5-25): Mainly spearmen (health=24),→
-
[52]
,→ ,→ Strategic considerations:
Northeast quadrant (around x:75-95, y:75-95): Mainly cavalry (health=12),→ Our forces are currently positioned in the Southwest quadrant (around x:5-40, y:5-40). ,→ ,→ Strategic considerations:
-
[53]
We should use cavalry against archers (NW group),→
-
[54]
Archers against spearmen (NE group)
-
[55]
Spearmen against cavalry (NE group) Here's my proposed plan: BEGIN PLAN Step 0: prerequisites: [] objective: position units: [0:250] - target position: (19, 49) - behavior: attack_and_move any units: [250:500] - target position: (21, 49) - behavior: attack_and_move any units: ...
-
[56]
First, we move all forces to the west bridge (Step 0),→
-
[57]
Then split our forces (Step 1): - Cavalry (500:750) goes north to engage archers,→ - Archers (250:500) goes east to engage spearmen,→ - Spearmen (0:250) goes northeast to engage cavalry,→
-
[58]
,→ ,→ ,→ ,→
Finally, each group moves to eliminate their targeted enemies (Step 2),→ This plan leverages each unit type's strengths against their natural enemies while using the bridges strategically to reach their positions. ,→ ,→ ,→ ,→
-
[59]
Markers with 4o-mini: To achieve your goal of moving the troops to the markers A, B, C, and D while ignoring enemies, we can create a plan that sequentially moves the units to each marker and then to the objective position at (61, 0). ,→ ,→ ,→ ,→ ,→ However, since the objectiv...
-
[60]
Move all units to marker A at (193, 85).,→
-
[61]
Move all units to marker B at (49, 136).,→
-
[62]
Move all units to marker C at (9, 134)
-
[63]
Move all units to marker D at (11, 9)
-
[64]
Would you like to make any adjustments or proceed with this plan? ,→ ,→ ,→ ,→
Finally, move all units to the objective position at (61, 0).,→ I will now create the detailed plan based on this sequence.,→ BEGIN PLAN Step 0: prerequisites: [] objective: position units: all - target position: (193, 85) - behavior: follow_map Step 1: prerequisites: [0] obje...
-
[65]
Terrain with 4o: Here is the detailed plan based on your instructions:,→ BEGIN PLAN Step 0: prerequisites: [] objective: position units: all - target position: (164, 71) - behavior: follow_map any Step 1: prerequisites: [0] objective: position units: all - target position: (33...
-
[66]
::" node |
Strategic points with o1-mini: BEGIN PLAN Step 0: prerequisites: [] objective: position units: [0:39] - target position: (45, 252) - behavior: attack_in_close_range any units: [350:389] - target position: (45, 252) - behavior: attack_in_long_range any units: [39:78] - target p...
-
[67]
Available for HIVE: Long-range attack: F(S(C( in_reach foe me_from_them high any) :: A (move away_from closest foe any)) :: A (attack random any) :: A (follow_map toward)) ,→ ,→ ,→ Close-range attack: F( A (attack random any) :: A (move toward closest foe any) :: A (follow_map...
-
[68]
Only present in the opposing units: The two following behavior trees are used by the opposing side in the Markers and Terrain ability test to avoid the enemy unit to get lost in the trees where they cannot see the player’s units any longer. Attack in long range and stay out of...
-
[69]
Judisiously place our long range units so that they are not in close combat
Make a plan that forms as many squads as you think necessary to cover the central row of the battlefield to eliminate all the enemies as fast as possible. Judisiously place our long range units so that they are not in close combat. ,→ ,→ ,→ ,→ ,→ ,→
-
[70]
Judisiously place our long range units so that they protected by our close range units
Design a plan that forms as many group of units as you think necessary to cover the central row of the battlefield to eliminate all the enemies as quickly as possible. Judisiously place our long range units so that they protected by our close range units. ,→ ,→ ,→ ,→ ,→ ,→ ,→ 22
-
[71]
Ensure to place our long range units so that they are not in close combat
Write down a plan that forms as many group of units as you think necessary to cover the central row of the map to eliminate all the enemies as fast as possible. Ensure to place our long range units so that they are not in close combat. ,→ ,→ ,→ ,→ ,→ ,→
-
[72]
Make sure to place our long range units so that they in far range of the enemies
Design a plan that forms as many group of units as you think necessary to cover the central row of the map to eliminate all the enemies as quickly as possible. Make sure to place our long range units so that they in far range of the enemies. ,→ ,→ ,→ ,→ ,→ ,→
-
[73]
Ensure to place our archers so that they protected by our close range units
Make a plan that forms as many squads as you think necessary to cover the middle row of the map to eliminate all the enemies as quickly as possible. Ensure to place our archers so that they protected by our close range units. ,→ ,→ ,→ ,→ ,→ ,→
-
[74]
Ensure to place our archers so that they in far range of the enemies
Design a plan that forms as many battalions as you think necessary to cover the middle row of the battlefield to eliminate all the enemies as fast as possible. Ensure to place our archers so that they in far range of the enemies. ,→ ,→ ,→ ,→ ,→ ,→
-
[75]
Make sure to place our archers so that they are not in close combat
Form as many battalions as you think necessary to cover the central row of the battlefield to eliminate all the enemies as fast as possible. Make sure to place our archers so that they are not in close combat. ,→ ,→ ,→ ,→ ,→
-
[76]
Make sure to place our long range units so that they protected by our close range units
Write down a plan that forms as many battalions as you think necessary to cover the central row of the battlefield to eliminate all the enemies as fast as possible. Make sure to place our long range units so that they protected by our close range units. ,→ ,→ ,→ ,→ ,→ ,→ ,→
-
[77]
Judisiously place our archers so that they in far range of the enemies
Make a plan that forms as many squads as you think necessary to cover the middle row of the battlefield to eliminate all the enemies as quickly as possible. Judisiously place our archers so that they in far range of the enemies. ,→ ,→ ,→ ,→ ,→ ,→
-
[78]
Make sure to place our archers so that they in far range of the enemies
Make a plan that forms as many group of units as you think necessary to cover the middle row of the battlefield to eliminate all the enemies as quickly as possible. Make sure to place our archers so that they in far range of the enemies. ,→ ,→ ,→ ,→ ,→ ,→
-
[79]
(b) Exploit weakness:
-
[80]
First, compute the positions and types of the enemy battalions. Then, design a plan that play on their weaknesses to split our units into three groups with specific strengths and send each one against one of the enemy squads to minimize our casualties. ,→ ,→ ,→ ,→ ,→ ,→
-
[81]
First, determine the positions and types of the enemy battalions. Then, make a plan that play on their weaknesses to split our army into three battalions with specific strengths and send each one against one of the enemy groups to maximize efficiency. ,→ ,→ ,→ ,→ ,→ ,→ ,→
-
[82]
First, analyse the situation to determine the positions and types of the enemy squads. Then, write a plan that play on their weaknesses to split our army into three battalions with specific strengths and send each one against one of the enemy battalions to minimize our casualt...
-
[83]
First, analyse the situation to determine the positions and types of the enemy groups. Then, make a plan that play on their weaknesses to split our units into three squads with specific strengths and send each one against one of the enemy battalions to minimize our casualties....
-
[84]
First, analyse the situation to compute the positions and types of the enemy squads. Then, design a plan that play on their weaknesses to split our army into three groups with specific strengths and send each one against one of the enemy battalions to minimize our casualties. ...
-
[85]
First, analyse the situation to compute the positions and types of the enemy groups. Then, design a plan that play on their weaknesses to split our army into three squads with specific strengths and send each one against one of the enemy groups to maximize efficiency. ,→ ,→ ,→...
-
[86]
First, analyse the situation to determine the positions and types of the enemy groups. Then, make a plan that play on their weaknesses to split our army into three squads with specific strengths and send each one against one of the enemy groups to minimize our casualties. ,→ ,...
-
[87]
First, analyse the situation to compute the positions and types of the enemy squads. Then, write a plan that play on their weaknesses to split our army into three groups with specific strengths and send each one against one of the enemy squads to maximize efficiency. ,→ ,→ ,→ ...
-
[88]
First, compute the positions and types of the enemy squads. Then, make a plan that play on their weaknesses to split our army into three squads with specific strengths and send each one against one of the enemy battalions to maximize efficiency. ,→ ,→ ,→ ,→ ,→ ,→
-
[89]
First, compute the positions and types of the enemy battalions. Then, design a plan that play on their weaknesses to split our army into three squads with specific strengths and send each one against one of the enemy groups to minimize our casualties. ,→ ,→ ,→ ,→ ,→ ,→
-
[90]
(c) Follow markers: The player mark four positions: • A: (193, 85); • B: (49, 136); • C: (9, 134); • D: (11, 9)
-
[91]
,→ ,→ ,→
Move the spearmen to each position in alphabetical order while ignoring enemies and then to the final objective position. ,→ ,→ ,→
-
[92]
,→ ,→ ,→
Write a plan to move our troups to each marker while ignoring enemies and then to the final objective position. ,→ ,→ ,→
-
[93]
,→ ,→ ,→
Design a plan to move our spearmen to each marker in alphabetical order while ignoring enemies and then to the final objective position. ,→ ,→ ,→
-
[94]
,→ ,→ ,→
Make a plan to move our spearmen to each position in alphabetical order while ignoring enemies and then to the final objective position. ,→ ,→ ,→
-
[95]
Write a plan to move our army to each position while ignoring enemies and then to the final objective position. ,→ ,→
-
[96]
,→ ,→ ,→
Write a plan to move my troups to the positions A, B, C, D while ignoring enemies and then to the objective position. ,→ ,→ ,→
-
[97]
Move our troups to the markers A, B, C, D while ignoring enemies and then to the objective position. ,→ ,→
-
[98]
,→ ,→ ,→
Make a plan to move our army to the markers A, B, C, D while ignoring enemies and then to the objective position. ,→ ,→ ,→
-
[99]
Move our spearmen to the positions ABCD while ignoring enemies and then to the objective position. ,→ ,→
-
[100]
,→ ,→ ,→
Make a plan to move the units to the markers A, B, C, D while ignoring enemies and then to the final objective position. ,→ ,→ ,→
-
[101]
(d) Exploit terrain:
-
[102]
Then make them hide deep inside the forest on the west of the map
Make a plan to move my spearmen inside the forest on the right to the bridge, then make them rush to the forest on the west, slightly on top of the bridge . Then make them hide deep inside the forest on the west of the map. Finally, make them rush south to the left-bottom corn...
-
[103]
Then make them hide inside the trees on the west of the map
Write a plan to move my units inside the trees on the east to the bridge, then make them rush to the forest on the west, slightly on top of the bridge . Then make them hide inside the trees on the west of the map. Finally, make them rush down to the south-east corner of the ma...
-
[104]
Then make them hide deep inside the forest on the west most part of the map
Write a plan to move my spearmen inside the trees on the right to the bridge, then make them rush to the forest on the west, a bit north of the bridge as it is closer. Then make them hide deep inside the forest on the west most part of the map. Finally, make them rush to the s...
-
[105]
Then make them hide inside the trees on the west of the map
Make a plan to move our units inside the forest on the east not too far to the bridge, then make them rush to the trees on the west, on top of the bridge to quickly regain cover. Then make them hide inside the trees on the west of the map. Finally, make them rush south to the ...
-
[106]
Then make them hide deep inside the forest on the left of the map
Design a plan to move our spearmen inside the trees on the east close to the bridge, then make them run to the forest on the left, on top of the bridge to quickly regain cover. Then make them hide deep inside the forest on the left of the map. Finally, make them rush south to ...
-
[107]
Then make them hide deep inside the forest on the west most part of the map
Write a plan to move our army inside the forest on the right to the bridge, then make them move to the forest on the west, on top of the bridge as it is closer. Then make them hide deep inside the forest on the west most part of the map. Finally, make them rush to the south-ea...
-
[108]
Then make them hide deep inside the forest on the west most part of the map
Write a plan to move my army inside the forest on the east not too far to the bridge, then make them rush to the forest on the west, north of the bridge to quickly regain cover. Then make them hide deep inside the forest on the west most part of the map. Finally, make them rus...
-
[109]
Then make them hide inside the trees on the left of the map
Move the spearmen inside the trees on the east close to the bridge, then make them rush to the forest on the west, a bit on top of the bridge to quickly regain cover. Then make them hide inside the trees on the left of the map. Finally, make them rush to the south-east corner ...
-
[110]
Then make them hide deep inside the trees on the west most part of the map
Design a plan to move our spearmen inside the forest on the east close to the bridge, then make them move to the forest on the west, slightly on top of the bridge . Then make them hide deep inside the trees on the west most part of the map. Finally, make them rush to the left-...
-
[111]
Then make them hide inside the trees on the left most part of the map
Make a plan to move the army inside the forest on the right not too far to the bridge, then make them run to the trees on the west, north of the bridge . Then make them hide inside the trees on the left most part of the map. Finally, make them rush to the south-east corner of ...
-
[112]
(e) Strategize points:
-
[113]
,→ ,→ ,→
Split our units on all the bridges to defend them and make sure to send archers and spearmen units on each one. ,→ ,→ ,→
-
[114]
,→ ,→ ,→
Make a plan to split our army on all the bridges to guard them and make sure to send long-range and spearmen units on each one. ,→ ,→ ,→
-
[115]
,→ ,→ ,→
Design a plan to split our units on all the bridges to protect them and make sure to send long-range and spearmen units on each bridge. ,→ ,→ ,→
-
[116]
,→ ,→ ,→
Make a plan to split our units on all the bridges to guard them and make sure to send long-range and close-range units on each one. ,→ ,→ ,→
-
[117]
,→ ,→ ,→
Write a plan to split our army on all the bridges to protect them and make sure to send archers and close-range units on each bridge. ,→ ,→ ,→
-
[118]
,→ ,→ ,→
Make a plan to split our units on all the bridges to protect them and make sure to send archers and spearmen units on each one. ,→ ,→ ,→
-
[119]
,→ ,→ ,→
Split our army on all the bridges to guard them and make sure to send long-range and close-range units on each one. ,→ ,→ ,→
-
[120]
,→ ,→ ,→
Split our army on all the bridges to protect them and make sure to send archers and spearmen units on each one. ,→ ,→ ,→
-
[121]
,→ ,→ ,→
Design a plan to split our units on all the bridges to defend them and make sure to send archers and close-range units on each bridge. ,→ ,→ ,→
-
[122]
,→ ,→ ,→ 25 J
Make a plan to split our army on all the bridges to guard them and make sure to send archers and close-range units on each bridge. ,→ ,→ ,→ 25 J. Prompts used for HIVE alone 'First, study the situation to find a good approach to achieve this mission then design the correspondi...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.