Pok\'eAI: A Goal-Generating, Battle-Optimizing Multi-agent System for Pokemon Red

Siwen Wang; Xinhang Sui; Yueran Song; Zihao Liu

REVIEW 3 major objections 5 minor 15 references

Pok\'eAI: A Goal-Generating, Battle-Optimizing Multi-agent System for Pokemon Red

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully text-based multi-agent LLM system plays Pokemon Red battles at an 80.8% win rate, within 5.2 points of an experienced human.

desk verdict Useful empirical battle module, but the undefined win metric makes the human comparison unreliable, and the 'first multi-agent framework' framing outruns what is actually built. read the letter →

arxiv 2506.23689 v1 pith:2A3NOSXH submitted 2025-06-30 cs.AI cs.MA

Zihao Liu , Xinhang Sui , Yueran Song , Siwen Wang This is my paper

classification cs.AIcs.MA

keywords gameAIlargelanguagemodelsmulti-agentsystemsPokemonRedbattlestrategytext-basedgameplayLLMplaystyleslong-termmemory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces PokeAI, a three-agent system—Planning, Execution, and Critique—that plays Pokemon Red entirely through text. As a first self-contained component, it built and tested a battle module: reading game memory, prompting a large language model, and executing its JSON moves. In 50 wild encounters at Mt. Moon, the module won 80.8% of battles, compared with 86% for one experienced human player. The paper also reports that battle win rates track general language-benchmark scores and that different LLMs show distinct battle styles. A pilot test suggests injected long-term memories can steer later decisions, such as fleeing a losing matchup.

What carries the argument

The central object is the closed-loop battle module inside the Execution Agent: it monitors memory address 0xD057, reads battle information, sends it to the LLM, receives a JSON action, and performs that action until the battle ends. This sits inside a three-agent system—Planning decomposes goals into tasks, Execution invokes a toolkit through function calling, and Critique verifies task completion—with each agent owning a vector memory bank. The ablation method (disabling switching, escape, or items) is what isolates which capabilities carry the win rate.

What would settle it

Run the same 50-battle Mt. Moon checkpoint with at least ten human players and compute the distribution of their win rates; if the human average falls at or below the agent's 80.8%, the near-human claim collapses.

Watch

Extended reading notes

Core claim

The paper claims that a fully text-based multi-agent LLM architecture can autonomously play Pokemon Red battles at near-human level. The battle loop—detect battle via memory address 0xD057, read state, ask the LLM, act on its response—achieved an average 80.8% win rate over 50 Mt. Moon wild encounters with a Level 15 Charmander and Pidgey and five potions. The paper claims this is only 5.2 percentage points below an experienced human player's 86% in the same scenario. It further claims that performance correlates with a model's score on a general public language ranking, and that action logs show each model develops a recognizable playstyle.

Load-bearing premise

The load-bearing premise is that one experienced player's 86% win rate fairly represents human performance in this scenario; without replication, the 'only 5.2 points lower' conclusion has no stable yardstick.

Editorial extensions

If this is right

The battle module is a self-contained component that can be integrated into the full Planning–Execution–Critique loop without further work.
Removing item use drops the win rate to 32.6%, so healing items carry most of the agent's survival capability.
Removing strategic switching drops the win rate to 58.8%, showing that choosing the right Pokemon is a major part of the agent's edge.
Battle win rates rise with a model's general language ranking, so language ability and strategic play move together in this setting.
Different LLMs produce distinct action distributions, which the paper points toward as a basis for varied, human-like NPC behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the single-human baseline means the 5.2-point gap is not a stable measurement; a multi-player study would be needed to know whether the agent truly trails humans.
Beyond the paper, the memory pilot tests only one injected episode, so the promise of long-term memory for full-game planning remains untested; a natural next experiment is the whole 50-battle suite with memory on and off.
Beyond the paper, the overthinking observation suggests reasoning-heavy backends may hurt low-level game control; this could be tested by running deep-reasoning and fast-response models on the same battles.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Bottom line up front: the paper has an honest, measurable battle module with real experiments, but the headline '80.8% win rate vs human 86%' is on shaky ground because 'win' is never defined. If successful escapes count as wins, the two numbers aren't measuring the same thing—the human played to defeat, not flee.

What's actually new: a text-only LLM battle module for Pokémon Red that reads memory addresses and outputs JSON actions. Several prior works use vision-language models on the same game; this is the first text-only one I've seen with results. The experimental setup is concrete: a fixed checkpoint, 10 runs of 50 encounters, encounter rates, and standard error bars on the ablation and LLM comparisons. The playstyle analysis (GPT-4o aggressive, DeepSeek-V3 resource-preserving, Claude switching once) is qualitative but plausible and could be useful to others building such agents. Code is promised on GitHub.

The soft spots, in order of severity. First, the win metric. The abstract says 'wins' but the human comparison is 'defeated 43 wild Pokémon'. The ablation includes escape as a component, and the text says the full model sometimes 'chose to flee unnecessarily'. That means a successful flee likely counts as a 'win' in the AI's tally. Without a clear definition and separate logs for escapes, the 'only 6% lower than an experienced human' claim is unsubstantiated. This is the load-bearing result, so it needs fixing before I'd trust the number. Second, the human baseline is a single run—no replication, no error bar. Third, the correlation with LLM Arena scores is read off a bar chart without coefficient or significance. Fourth, the abstract and introduction claim 'the first fully text-based, multi-agent LLM framework', but only the battle module is implemented; the Planning and Critique agents are future work. The body is honest about this, but the framing overreaches.

None of these are fatal—they're fixable in a revision. The central observation, that LLM battle performance roughly tracks language-task ability, is interesting but not yet a strong claim.

This is a workshop-to-short-paper level contribution. I'd bring it to a reading group for the metric blunder and the LLM playstyle discussion, but I wouldn't cite it yet. It deserves a serious referee—there's enough experimental substance—but only after the authors define 'win', log escape outcomes, and scale back the 'first' claim.

Referee Report

3 major / 5 minor

Summary. The paper introduces PokéAI, a three-agent (Planning, Execution, Critique) LLM system for playing Pokémon Red, implemented as a text-only agent loop with vector memory and function calling. The present manuscript focuses on a preliminary battle module inside the Execution Agent. The module reads memory addresses (e.g., 0xD057) to detect battles, sends state to an LLM, parses a JSON action, and loops until battle end. Using a Mt. Moon checkpoint (Level 15 Charmander and Pidgey, five potions), the authors report an 80.8% average win rate across 10 repetitions of 50 wild encounters with DeepSeek-V3, compare this with an experienced human player who defeated 43 of 50 encounters (86%), run ablations over switching, item use, and escape, compare several LLM backends, analyze action distributions by model, and pilot a long-term-memory mechanism with Letta.

Significance. The paper's value is as an existence proof: a purely text-based LLM loop can control a nontrivial game subtask, and the public code release supports reproducibility. The battle-module design is simple and the ablation idea is sensible. The play-style observations (e.g., Claude 3.5's single early switch, GPT-4o's aggressive play) are concrete and falsifiable in principle. The quantitative headline, however, is not yet established: 'win' is undefined, escape may be counted as a win, the human baseline is one run, and the claimed LLM Arena correlation has no reported statistic. These issues are fixable with additional logging, re-analysis, and replication, so the work is better treated as a promising preliminary report than a settled result.

major comments (3)

[Section III-A and III-B] The central metric is ambiguous: 'win' is never defined, and the human comparison is explicitly 'managed to defeat 43 wild Pokémon' (86%), while the agent's 'wins an average of 40.4 out of 50 battles' may include successful escapes. Section III-B strengthens this concern by listing escape as a component and noting that the full model 'chose to flee unnecessarily'; if those escapes are counted as wins, the AI's 80.8% and the human's 86% are not measuring the same outcome. Please report the per-run counts of opponent KO, escape, own faint, and early run termination, and recompute the win rate under a defeat-only definition, or justify a definition of 'win' that treats fleeing as success.
[Section III-A] The human baseline consists of one run by one player (43 wins out of 50). The abstract's claim that the battle AI is 'only 6% lower' than human performance rests on a 5.2 percentage point difference between a 10-run mean and a single observation, with no human error bar and no statement of the AI run-to-run spread in the text (error bars appear in figures but no numeric CI). Please report the AI distribution, conduct additional human runs or at least provide a confidence interval, and clarify whether the human played exactly 50 encounters and how the denominator of 50 was obtained.
[Section III-C / Abstract] The abstract states that battle performance 'correlates strongly with its LLM Arena score', but Section III-C only asserts proportionality 'with the exception of Claude 3.5 Sonnet' and reports no correlation coefficient, significance test, or point count. Given the small number of models and acknowledged outlier, the strong correlation claim is unsupported. Please report the statistic (e.g., Spearman rho with confidence interval) or downgrade the claim to a qualitative observation.

minor comments (5)

[Section III-B] The text says 'As shown in Fig.4' for the ablation study, but the ablation figure is captioned as Fig. 3; figure cross-references should be corrected.
[Section III-C] The 'LLM Arena Score' is never defined or cited; please state which leaderboard and snapshot is used.
[Section III-E] The long-term-memory pilot consists of a single injected-memory scenario (Level 6 Squirtle vs Level 9 Pikachu) in which the agent flees; the sentence 'improve strategic decision-making and overall battle performance' overstates what one anecdote can show and should be labeled as preliminary.
[Section III-D] The citation [13] for GPT-4o's 'creative rule-bending' is to an instruction-following paper; reference [14] on in-context scheming is more directly relevant and should be cited here.
[Abstract / Section I] The claim of being 'the first text-based ... open-source agent framework' for Pokémon Red is not supported by a literature comparison beyond the multimodal examples given; please either substantiate the 'first' claim or weaken it, and clarify the repository's license.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all headline results are direct game-environment measurements or external-benchmark comparisons, with no fitted parameter renamed as a prediction.

full rationale

The paper makes no mathematical derivation and fits no model parameters to the reported outcomes. The central result—an 80.8% win rate across 50 wild encounters—is an empirical measurement inside the game emulator, and the comparison to an experienced human player is a separate, external measurement of the same scenario. The ablation study compares variants of the battle agent against the same measured baseline, and the LLM Arena correlation uses an independent public benchmark rather than a quantity derived from the paper's own outputs. The playstyle analysis is qualitative and does not claim to predict the win rate from a fitted quantity. The human-comparison weakness (single human run, no error bar) is a statistical robustness concern, not a circularity concern. The possible ambiguity in whether fleeing counts as a win is a metric-definition or measurement-validity issue; it does not make any claimed derivation equivalent to its inputs by construction. No self-citations are load-bearing, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The paper is self-contained as an empirical system report, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The Planning, Execution, and Critique agents are proposed system components, not invented entities in the sense of the ledger. All numbers in the evaluation are experimental design choices, not free parameters fitted to data.

assumptions (3)

domain assumption Memory address 0xD057's rightmost bit toggling from 0 to 1 signals a battle in Pokémon Red.
The entire battle module depends on this emulator memory mapping, which is not independently verified in the paper (Section II).
domain assumption The encounter rates for Mt. Moon (79% Zubat, 15% Geodude, 5% Paras, 1% Clefairy) from the cited fan site are accurate.
The evaluation's difficulty distribution rests on this external non-peer-reviewed source (Section III-A, reference [8]).
domain assumption LLM Arena scores are a valid proxy for general linguistic ability.
The claimed correlation in Section III-C assumes the Arena ranking is meaningful and that performance differences are due to language ability rather than other confounds.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pok\'eAI: A Goal-Generating, Battle-Optimizing Multi-agent System for Pokemon Red." pith.science (2026). https://pith.science/paper/2A3NOSXH

@misc{pith2026250623689,
  author       = {Pith},
  title        = {Pith review of: Pok\'eAI: A Goal-Generating, Battle-Optimizing Multi-agent System for Pokemon Red},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2A3NOSXH}},
  note         = {Machine review of arXiv:2506.23689}
}

read the original abstract

We introduce Pok\'eAI, the first text-based, multi-agent large language model (LLM) framework designed to autonomously play and progress through Pok\'emon Red. Our system consists of three specialized agents-Planning, Execution, and Critique-each with its own memory bank, role, and skill set. The Planning Agent functions as the central brain, generating tasks to progress through the game. These tasks are then delegated to the Execution Agent, which carries them out within the game environment. Upon task completion, the Critique Agent evaluates the outcome to determine whether the objective was successfully achieved. Once verification is complete, control returns to the Planning Agent, forming a closed-loop decision-making system. As a preliminary step, we developed a battle module within the Execution Agent. Our results show that the battle AI achieves an average win rate of 80.8% across 50 wild encounters, only 6% lower than the performance of an experienced human player. Furthermore, we find that a model's battle performance correlates strongly with its LLM Arena score on language-related tasks, indicating a meaningful link between linguistic ability and strategic reasoning. Finally, our analysis of gameplay logs reveals that each LLM exhibits a unique playstyle, suggesting that individual models develop distinct strategic behaviors.

Figures

Figures reproduced from arXiv: 2506.23689 by the authors.

**Figure 2.** Battle Module flowchart A. Battle Module Performance Evaluation We evaluate the battle AI’s performance by placing it in Mt. Moon and allowing it to engage in 50 wild encounter battles. The evaluation uses a game checkpoint featuring a Level 15 Charmander with the moves Scratch, Growl, Ember, and Leer, a Level 15 Pidgey with Gust, Sand Attack, and Quick Attack, and five potions in the bag. Mt. Moon features four wil… view at source ↗

**Figure 1.** PokeAI Overall Framework ´ III. PRELIMINARY RESULTS We began assembling PokeAI by first designing the battle ´ module within the Execution Agent, as it is the most selfcontained component and functions independently of the other agents. It operates by monitoring a specific memory location in the game to detect the occurrence of a battle. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 4.** Performance comparison across LLMs and experienced [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Ablation study on action availability and win rate. Error [PITH_FULL_IMAGE:figures/full_fig_p003_3.png]

**Figure 5.** Figure 5: Action Distribution by LLM Models and Experience Human Player During Battles [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Long-term memory with Letta Agent, past battle logs [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages

[1]

V oyager: An open- ended embodied agent with large language models

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open- ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

arXiv 2023
[2]

Pokemon red via reinforcement learning, 2025

Marco Pleines, Daniel Addis, David Rubinstein, Frank Zimmer, Mike Preuss, and Peter Whidden. Pokemon red via reinforcement learning, 2025

2025
[3]

Conditional imitation learning for multi-agent games

Andy Shih, Stefano Ermon, and Dorsa Sadigh. Conditional imitation learning for multi-agent games. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 166–175, 2022

work page 2022
[4]

Gemini plays pokemon

Gemini Plays Pokemon. Gemini plays pokemon. https://www.twitch.tv/ gemini plays pokemon, 2025. Accessed: 2025-05-27

work page 2025
[5]

Claudeplayspokemon

ClaudePlaysPokemon. Claudeplayspokemon. https://www.twitch.tv/ claudeplayspokemon, 2025. Accessed: 2025-05-27

work page 2025
[6]

Beating the world record in pok ´emon emerald

NunuAI. Beating the world record in pok ´emon emerald. https://nunu. ai/case-studies/pokemon-emerald, 2025. Accessed: May 27, 2025

work page 2025
[7]

Function calling in large language models: Industrial practices, challenges, and future directions

MAOLIN W ANG, YINGYI ZHANG, CUNYIN PENG, YICHENG CHEN, WEI ZHOU, JINJIE GU, CHENYI ZHUANG, RUOCHENG GUO, BOWEN YU, W ANYU W ANG, et al. Function calling in large language models: Industrial practices, challenges, and future directions. 2025

work page 2025
[8]

Altissimo’s pok ´emon site

Altissimo. Altissimo’s pok ´emon site. https://altissimo1.github. io/Main-Series/Kanto-Locations/mt-moon.html?rby, 2025. Accessed: 2025-05-26

work page 2025

Show all 15 references

[9]

The Art of War .(The Oldest Military Treatise In The World)

Sun Tzu. The Art of War .(The Oldest Military Treatise In The World) . Lulu. com, 2017

2017
[10]

The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks

Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025

2025 arXiv
[11]

Personality traits in large language models

Gregory Serapio-Garc ´ıa, Mustafa Safdari, Cl ´ement Crepy, Luning Sun, Stephen Fitz, Marwa Abdulhai, Aleksandra Faust, and Maja Matari ´c. Personality traits in large language models. 2023

2023
[12]

Big5-chat: Shaping llm personalities through training on human- grounded data

Wenkai Li, Jiarui Liu, Andy Liu, Xuhui Zhou, Mona Diab, and Maarten Sap. Big5-chat: Shaping llm personalities through training on human- grounded data. arXiv preprint arXiv:2410.16491 , 2024

2024 arXiv
[13]

Evaluating the instruction-following abilities of language models using knowledge tasks

Rudra Murthy, Praveen Venkateswaran, Prince Kumar, and Danish Contractor. Evaluating the instruction-following abilities of language models using knowledge tasks. arXiv preprint arXiv:2410.12972 , 2024

2024 arXiv
[14]

Frontier models are capable of in-context scheming

Alexander Meinke, Bronson Schoen, J ´er´emy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984 , 2024

2024 arXiv
[15]

Memgpt: Towards llms as operating systems

Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems. 2023

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

V oyager: An open- ended embodied agent with large language models

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open- ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

arXiv 2023

[2] [2]

Pokemon red via reinforcement learning, 2025

Marco Pleines, Daniel Addis, David Rubinstein, Frank Zimmer, Mike Preuss, and Peter Whidden. Pokemon red via reinforcement learning, 2025

2025

[3] [3]

Conditional imitation learning for multi-agent games

Andy Shih, Stefano Ermon, and Dorsa Sadigh. Conditional imitation learning for multi-agent games. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 166–175, 2022

work page 2022

[4] [4]

Gemini plays pokemon

Gemini Plays Pokemon. Gemini plays pokemon. https://www.twitch.tv/ gemini plays pokemon, 2025. Accessed: 2025-05-27

work page 2025

[5] [5]

Claudeplayspokemon

ClaudePlaysPokemon. Claudeplayspokemon. https://www.twitch.tv/ claudeplayspokemon, 2025. Accessed: 2025-05-27

work page 2025

[6] [6]

Beating the world record in pok ´emon emerald

NunuAI. Beating the world record in pok ´emon emerald. https://nunu. ai/case-studies/pokemon-emerald, 2025. Accessed: May 27, 2025

work page 2025

[7] [7]

Function calling in large language models: Industrial practices, challenges, and future directions

MAOLIN W ANG, YINGYI ZHANG, CUNYIN PENG, YICHENG CHEN, WEI ZHOU, JINJIE GU, CHENYI ZHUANG, RUOCHENG GUO, BOWEN YU, W ANYU W ANG, et al. Function calling in large language models: Industrial practices, challenges, and future directions. 2025

work page 2025

[8] [8]

Altissimo’s pok ´emon site

Altissimo. Altissimo’s pok ´emon site. https://altissimo1.github. io/Main-Series/Kanto-Locations/mt-moon.html?rby, 2025. Accessed: 2025-05-26

work page 2025