Monte Carlo Planning with Large Language Model for Text-Based Game Agents

Ling Chen; Meng Fang; Zijing Shi

REVIEW 4 major objections 6 minor 1 cited by

Monte Carlo Planning with Large Language Model for Text-Based Game Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM with reflection memory can replace the trained policy network in Monte Carlo tree search, beating four-iteration baselines on text games.

desk verdict A legitimate integration of LLM priors and reflection memory into MCTS for text-based games, with a headline win on Deephome that needs stronger statistical backing. read the letter →

arxiv 2504.16855 v1 pith:WYQGG7FK submitted 2025-04-23 cs.CL

Zijing Shi , Meng Fang , Ling Chen This is my paper · ORCID

classification cs.CL

keywords text-basedgamesMonteCarlotreesearchlargelanguagemodelsPUCTreflectionmemorydynamicreinforcementlearningJerichobenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-based game agents that use Monte Carlo tree search usually need a planning-then-learning loop: search generates data, a policy network is trained on it, and the loop repeats for several iterations. The paper presents MC-DML (Monte Carlo planning with Dynamic Memory-guided Large language model) to skip that loop by letting a large language model act as the action prior inside PUCT, a tree-search rule that weights exploration by a prior over actions. The LLM is equipped with two memories—a short in-trial window of the last observation, action, and observation, plus up to three cross-trial reflections on failed trajectories—so its action probabilities change as planning proceeds. In experiments on nine Jericho games, MC-DML reaches 48.66 on Zork1 and 67 on Deephome in its initial planning phase, outperforming or matching PUCT-RL and MC-LAVE-RL that need four iterations to converge.

What carries the argument

The load-bearing mechanism is the PUCT selection rule with the LLM as the prior policy, written in the paper as $a^* = \arg\max_{a \in A} [Q(s,a) + C_{\text{puct}} \cdot \mathrm{LLM}(a \mid M_i, M_c, p) \cdot \sqrt{N(s)} / (1 + N(s,a))]$. Here $M_i$ is the in-trial memory—in practice the last observation-action-observation triplet—and $M_c$ is the cross-trial memory, up to three LLM-generated reflections on failed trajectories encountered during search. The LLM's token log-probabilities are normalized with a temperature-5 softmax to form an action prior; reflections are produced by prompting GPT-3.5 with the failed trajectory and asking for a one-sentence improvement suggestion. The novelty is that the prior is dynamic: each failure adds a reflection that changes $\mathrm{LLM}(a \mid M_i, M_c, p)$ in later simulations, so the tree search corrects its own exploration bias during a single planning episode.

What would settle it

On a held-out set of states sampled from the Zork1 and Deephome walkthroughs, query the MC-DML LLM with its standard prompt before any reflections and compute the rank correlation between the LLM's action probabilities and whether the walkthrough action is correct; if that correlation is near zero across states, the claim that the LLM prior drives the single-planning-phase success is falsified, since the search would have no informative prior to amplify.

Watch

Extended reading notes

Core claim

MC-DML's central claim is that the prior policy in PUCT does not have to be learned from iterative self-play data; a frozen LLM can serve as the prior, provided it is given a compact in-trial memory and a cross-trial memory of reflections on failed simulations. Concretely, the LLM assigns a probability to each valid action using the current observation window and any stored reflections, and PUCT combines this prior with Q-values and visit counts. When a simulation ends in death, the LLM is prompted to produce a one-sentence reflection, which is stored and fed into subsequent simulations from the same root, letting the prior correct itself mid-plan. The paper reports that this single planning phase achieves 48.66 ± 1.89 in Zork1 versus 45.2 ± 1.2 for MC-LAVE-RL after four iterations, 67 ± 1.41 in Deephome versus 35 ± 0.6, and full completion of Pentari and Detective, and that removing the memory components degrades performance across the board.

Load-bearing premise

The paper assumes the probability scores GPT-3.5 assigns to valid actions, based only on the most recent observation-action-observation and up to three stored reflections, reliably identify which actions are worth exploring; if those scores are noisy, or if the short memory window leaves out the clue behind a repeated failure, the reported gains would not replicate.

Editorial extensions

If this is right

Text-based game agents can reach competitive scores after one planning phase, eliminating the repeated search-train iterations that make planning-then-learning methods time-consuming.
The action prior does not need to be a neural network trained on environment-specific data; a pretrained LLM generalizes across the nine Jericho games without game-specific walkthrough hints.
Bottleneck states such as the Zork1 trapdoor can be resolved by reflecting on a failed trajectory and storing the reflection, shifting the LLM's probabilities toward actions with no immediate reward but long-term value.
The memory components are load-bearing: the ablation in Table 4 shows that removing cross-trial memory lowers scores, and removing both in-trial and cross-trial memory lowers them further.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Outside the paper, the same recipe suggests a general pattern for sparse-reward decision problems: let an LLM provide a soft prior, and let the environment's failures generate episodic reflections that revise that prior during search, rather than training a policy from scratch.
The paper's reliance on GPT-3.5's token log-probabilities at a single answer position is a practical fragility; a natural test is whether the result survives with models that do not expose comparable log-probs, using the self-consistency or verbalized fallbacks the paper mentions but does not evaluate.
Because the in-trial memory is only the last three elements, the method may miss puzzles whose clue appeared far earlier—the paper's own Limitation section says so; an inference worth testing is whether retrieval over the full trajectory restores those scores.
The dynamic pruning rule—increase search depth only when the best Q-value is zero—could be interpreted as a reward-adaptive curriculum; testing it on other sparse-reward domains would separate its contribution from the LLM prior.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: MC-DML is a sensible integration of an LLM prior into PUCT with both in-trial and cross-trial memory, applied to text-based games. It is not a breakthrough, but it is a legitimate new combination, and the Deephome result (67 vs 35) is striking. The evidence, however, is not as clean as the headline suggests.

What is actually new: the memory mechanism updates the LLM's action prior during a single planning episode, using reflections from failed trajectories; that dynamic adjustment is not in Zhao et al. (2024) or Shinn et al. (2024), and applying it to Jericho games is a genuine contribution. The paper also does a few things right: it reports ablated variants, shows the memory components matter, and its comparisons to the LLM-only and Reflection baselines are sensible.

The soft spots are the usual ones for this line of work. The baseline numbers for PUCT-RL and MC-LAVE-RL are taken from Jang et al. (2020) without re-running; the MC-DML runs are n=3; and on Zork1 the margin over MC-LAVE-RL at iteration 4 (48.66±1.89 vs 45.2±1.2) is within plausible sampling error, so 'superior results with its initial planning' is not established for that game. The per-game Cpuct choices (20 for Deephome, 200 for Detective) and the dynamic depth schedule add a tuning confound. The stress-test note is on target here. I'd add that the more basic prior-reliability question—how well the log-prob softmax at one answer index really matches PUCT's π(a|s)—is never validated on held-out states; that doesn't kill the reported wins, but it limits generalization.

The Deephome and Ztuu gains are large enough that I wouldn't dismiss the method. The paper deserves a serious referee, and with code, significance tests, and held-out hyperparameters it could be a useful contribution rather than a promising preprint. The right next step is to re-run the baselines in the same harness and report per-game error bars.

Who is this for: people working on LLM agents in text-based games or using LLM priors in MCTS. I'd bring it to a reading group if the group cares about evidence standards in LLM-agent papers. My recommendation: engage, but treat the headline as an existence proof until the comparison is made fair.

Referee Report

4 major / 6 minor

Summary. MC-DML combines Monte Carlo tree search with an LLM action prior. The LLM is prompted with a short in-trial memory (the previous observation, action, and current observation) and a cross-trial memory of reflections generated from failed trajectories; its log-probability output is converted into a PUCT-style prior. The tree search uses dynamic depth pruning that extends the horizon when all Q-values are zero. The paper evaluates on nine Jericho benchmark games and reports that MC-DML outperforms RL, LLM, and MCTS baselines at the initial planning phase, with ablations showing that both memory components and dynamic pruning contribute. The central claim is that the method achieves sample-efficient planning without the multi-iteration planning-then-learning loop used by PUCT-RL and MC-LAVE-RL.

Significance. If the reported results are robust, this is a valuable practical contribution: it demonstrates that an LLM prior can be improved online with episodic memory inside tree search, avoiding the expensive iterations of prior MCTS+RL agents on text-based games. The method is simple, the code is released, and the Deephome result (67 vs 35) is striking. The contribution is nevertheless conditional on resolving the comparison-protocol issues detailed below; the current evidence does not yet support the headline claim that the initial-planning advantage is statistically reliable and not an artifact of per-game tuning or unmatched compute.

major comments (4)

[§4.2, Tables 1–3] The comparison against PUCT-RL and MC-LAVE-RL relies on baseline scores taken from prior work without re-running them in the same harness, and the reported margins are not tested for significance. On Zork1, MC-DML scores 48.66±1.89 while MC-LAVE-RL's iteration-4 RL score is 45.2±1.2; with only three MC-DML runs these distributions overlap substantially. On Ludicorp, MC-DML (19.67±1.7) is worse than MC-LAVE-RL (22.8±0.2), which is not discussed in the main text. The paper should re-run at least the two direct MCTS baselines under identical evaluation conditions, or provide a paired significance test, before claiming 'superior results' in Section 4.2.
[§4.1, Appendix B] Per-game hyperparameter tuning is not held out. Cpuct is set to 20 for Deephome and Library and to 200 for Detective, with a default of 50 elsewhere, and the dynamic pruning schedule (dmin=10, dmax=30, Δd=20) is a fixed design choice across games but is not validated on a development set separate from the evaluation games. Since these constants directly scale the LLM prior term in Eq. (3), the reported gains may partly reflect tuning to the benchmark. Please report sensitivity to Cpuct and depth parameters, or use a single fixed configuration across all games.
[§4.2, Table 3 and Algorithm 1] The 'initial planning' comparison is confounded by compute and search depth. MC-DML uses dynamic pruning up to depth 30 and 50×len(A) simulations, whereas the iteration-1 baseline scores for PUCT-RL and MC-LAVE-RL are taken from Jang et al. (2020) with their fixed-depth configuration. Without matching the number of simulations and depth, the comparison conflates algorithmic benefit with additional search effort. Please report results with matched compute budgets or explicitly state and justify the compute difference.
[§3.1, Eq. (3), Algorithm 1] The LLM prior in Eq. (3) is not fixed during search: cross-trial memory Mc is updated whenever a failed trajectory is reflected on, so π(a|s) changes across simulations. This is a deliberate feature, but it departs from standard PUCT assumptions, and the paper does not isolate how much of the gain comes from the dynamic prior versus the richer in-trial memory itself. I recommend a concrete test: compare MC-DML against a variant with a fixed prior (e.g., reflections frozen before search), and measure the LLM prior's agreement with held-out expert actions from a walkthrough. This would substantiate the claimed mechanism rather than leaving it as an informal explanation.

minor comments (6)

[§2.1, Eq. (1)–(2)] The exploration constant is denoted Cuct in Eq. (1) and Cpuct in Eq. (2) and elsewhere; please unify the notation.
[Algorithm 1, line 34] The UCT exploration term is written as N(s,a)+1 in the denominator, while Eq. (3) writes 1+N(s,a). Make the forms consistent.
[§3.1, after Eq. (3)] The text defines Q(o,a) in the context of Eq. (3), but the standard definition uses Q(s,a). Please correct the state argument.
[§4.3, ablation paragraph] The enumeration 'without DP, without Mc, without DP, and without Mc and DP, Mc, and Mi' is confusing and appears to contain a typo; the column headers in Table 4 are much clearer than the prose.
[§4.3, text after Table 4] The game name 'Zutt' appears twice; this should be 'Ztuu' to match the tables.
[Appendix C.1] The prompt template contains 'CORSS_TRIAL_MEMORY'; this should be 'CROSS_TRIAL_MEMORY'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MC-DML's LLM prior and memory mechanism are not fitted to final scores, and the central results are benchmarked against independently published baselines.

full rationale

The paper's derivation chain is self-contained with respect to its inputs. The LLM prior is obtained from a fixed GPT-3.5 model conditioned on in-trial memory (ot-1, at-1, ot) and up to three cross-trial reflections; none of these are trained or tuned to maximize the final game scores reported in Tables 1-3. The final action is selected by argmax Q after Monte Carlo rollouts, where Q is updated from environment rewards and discounted rollout returns, not from the claimed outcome. Reflections are generated online from failed trajectories during search and then fed back into the prior, which is an algorithmic mechanism rather than a fitted parameter renamed as a prediction. The main comparisons use baseline scores from Jang et al. (2020) and related published work, i.e., external evidence, not a subset of the paper's own fitted data. Hand-chosen constants such as the softmax temperature of 5, Cpuct values, and memory size are hyperparameter choices; they may affect generalization or statistical rigor, but they do not make the reported performance equivalent to the method's inputs by construction. Self-citations in the related-work section (Shi et al. 2023a,b; Fang et al. 2024) are bibliographic context and are not load-bearing for the novelty or the empirical claims. Concerns about underpowered baseline comparisons, per-game tuning, or uncontrolled search depth are correctness or experimental-design risks, not circularity. Therefore no circular step can be exhibited from the paper's equations or citations, and the appropriate score is 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The algorithm's performance rests on a set of hand-chosen hyperparameters (per-game Cpuct, memory size, depth schedule, softmax temperature) and on the unverified assumption that GPT-3.5's log-probability outputs provide a stable, useful prior while reflection memory changes that prior during search.

free parameters (8)

C_puct = 50 (20 for Deephome, Library; 200 for Detective)
Balances LLM prior and Q-values in PUCT; adapted per game from Jang et al. (2020), tuned on the benchmark.
softmax temperature for LLM log probs = 5
Normalizes action probabilities; chosen without reported sensitivity analysis.
log probability floor for absent tokens = -10
Arbitrary placeholder for actions not in top-20 log probs.
cross-trial memory size k = 3
Number of stored reflections per root node; chosen, no ablation across k reported.
in-trial memory window = (o_{t-1}, a_{t-1}, o_t)
Short window defining current state; limitation acknowledged in Section 6.
dynamic pruning depths = dmin=10, dmax=30, Δd=20
Heuristic to handle sparse rewards; ablation shows large effect on Ztuu.
number of simulations = 50 × |A|
Computational budget per root; no sensitivity analysis.
discount factor γ = 0.95
Standard, follows prior work; not tuned here.

assumptions (4)

domain assumption Text-based games with Jericho valid-action filtering are a POMDP.
Section 2.2 models games as (S,T,A,O,R,γ) and Section 4.1 uses the valid-action handicap; all baselines rely on this framing.
domain assumption GPT-3.5 token log probabilities at the answer-index output position, after temperature-5 softmax, form a valid action prior π(a|s).
Section 3.1 and Appendix B define the LLM policy this way; no validation of the prior against held-out states is provided.
domain assumption Reflections from failed trajectories transfer to later states in the same game.
Section 3.1 stores reflections in Mc and feeds them into subsequent simulations; the paper does not test whether reflections generalize across distinct puzzles.
standard math PUCT convergence behavior remains valid when the prior changes during search as Mc is updated.
Algorithm 1 appends reflections to Mc inside the repeat loop, so π(a|s) is non-stationary, which is outside standard PUCT assumptions; the paper does not discuss this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Monte Carlo Planning with Large Language Model for Text-Based Game Agents." pith.science (2026). https://pith.science/paper/WYQGG7FK

@misc{pith2026250416855,
  author       = {Pith},
  title        = {Pith review of: Monte Carlo Planning with Large Language Model for Text-Based Game Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYQGG7FK}},
  note         = {Machine review of arXiv:2504.16855}
}

read the original abstract

Text-based games provide valuable environments for language-based autonomous agents. However, planning-then-learning paradigms, such as those combining Monte Carlo Tree Search (MCTS) and reinforcement learning (RL), are notably time-consuming due to extensive iterations. Additionally, these algorithms perform uncertainty-driven exploration but lack language understanding and reasoning abilities. In this paper, we introduce the Monte Carlo planning with Dynamic Memory-guided Large language model (MC-DML) algorithm. MC-DML leverages the language understanding and reasoning capabilities of Large Language Models (LLMs) alongside the exploratory advantages of tree search algorithms. Specifically, we enhance LLMs with in-trial and cross-trial memory mechanisms, enabling them to learn from past experiences and dynamically adjust action evaluations during planning. We conduct experiments on a series of text-based games from the Jericho benchmark. Our results demonstrate that the MC-DML algorithm significantly enhances performance across various games at the initial planning phase, outperforming strong contemporary methods that require multiple iterations. This demonstrates the effectiveness of our algorithm, paving the way for more efficient language-grounded planning in complex environments.

Figures

Figures reproduced from arXiv: 2504.16855 by the authors.

**Figure 2.** A comparison of the PUCT and MC-DML algorithms. PUCT trains its policy through [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities
cs.AI 2025-06 conditional novelty 5.0 of 10

A survey that groups graph-empowered AI agent research into planning, execution, memory, and multi-agent coordination, plus agents-for-graphs and applications.

Reference graph

Works this paper leans on

30 extracted references · 13 canonical work pages · cited by 1 Pith paper

[2]

How to avoid being eaten by a grue: Structured exploration strategies for textual worlds

Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07409,

arXiv 2006
[3]

Case- based reasoning for better generalization in textual reinforcement learning

Mattia Atzeni, Shehzaad Dhuliawala, Keerthiram Murugesan, and Mrinmaya Sachan. Case- based reasoning for better generalization in textual reinforcement learning. arXiv preprint arXiv:2110.08470,

arXiv
[5]

Textworld: A learning environment for text-based games

Marc-Alexandre Cˆot´e, Akos K´ad´ar, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Con- junction with the 27th International Conference on Artificial Intelligence, IJCAI 20...

work page 2018
[8]

Everything of thoughts: Defying the law of penrose triangle for thought generation

Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Ra- jmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254,

arXiv
[10]

Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning

Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. Interac- tive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.02386,

work page Pith review arXiv 2010
[11]

Deep reinforcement learning with a natural language action space

Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv preprint arXiv:1511.04636,

arXiv
[14]

Language understanding for text-based games using deep reinforcement learning

Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941,

arXiv
[16]

doi: 10.18653/v1/2022.acl-short.56

Association for Computational Lin- guistics. doi: 10.18653/v1/2022.acl-short.56. URL https://aclanthology.org/2022. acl-short.56. Zijing Shi, Meng Fang, Yunqiu Xu, Ling Chen, and Yali Du. Stay moral and explore: Learn to behave morally in text-based games. In The Eleventh International Conference on Learning Representations, 2023a. Zijing Shi, Yunqiu Xu, ...

work page doi:10.18653/v1/2022.acl-short.56 2022

Show all 30 references

[17]

Alfworld: Aligning text and embodied environments for interactive learning

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆot´e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768,

2010 arXiv
[18]

Lgts: Dynamic task sampling using llm-generated sub-goals for reinforcement learning agents

Yash Shukla, Wenchang Gao, Vasanth Sarathy, Alvaro Velasquez, Robert Wright, and Jivko Sinapov. Lgts: Dynamic task sampling using llm-generated sub-goals for reinforcement learning agents. arXiv preprint arXiv:2310.09454,

arXiv
[19]

Simplified belief-dependent reward mcts planning with guaranteed tree consistency

Ori Sztyglic, Andrey Zhitnikov, and Vadim Indelman. Simplified belief-dependent reward mcts planning with guaranteed tree consistency. arXiv preprint arXiv:2105.14239,

arXiv
[20]

Multi-stage episodic control for strategic exploration in text games

Jens Tuyls, Shunyu Yao, Sham Kakade, and Karthik Narasimhan. Multi-stage episodic control for strategic exploration in text games. arXiv preprint arXiv:2201.01251,

arXiv
[21]

Self-consistency improves chain of thought reasoning in language models

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

arXiv
[22]

Generalization in text-based games via hierarchical reinforcement learning

12 Published as a conference paper at ICLR 2025 Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, and Chengqi Zhang. Generalization in text-based games via hierarchical reinforcement learning. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pp. 1343–1353, Pu...

2025
[23]

doi: 10.18653/v1/2021.findings-emnlp.116

Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.116. URL https://aclanthology.org/2021.findings-emnlp.116. Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Zhou, and Chengqi Zhang. Perceiving the world: Question-guided reinforcement learning for t...

2021 doi
[24]

URL https://aclanthology.org/2022.acl-long.41

18653/v1/2022.acl-long.41. URL https://aclanthology.org/2022.acl-long.41. Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep CALM and ex- plore: Language models for action generation in text-based games. In Proceedings of the Conference on Empirical Method...

2022
[25]

URL https://aclanthology.org/ 2020.emnlp-main.704

doi: 10.18653/v1/2020.emnlp-main.704. URL https://aclanthology.org/ 2020.emnlp-main.704. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Ne...

2020 doi
[27]

Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory

Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint...

arXiv
[28]

Ludicorp

13 Published as a conference paper at ICLR 2025 A G AME STATISTICS We conduct experiments upon 9 games provided by the Jericho Game Suite (Hausknecht et al., 2020). Different from those generated through pre-defined simple rules (C ˆot´e et al., 2019), the games we use are mor...

2025
[30]

C.1 P ROMPTS FOR ACTION VALUE ESTIMATES You are a player in a text-based adventure game

14 Published as a conference paper at ICLR 2025 C LLM P ROMPTS In this section, we provide the prompts used for action value estimation by the LLM, as well as the prompts used for reflection. C.1 P ROMPTS FOR ACTION VALUE ESTIMATES You are a player in a text-based adventure ga...

2025
[50]

The above configuration follows the work of Jang et al

Specifically, it is set to 20 for the gamesDeephome and Library, and to 200 for the game Detective. The above configuration follows the work of Jang et al. (2020). We set dmin to 10, dmax to 30, and the step increment ∆d to

2020
[2006]

Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks

Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.01534,

arXiv
[2012]

A survey of monte carlo tree search methods

10 Published as a conference paper at ICLR 2025 Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Tran...

2025
[2015]

A survey of text games for reinforcement learning informed by natural language

11 Published as a conference paper at ICLR 2025 Philip Osborne, Heido N˜omm, and Andr´e Freitas. A survey of text games for reinforcement learning informed by natural language. Transactions of the Association for Computational Linguistics, 10: 873–887,

2025
[2018]

How can llm guide rl? a value-based approach

Shenao Zhang, Sirui Zheng, Shuqi Ke, Zhihan Liu, Wanxin Jin, Jianbo Yuan, Yingxiang Yang, Hongxia Yang, and Zhaoran Wang. How can llm guide rl? a value-based approach. arXiv preprint arXiv:2402.16181,

arXiv
[2020]

Graph constrained reinforcement learning for natural language action spaces

Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces. arXiv preprint arXiv:2001.08837,

2001 arXiv
[2021]

Teaching models to express their uncertainty in words

Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334,

arXiv
[2022]

Rl-gpt: Integrating reinforcement learning and code-as-policy

Shaoteng Liu, Haoqi Yuan, Minda Hu, Yanwei Li, Yukang Chen, Shu Liu, Zongqing Lu, and Jiaya Jia. Rl-gpt: Integrating reinforcement learning and code-as-policy. arXiv preprint arXiv:2402.19299,

arXiv
[2023]

Large language models are neurosymbolic reasoners

Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, and Jun Wang. Large language models are neurosymbolic reasoners. arXiv preprint arXiv:2401.09334,

arXiv
[2024]

Playing text-based games with common sense

Sahith Dambekodi, Spencer Frazier, Prithviraj Ammanabrolu, and Mark O Riedl. Playing text-based games with common sense. arXiv preprint arXiv:2012.02757,

2012 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [2]

How to avoid being eaten by a grue: Structured exploration strategies for textual worlds

Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07409,

arXiv 2006

[2] [3]

Case- based reasoning for better generalization in textual reinforcement learning

Mattia Atzeni, Shehzaad Dhuliawala, Keerthiram Murugesan, and Mrinmaya Sachan. Case- based reasoning for better generalization in textual reinforcement learning. arXiv preprint arXiv:2110.08470,

arXiv

[3] [5]

Textworld: A learning environment for text-based games

Marc-Alexandre Cˆot´e, Akos K´ad´ar, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Con- junction with the 27th International Conference on Artificial Intelligence, IJCAI 20...

work page 2018

[4] [8]

Everything of thoughts: Defying the law of penrose triangle for thought generation

Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Ra- jmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254,

arXiv

[5] [10]

Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning

Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. Interac- tive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.02386,

work page Pith review arXiv 2010

[6] [11]

Deep reinforcement learning with a natural language action space

Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv preprint arXiv:1511.04636,

arXiv

[7] [14]

Language understanding for text-based games using deep reinforcement learning

Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941,

arXiv

[8] [16]

doi: 10.18653/v1/2022.acl-short.56

Association for Computational Lin- guistics. doi: 10.18653/v1/2022.acl-short.56. URL https://aclanthology.org/2022. acl-short.56. Zijing Shi, Meng Fang, Yunqiu Xu, Ling Chen, and Yali Du. Stay moral and explore: Learn to behave morally in text-based games. In The Eleventh International Conference on Learning Representations, 2023a. Zijing Shi, Yunqiu Xu, ...

work page doi:10.18653/v1/2022.acl-short.56 2022