{"id":"41be99cf-0959-4d89-90b0-2ee4b4575829","arxiv_id":"2504.16855","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"MC-DML uses GPT-3.5 as an MCTS action-prior policy with in-trial and cross-trial reflection memory, improving initial-planning scores on Jericho text games like Zork1, Deephome, and Ztuu.","lead":"The paper introduces MC-DML, an algorithm that pairs Monte Carlo tree search with a large language model that keeps two kinds of memory, to plan actions in text-based adventure games. It reports higher scores than prior tree-search agents on several Jericho benchmark games without needing repeated training iterations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed initial-planning advantage rests on an un-re-run, underpowered comparison: Zork1 gain over MC-LAVE-RL at iteration 4 is 48.66±1.89 vs 45.2±1.2 (n=3), and per-game Cpuct/dynamic depth are uncontrolled.","rationale":"Good-faith reading: The paper's novelty is using an LLM as a dynamic prior in PUCT, with in-trial/cross-trial memory, to avoid the planning-then-learning iterations of PUCT-RL/MC-LAVE-RL. The ablations (§4.3) show memory and dynamic pruning contribute, and the qualitative analysis (§4.4) is consistent. These are real internal evidences. The strongest empirical result is Deephome (67 vs 35), which is large; the weakest is Zork1, where the headline iteration-1 superiority is framed. I do not see an internal inconsistency in Algorithm 1, though the non-stationary prior is a heuristic. The most load-bearing vulnerability is the evaluation: a 3-run result, borrowed baselines, tuned hyperparameters, and no significance tests. The reader's weakest_assumption focuses on LLM prior noise/memory cap, which is a generalization risk; my concern is more immediately about the validity of the comparison. Both support a conditional verdict; I would not reject the paper. Hence verdict_should_be = UNCHANGED (reader's CONDITIONAL stands), agreement_with_reader = partial.","tokens_in":19963,"tokens_out":11662,"duration_ms":107197,"concrete_test":"Obtain the released code for PUCT-RL and MC-LAVE-RL (Jang et al., 2020) and the authors' MC-DML code. Re-run all three on Zork1 and Deephome in the same Jericho environment and evaluation protocol, with identical per-state simulation counts (50·|A|), identical action-validity filters, and at least 10 seeds per method. Run two configurations of the baselines: (a) the original fixed-depth search, and (b) a version with MC-DML's dynamic pruning and the same Cpuct schedule. Also run MC-DML with per-game Cpuct fixed at 50. Compute bootstrap 95% confidence intervals for the score difference between MC-DML and the best baseline at iteration 4, and a Mann-Whitney U test. If the Zork1 difference is not significant at p<0.05, or if the Deephome advantage disappears when the baselines are given the dynamic-pruning budget, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—'superior results with its initial planning' (Section 4.2, Table 3)—requires that comparing MC-DML against PUCT-RL/MC-LAVE-RL at their iteration 1 and iteration 4 scores is valid and that the margins are not noise. This premise is least secure for three concrete reasons. (1) The baseline numbers are taken from Jang et al. (2020) without re-running in the same harness; Table 3 does not report baseline variance for a same-seed comparison, and MC-DML has only 3 runs. On Zork1, MC-DML (48.66 ± 1.89) vs MC-LAVE-RL RL at iteration 4 (45.2 ± 1.2) is within plausible sampling error; a proper significance test is absent. (2) Per-game hyperparameters are tuned for MC-DML: Cpuct is set to 20 for Deephome and Library and 200 for Detective (§4.1), which is not held-out validation. (3) MC-DML's dynamic pruning searches up to depth 30 (dmax) when Q=0, whereas the baselines use a fixed search depth from Jang et al.; if the baseline tree search at iteration 1 uses shallower/search-light budgets, the 'initial planning' comparison is confounded by compute. If the comparison is unfair or underpowered, the headline outperformance claim fails even if the LLM prior is well-calibrated. By contrast, the reader's prior-reliability concern would only limit generalization, not the reported win.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MC-DML combines Monte Carlo tree search with an LLM action prior. The LLM is prompted with a short in-trial memory (the previous observation, action, and current observation) and a cross-trial memory of reflections generated from failed trajectories; its log-probability output is converted into a PUCT-style prior. The tree search uses dynamic depth pruning that extends the horizon when all Q-values are zero. The paper evaluates on nine Jericho benchmark games and reports that MC-DML outperforms RL, LLM, and MCTS baselines at the initial planning phase, with ablations showing that both memory components and dynamic pruning contribute. The central claim is that the method achieves sample-efficient planning without the multi-iteration planning-then-learning loop used by PUCT-RL and MC-LAVE-RL.","tokens_in":20348,"tokens_out":4831,"duration_ms":41522,"significance":"If the reported results are robust, this is a valuable practical contribution: it demonstrates that an LLM prior can be improved online with episodic memory inside tree search, avoiding the expensive iterations of prior MCTS+RL agents on text-based games. The method is simple, the code is released, and the Deephome result (67 vs 35) is striking. The contribution is nevertheless conditional on resolving the comparison-protocol issues detailed below; the current evidence does not yet support the headline claim that the initial-planning advantage is statistically reliable and not an artifact of per-game tuning or unmatched compute.","major_comments":[{"comment":"The comparison against PUCT-RL and MC-LAVE-RL relies on baseline scores taken from prior work without re-running them in the same harness, and the reported margins are not tested for significance. On Zork1, MC-DML scores 48.66±1.89 while MC-LAVE-RL's iteration-4 RL score is 45.2±1.2; with only three MC-DML runs these distributions overlap substantially. On Ludicorp, MC-DML (19.67±1.7) is worse than MC-LAVE-RL (22.8±0.2), which is not discussed in the main text. The paper should re-run at least the two direct MCTS baselines under identical evaluation conditions, or provide a paired significance test, before claiming 'superior results' in Section 4.2.","section":"§4.2, Tables 1–3"},{"comment":"Per-game hyperparameter tuning is not held out. Cpuct is set to 20 for Deephome and Library and to 200 for Detective, with a default of 50 elsewhere, and the dynamic pruning schedule (dmin=10, dmax=30, Δd=20) is a fixed design choice across games but is not validated on a development set separate from the evaluation games. Since these constants directly scale the LLM prior term in Eq. (3), the reported gains may partly reflect tuning to the benchmark. Please report sensitivity to Cpuct and depth parameters, or use a single fixed configuration across all games.","section":"§4.1, Appendix B"},{"comment":"The 'initial planning' comparison is confounded by compute and search depth. MC-DML uses dynamic pruning up to depth 30 and 50×len(A) simulations, whereas the iteration-1 baseline scores for PUCT-RL and MC-LAVE-RL are taken from Jang et al. (2020) with their fixed-depth configuration. Without matching the number of simulations and depth, the comparison conflates algorithmic benefit with additional search effort. Please report results with matched compute budgets or explicitly state and justify the compute difference.","section":"§4.2, Table 3 and Algorithm 1"},{"comment":"The LLM prior in Eq. (3) is not fixed during search: cross-trial memory Mc is updated whenever a failed trajectory is reflected on, so π(a|s) changes across simulations. This is a deliberate feature, but it departs from standard PUCT assumptions, and the paper does not isolate how much of the gain comes from the dynamic prior versus the richer in-trial memory itself. I recommend a concrete test: compare MC-DML against a variant with a fixed prior (e.g., reflections frozen before search), and measure the LLM prior's agreement with held-out expert actions from a walkthrough. This would substantiate the claimed mechanism rather than leaving it as an informal explanation.","section":"§3.1, Eq. (3), Algorithm 1"}],"minor_comments":[{"comment":"The exploration constant is denoted Cuct in Eq. (1) and Cpuct in Eq. (2) and elsewhere; please unify the notation.","section":"§2.1, Eq. (1)–(2)"},{"comment":"The UCT exploration term is written as N(s,a)+1 in the denominator, while Eq. (3) writes 1+N(s,a). Make the forms consistent.","section":"Algorithm 1, line 34"},{"comment":"The text defines Q(o,a) in the context of Eq. (3), but the standard definition uses Q(s,a). Please correct the state argument.","section":"§3.1, after Eq. (3)"},{"comment":"The enumeration 'without DP, without Mc, without DP, and without Mc and DP, Mc, and Mi' is confusing and appears to contain a typo; the column headers in Table 4 are much clearer than the prose.","section":"§4.3, ablation paragraph"},{"comment":"The game name 'Zutt' appears twice; this should be 'Ztuu' to match the tables.","section":"§4.3, text after Table 4"},{"comment":"The prompt template contains 'CORSS_TRIAL_MEMORY'; this should be 'CROSS_TRIAL_MEMORY'.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an ICLR 2025 camera-ready and the evaluation is typical for that venue, but for a journal version the comparison-protocol and hyperparameter-tuning issues are serious enough that the central claim needs strengthening. The code availability and the strength of the Deephome and Ztuu results make the paper worth a revision. I would not reject outright, but the authors should either re-run baselines in their own harness or temper the 'superior results' claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: MC-DML is a sensible integration of an LLM prior into PUCT with both in-trial and cross-trial memory, applied to text-based games. It is not a breakthrough, but it is a legitimate new combination, and the Deephome result (67 vs 35) is striking. The evidence, however, is not as clean as the headline suggests.\n\nWhat is actually new: the memory mechanism updates the LLM's action prior during a single planning episode, using reflections from failed trajectories; that dynamic adjustment is not in Zhao et al. (2024) or Shinn et al. (2024), and applying it to Jericho games is a genuine contribution. The paper also does a few things right: it reports ablated variants, shows the memory components matter, and its comparisons to the LLM-only and Reflection baselines are sensible.\n\nThe soft spots are the usual ones for this line of work. The baseline numbers for PUCT-RL and MC-LAVE-RL are taken from Jang et al. (2020) without re-running; the MC-DML runs are n=3; and on Zork1 the margin over MC-LAVE-RL at iteration 4 (48.66±1.89 vs 45.2±1.2) is within plausible sampling error, so 'superior results with its initial planning' is not established for that game. The per-game Cpuct choices (20 for Deephome, 200 for Detective) and the dynamic depth schedule add a tuning confound. The stress-test note is on target here. I'd add that the more basic prior-reliability question—how well the log-prob softmax at one answer index really matches PUCT's π(a|s)—is never validated on held-out states; that doesn't kill the reported wins, but it limits generalization.\n\nThe Deephome and Ztuu gains are large enough that I wouldn't dismiss the method. The paper deserves a serious referee, and with code, significance tests, and held-out hyperparameters it could be a useful contribution rather than a promising preprint. The right next step is to re-run the baselines in the same harness and report per-game error bars.\n\nWho is this for: people working on LLM agents in text-based games or using LLM priors in MCTS. I'd bring it to a reading group if the group cares about evidence standards in LLM-agent papers. My recommendation: engage, but treat the headline as an existence proof until the comparison is made fair.","headline":"A legitimate integration of LLM priors and reflection memory into MCTS for text-based games, with a headline win on Deephome that needs stronger statistical backing.","tokens_in":20911,"tokens_out":2807,"would_cite":true,"duration_ms":25516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM with reflection memory can replace the trained policy network in Monte Carlo tree search, beating four-iteration baselines on text games.","keywords":["text-based games","Monte Carlo tree search","large language models","PUCT","reflection memory","dynamic memory","reinforcement learning","Jericho benchmark"],"falsifier":"On a held-out set of states sampled from the Zork1 and Deephome walkthroughs, query the MC-DML LLM with its standard prompt before any reflections and compute the rank correlation between the LLM's action probabilities and whether the walkthrough action is correct; if that correlation is near zero across states, the claim that the LLM prior drives the single-planning-phase success is falsified, since the search would have no informative prior to amplify.","tokens_in":19750,"feed_emoji":"🕹️","tokens_out":10817,"duration_ms":92606,"temperature":0.7,"pith_summary":"Text-based game agents that use Monte Carlo tree search usually need a planning-then-learning loop: search generates data, a policy network is trained on it, and the loop repeats for several iterations. The paper presents MC-DML (Monte Carlo planning with Dynamic Memory-guided Large language model) to skip that loop by letting a large language model act as the action prior inside PUCT, a tree-search rule that weights exploration by a prior over actions. The LLM is equipped with two memories—a short in-trial window of the last observation, action, and observation, plus up to three cross-trial reflections on failed trajectories—so its action probabilities change as planning proceeds. In experiments on nine Jericho games, MC-DML reaches 48.66 on Zork1 and 67 on Deephome in its initial planning phase, outperforming or matching PUCT-RL and MC-LAVE-RL that need four iterations to converge.","feed_headline":"LLM-guided tree search beats multi-iteration RL on text games","feed_subtitle":"MC-DML scores 48.66 on Zork1 and 67 on Deephome in one planning phase, without training a policy network.","key_machinery":"The load-bearing mechanism is the PUCT selection rule with the LLM as the prior policy, written in the paper as $a^* = \\arg\\max_{a \\in A} [Q(s,a) + C_{\\text{puct}} \\cdot \\mathrm{LLM}(a \\mid M_i, M_c, p) \\cdot \\sqrt{N(s)} / (1 + N(s,a))]$. Here $M_i$ is the in-trial memory—in practice the last observation-action-observation triplet—and $M_c$ is the cross-trial memory, up to three LLM-generated reflections on failed trajectories encountered during search. The LLM's token log-probabilities are normalized with a temperature-5 softmax to form an action prior; reflections are produced by prompting GPT-3.5 with the failed trajectory and asking for a one-sentence improvement suggestion. The novelty is that the prior is dynamic: each failure adds a reflection that changes $\\mathrm{LLM}(a \\mid M_i, M_c, p)$ in later simulations, so the tree search corrects its own exploration bias during a single planning episode.","core_discovery":"MC-DML's central claim is that the prior policy in PUCT does not have to be learned from iterative self-play data; a frozen LLM can serve as the prior, provided it is given a compact in-trial memory and a cross-trial memory of reflections on failed simulations. Concretely, the LLM assigns a probability to each valid action using the current observation window and any stored reflections, and PUCT combines this prior with Q-values and visit counts. When a simulation ends in death, the LLM is prompted to produce a one-sentence reflection, which is stored and fed into subsequent simulations from the same root, letting the prior correct itself mid-plan. The paper reports that this single planning phase achieves 48.66 ± 1.89 in Zork1 versus 45.2 ± 1.2 for MC-LAVE-RL after four iterations, 67 ± 1.41 in Deephome versus 35 ± 0.6, and full completion of Pentari and Detective, and that removing the memory components degrades performance across the board.","pith_inferences":["Outside the paper, the same recipe suggests a general pattern for sparse-reward decision problems: let an LLM provide a soft prior, and let the environment's failures generate episodic reflections that revise that prior during search, rather than training a policy from scratch.","The paper's reliance on GPT-3.5's token log-probabilities at a single answer position is a practical fragility; a natural test is whether the result survives with models that do not expose comparable log-probs, using the self-consistency or verbalized fallbacks the paper mentions but does not evaluate.","Because the in-trial memory is only the last three elements, the method may miss puzzles whose clue appeared far earlier—the paper's own Limitation section says so; an inference worth testing is whether retrieval over the full trajectory restores those scores.","The dynamic pruning rule—increase search depth only when the best Q-value is zero—could be interpreted as a reward-adaptive curriculum; testing it on other sparse-reward domains would separate its contribution from the LLM prior."],"forward_implications":["Text-based game agents can reach competitive scores after one planning phase, eliminating the repeated search-train iterations that make planning-then-learning methods time-consuming.","The action prior does not need to be a neural network trained on environment-specific data; a pretrained LLM generalizes across the nine Jericho games without game-specific walkthrough hints.","Bottleneck states such as the Zork1 trapdoor can be resolved by reflecting on a failed trajectory and storing the reflection, shifting the LLM's probabilities toward actions with no immediate reward but long-term value.","The memory components are load-bearing: the ablation in Table 4 shows that removing cross-trial memory lowers scores, and removing both in-trial and cross-trial memory lowers them further."],"supporting_citations":[{"why":"Defines the PUCT-RL and MC-LAVE-RL baselines and the fixed-depth search settings that MC-DML is compared against and extends.","marker":"Jang et al. (2020)"},{"why":"Introduces the PUCT prior-policy term that MC-DML replaces with an LLM action distribution.","marker":"Silver et al. (2017)"},{"why":"Demonstrates the AlphaGo-style planning-then-learning loop MC-DML claims to avoid by using a frozen LLM prior.","marker":"Silver et al. (2018)"},{"why":"Provides the UCT bandit exploration rule from which the PUCT bonus derives.","marker":"Kocsis & Szepesvári (2006)"},{"why":"Supplies the Reflection pattern of verbal reinforcement that becomes MC-DML's cross-trial memory.","marker":"Shinn et al. (2024)"},{"why":"Provides the Jericho benchmark and the nine text games whose scores ground the experimental comparison.","marker":"Hausknecht et al. (2020)"},{"why":"The closest prior LLM-as-prior MCTS approach; MC-DML's dynamic memory is positioned as the improvement over it.","marker":"Zhao et al. (2024)"}],"fun_headline_variants":["Single-pass LLM-tree search outdoes iterative RL in text games","LLM prior with memory beats RL after one planning round","MC-DML: one-shot planning with LLM memory tops RL baselines","Frozen LLM as PUCT prior: faster text-game planning than RL","LLM reflections guide tree search past multi-iteration RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the probability scores GPT-3.5 assigns to valid actions, based only on the most recent observation-action-observation and up to three stored reflections, reliably identify which actions are worth exploring; if those scores are noisy, or if the short memory window leaves out the clue behind a repeated failure, the reported gains would not replicate.","fun_headline_variants_meta":{"raw":{"variants":["Single-pass LLM-tree search outdoes iterative RL in text games","LLM prior with memory beats RL after one planning round","MC-DML: one-shot planning with LLM memory tops RL baselines","Frozen LLM as PUCT prior: faster text-game planning than RL","LLM reflections guide tree search past multi-iteration RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":3000,"prompt_tokens":955,"completion_tokens":2045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1954}},"tokens_in":571,"tokens_out":2045,"duration_ms":11721,"temperature":1.0,"reasoning_tokens":1954,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:53:37.881010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of states sampled from the Zork1 and Deephome walkthroughs, query the MC-DML LLM with its standard prompt before any reflections and compute the rank correlation between the LLM's action probabilities and whether the walkthrough action is correct; if that correlation is near zero across states, the claim that the LLM prior drives the single-planning-phase success is falsified, since the search would have no informative prior to amplify.","supporting_citations":[],"review_version":1}