{"id":"1e0211dc-10ff-44a0-a1e6-4cf5264e8494","arxiv_id":"2505.10543","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Prompting strategies on dynamic game benchmarks do not produce emergent reasoning in open-source LLMs; they add variance and help smaller models only sometimes.","lead":"This paper tests four open-source chatbots as agents in dynamic games using reflection, rule-mutation, and planning prompts. It finds that these prompts rarely produce reliable reasoning gains and that the models still struggle with planning and spatial tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random-baseline control in Table II shows LLMs are worse than a policy that ignores the game rules, so the text-only task interface—not reasoning—may drive the 'little evidence for emergent reasoning' conclusion.","rationale":"I read this as a transparent empirical study: the authors report min/median/max over runs, release code, and explicitly flag the non-comparable human baseline. The strongest evidence, however, is the random-baseline control in Table II, which the paper itself reports. The fact that a uniform-random policy outperforms every LLM strategy on 3-disk Hanoi without adjustments—on goal completion, disk placement, and invalid moves—means the task setup is not a neutral measurement of reasoning. The LLMs are not merely failing to reason; they are being misled by something in the state representation or the long prompt. This makes the strong claims about 'little evidence for emergent reasoning', 'knowing-doing gap', and 'language-embodiment gap' broader than the evidence supports. The human baseline mismatch is a related but secondary issue, since the conclusion could be revised to drop the human comparison and still report low absolute performance. A symbolic-state representation control would directly test whether the deficit is representational or genuinely reasoning-based. Because the paper can be repaired with such a control and with more modest conclusions, I keep the reader's conditional verdict rather than moving to rejection or unverdictable.","tokens_in":13949,"tokens_out":11464,"duration_ms":117963,"concrete_test":"Run the 3-disk Hanoi experiment with the same models and strategies but replace the list-style rod representation with an explicit symbolic state (e.g., JSON 'rod_a': [2,1,0], 'top_a': 0) and add a scratchpad parsing step, keeping the manual and all prompts otherwise identical. If goal-completion rates increase substantially (e.g., above 10%) or invalid-move rates drop below 50%, the failures are largely an interface/representation artifact rather than evidence of fundamental reasoning limits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative conclusion rests on the assumption that the text-only SmartPlay protocol measures reasoning rather than an interface-confusion artifact. That assumption is directly challenged by the paper's own Table II: in the 3-disk Hanoi 'No adjustments' condition, the uniform-random policy achieves 2.0% goal completions and 0.9 mean disks placed, while every LLM strategy (Base, Reflection, Reflection+Oracle, Reflection+Planner) achieves 0.0% goals and 0.7 or fewer disks, with higher invalid-move rates. A policy that ignores the game manual and rules beats all prompted LLMs on both correctness and legality. This indicates that the long text-state descriptions (e.g., rod stacks rendered as 'bottom [1,0] top') and the extended prompts are actively confusing rather than merely challenging. The paper's own qualitative examples confirm Planner state misreading. If the representation or prompt length—not the underlying reasoning—causes the failures, then the 'fundamental shortcomings' and 'knowing-doing gap' conclusions are overgeneralized. The related human baseline is also acknowledged as non-comparable in the Table I footnote, further weakening the 'compared to human performance' framing in the abstract and conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical evaluation of four prompting strategies (Base, Reflection, Reflection+Oracle, Reflection+Planner) applied to four SmartPlay environments (Bandit, Rock-Paper-Scissors, Tower of Hanoi, Messenger) using four open-source LLMs (Llama 3-8B, Mistral-NeMo-12B, DeepSeek-R1-14B, Llama 3.3-70B). The authors report that larger models generally score higher, that prompting can help smaller models close the gap but with high variance, that overly long prompts can hurt small models, and they conclude that there is 'little evidence for emergent reasoning,' alongside a knowing-doing gap and a language-embodiment gap in dynamic tasks. Additional experiments in Section IV-C investigate reward shaping, synonym removal, and valid-action hints, and include random-action baselines.","tokens_in":14193,"tokens_out":4555,"duration_ms":40428,"significance":"The study is a serious, broad empirical comparison of prompting strategies on a dynamic benchmark, with a public code repository, open models, and concrete qualitative examples of failures. The knowing-doing gap is concretely demonstrated (models that can recite the optimal Hanoi sequence fail to execute it), and the inclusion of random baselines and interface modifications in Section IV-C is informative. If the identified measurement concerns are addressed, the findings would be a valuable contribution to the debate on emergent reasoning and on the limitations of static benchmarks for evaluating LLM agents.","major_comments":[{"comment":"The random-action baseline in the 3-disk Hanoi 'No adjustments' condition achieves higher goal completion (2.0% vs 0.0%), higher average disks placed (0.9 vs ≤0.7), and fewer invalid moves (69.7% vs ≥74.1%) than every LLM-based strategy. Because the state is rendered as text-only rod stacks and the prompts are long, this result is equally consistent with an interface-representation failure as with a reasoning failure; the conclusion in the same section that this reveals a 'fundamental inability to internalise the 3-disk Tower-of-Hanoi rules' is therefore not justified without an ablation that varies the state encoding or prompt length. This directly affects the paper's central negative claim.","section":"§IV-C-1, Table II"},{"comment":"All headline results rest on three runs per cell, with no confidence intervals, significance tests, or effect sizes. For example, the claim that prompting 'can close the gap' between Llama 3-8B and Llama 3.3-70B on RPS uses Llama 3-8B Reflection+Oracle median 26.00 with range [12.00–36.00] versus the 70B baseline 22.20 [21.55–25.05]; the ranges overlap substantially, so the conclusion is not statistically supported. The paper should present per-cell variance, confidence intervals, and significance tests (or equivalent), and temper claims that depend on point estimates.","section":"§IV-B, Table I"},{"comment":"The human baselines are obtained under different conditions (4-step horizon, 100 episodes for Messenger, and original SmartPlay settings) while this work uses a 10-step horizon and 20 episodes, and the footnote states they are 'not directly comparable.' Despite this, the abstract and conclusion claim 'compared to human performance, little evidence of true emergent reasoning.' The non-comparability should be stated in the abstract and conclusion, or the human-comparison framing should be removed from the central claims.","section":"Table I footnote and §V"},{"comment":"The weighted aggregation in Eq. (1) uses SmartPlay's challenge weights and min–max scaling of raw scores, but the paper does not report the weights used for each game or how the scaling interacts with the differing score ranges across games (e.g., Hanoi 0–2 vs Messenger -1 to 1). Since Figure 2 and the associated text use this aggregation to claim that 'all models consistently improve on Instruction Following,' the aggregation should be validated with sensitivity analysis or the raw scores and weights should be made available.","section":"§IV-B-2, Eq. (1)"}],"minor_comments":[{"comment":"Reference [11] is cited for Reflexion in Section II and Section III-A-1, but [11] is the Agent-Pro paper; the correct reference for Reflexion is [2] (Shinn et al.).","section":"References [2] and [11]"},{"comment":"In the 2-disk rows, the method column lists 'Planner' but the other rows use 'Reflection + Planner'; the label should be harmonized for consistency with the text.","section":"Table II"},{"comment":"The sentence 'except the Planner' should read 'except Reflection + Planner' to match the table and the rest of the section.","section":"§IV-C-1"},{"comment":"The claim that 'excessive reasoning harms performance' and the 'signal-to-noise ratio' interpretation would be more convincing if the paper reported prompt lengths or token counts per condition, since no quantitative measure of prompt length is given.","section":"Abstract and §V"}],"recommendation":"major_revision","confidential_remarks":"The random-baseline result in Table II is the key risk to the central claim. The paper already contains the seeds of a proper interface-confound analysis in Section IV-C, but the conclusions are drawn as if the confound were resolved. I recommend major revision rather than rejection because the data are useful and the additional ablations could, with modest further work, support or substantially temper the claims. Note also the citation error for Reflexion ([11] should be [2]), which should be corrected in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before citing it as evidence that LLMs can't reason in dynamic tasks. It's a systematic comparison of three prompting strategies (reflection, a LLaMEA-style heuristic mutation oracle, and a lookahead planner) across four open models on four SmartPlay games. What's actually new is the breadth of that comparison plus the instability analysis: min-med-max over three runs, with several striking examples where the same prompt that gives a big gain in one run collapses in another. The reward-shaping experiments are a genuine addition, and the authors are transparent about variance and about the non-comparable human baseline in Table I's footnote. Code is released. Those are real strengths.\n\nThe soft spots are proportionate to the claims. Three runs per cell with no confidence intervals or significance tests means the ranking claims are suggestive, not established. Bigger issue: Table II shows that in the unmodified 3-disk Hanoi, a uniform-random policy beats every LLM strategy on goal completion, disk placement, and illegal-move rate. That is a red flag that the long text-state rendering and prompt length are confusing the models rather than measuring their reasoning. The paper's own qualitative examples (the planner misreading rod states, the airplane mistaken for the goal) point the same way. The authors do partially address this by showing that valid-action hints help, and that 2-disk Hanoi is more tractable, so the right reading is that both interface and reasoning deficits are present. But the conclusion of 'little evidence for emergent reasoning' is broader than four games and three runs can support, and the 'compared to human performance' framing is weakened by their own footnote saying the human baseline isn't comparable.\n\nBottom line: the paper is worth engaging. The instability result is useful, the reward-shaping experiments are a contribution, and the authors are honest about their caveats. It deserves a serious referee, but the referee should ask for more runs or statistical support, a direct discussion of the random-baseline result, and a tempering of the headline conclusion.\n\nI'd bring it to reading group and would cite it for the instability finding, not for the emergent-reasoning verdict.","headline":"A useful four-model comparison of prompting strategies on SmartPlay with honest variance reporting, but the text-only interface may be doing more of the work than the reasoning claims acknowledge.","tokens_in":14709,"tokens_out":1372,"would_cite":true,"duration_ms":15389,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that dynamic benchmarks expose a knowing-doing gap in LLM agents: models can recite optimal strategies yet fail to execute them under planning and spatial-coordination demands.","keywords":["large language models","dynamic environments","in-context learning","self-reflection","planning","spatial reasoning","SmartPlay","emergent reasoning"],"falsifier":"Run the same four models on three-disk Tower of Hanoi and Messenger with a structured state encoding (for example, explicit disk-stack arrays and grid coordinates) under identical prompting strategies and many seeds: if goal-completion rises to near human baseline, the failures are interface artifacts and 'little evidence for emergent reasoning' is falsified; if scores stay near zero, the deficit is in reasoning and planning.","tokens_in":13755,"feed_emoji":"🎮","tokens_out":10268,"duration_ms":98054,"temperature":0.7,"pith_summary":"The paper asks whether in-context prompting alone can turn large language models into self-learning agents in dynamic environments. The authors test four open-source models on four SmartPlay games, comparing a base agent with three augmentations: self-reflection, heuristic mutation, and forward planning. They find that prompting can close the gap between small and large models on complex tasks, but that it also raises variance, sometimes sharply lowering scores. Against human baselines, they see little evidence of emergent reasoning: models can recite optimal solutions yet fail to execute them, a pattern they call a knowing-doing gap, alongside a language-embodiment gap in which text understanding does not produce grounded spatial planning. If right, static question-answering and math benchmarks overstate LLM reasoning, and dynamic environments should become a standard test bed.","feed_headline":"Little evidence that prompting creates emergent reasoning","feed_subtitle":"In SmartPlay tests, models recite optimal plans but fail at spatial tasks even with reflection and planning.","key_machinery":"The load-bearing object is a text-only agent loop on SmartPlay, a suite of text-based games in which an agent must act from natural-language observations. At each timestep the model receives a game manual, the episode history as (state, action, reward, next state) tuples, a current observation, and a list of legal actions, and must choose one. Three modules alter that prompt: Reflection appends retrospective feedback after each step; Oracle mutates a candidate set of textual heuristics between episodes and keeps a mutation only when it improves cumulative reward; Planner simulates futures up to three steps ahead and recommends the action with the highest predicted reward. Comparing these variants, along with reward-shaping and state-encoding adjustments, against the base agent and human baselines is what carries the paper's conclusion that prompting gains are task-specific, unstable, and do not amount to emergent reasoning.","core_discovery":"On the paper's own terms, the central finding is that advanced prompting does not make LLMs reliable dynamic reasoners. In the Tower of Hanoi, for example, the largest model can state the optimal seven-move sequence when prompted, yet in the game it averages about 30 moves without completing the puzzle and repeatedly makes illegal moves; in the unmodified three-disk setting, a uniform random policy can beat every prompted agent on goal completion and move legality. In Messenger, models confuse object identities and fail to navigate even when they understand the synonym task, and the Reflection + Planner module can collapse a strong model's score while helping smaller ones. The authors attribute these patterns to excessive context diluting signal, overthinking, sparse reward signals, and deficits in stateful planning and spatial coordination. They conclude that there is little evidence for emergent reasoning, framing the results as a knowing-doing gap and a language-embodiment gap that prompt engineering alone does not close.","pith_inferences":["The authors do not test this, but their random-policy result in the unmodified three-disk Tower of Hanoi suggests part of the deficit is caused by the text state representation: a structured encoding of disk stacks might recover much of the lost performance and would re-scope the 'no emergent reasoning' claim from reasoning failure to interface failure.","A concrete extension of the language-embodiment gap would be to add a rendered grid or visual map to the prompt; if Messenger navigation improves sharply, the bottleneck is grounding rather than planning.","With only three runs per condition, the large min-max spreads may be sampling noise rather than stable properties of a strategy; a many-seed replication would determine whether Reflection + Planner's RPS gains and Bandit collapses are reproducible or stochastic."],"forward_implications":["Prompt engineering alone cannot be treated as a reliable way to turn LLMs into autonomous planning agents; dynamic deployments that require spatial coordination need additional grounding, memory, or perception.","Smaller models are the main beneficiaries of advanced prompting, so comparisons of models and strategies should be reported per model size and task rather than as a single aggregate rank.","Because the same strategy's best and worst runs differ widely, reporting only median scores or accuracies hides instability; future benchmarks should report per-run distributions and failure modes.","Dense, task-aligned reward shaping can serve as a simpler substitute for elaborate prompt engineering in complex tasks, improving both two-disk Hanoi completion and Messenger message pickup.","Static reasoning benchmarks such as math word problems will overstate general reasoning ability, so dynamic, interactive tasks should be included whenever claims about reasoning are made."],"supporting_citations":[{"why":"Supplies the SmartPlay environment suite and the human baselines; without it the paper's comparisons have no task domain.","marker":"[8]"},{"why":"The methodology's stated source for the Reflection module, the core self-reflection intervention tested against baseline.","marker":"[11]"},{"why":"Cited for predicting future states and rewards from interaction history, the mechanism the Planner extends into short-horizon rollouts.","marker":"[14]"},{"why":"Cited for evolutionary prompt and metaheuristic generation, the idea behind the Oracle's textual heuristic mutation.","marker":"[16]"},{"why":"Chain-of-Thought, the static reasoning baseline the paper contrasts with dynamic-task performance.","marker":"[9]"},{"why":"Supports the signal-to-noise explanation for why long prompts degrade smaller models on simple tasks.","marker":"[17]"},{"why":"Supports the overthinking explanation for performance drops when advanced reasoning is applied to simple tasks.","marker":"[19]"},{"why":"Supplies the 'knowing-doing gap' label that names the paper's central observed deficit.","marker":"[23]"}],"fun_headline_variants":["LLMs recite plans but fail dynamic tasks","Prompting can't bridge LLM knowing-doing gap","Dynamic tests expose LLM reasoning limits","Reflection fails to spark emergent reasoning","Spatial and planning skills elude LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the SmartPlay setup—four games, a 10-step Messenger horizon, a 20-episode cap, three runs per cell, and text-only state descriptions—measures LLM reasoning neutrally rather than penalizing the interface; if a cleaner state representation, longer episode budget, or more runs lets prompted agents solve the tasks, the 'little evidence for emergent reasoning' conclusion weakens.","fun_headline_variants_meta":{"raw":{"variants":["LLMs recite plans but fail dynamic tasks","Prompting can't bridge LLM knowing-doing gap","Dynamic tests expose LLM reasoning limits","Reflection fails to spark emergent reasoning","Spatial and planning skills elude LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1367,"prompt_tokens":952,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":568,"tokens_out":415,"duration_ms":4119,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:07:04.208605+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four models on three-disk Tower of Hanoi and Messenger with a structured state encoding (for example, explicit disk-stack arrays and grid coordinates) under identical prompting strategies and many seeds: if goal-completion rises to near human baseline, the failures are interface artifacts and 'little evidence for emergent reasoning' is falsified; if scores stay near zero, the deficit is in reasoning and planning.","supporting_citations":[{"cited_title":"Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics,","cited_arxiv_id":null,"evidence_quote":"Cited for evolutionary prompt and metaheuristic generation, the idea behind the Oracle's textual heuristic mutation."},{"cited_title":"Focused transformer: Contrastive training for context scaling,","cited_arxiv_id":null,"evidence_quote":"Supports the signal-to-noise explanation for why long prompts degrade smaller models on simple tasks."}],"review_version":1}