Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Baba is LLM: Reasoning in a Game with Dynamic Rules

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper establishes that large language models, including state-of-the-art GPT-4o and models finetuned on game data, still fail to reason about dynamic rule changes in the puzzle game Baba is You, with the core difficulty being the…

desk verdict A useful empirical follow-up on LLMs and dynamic-rule games; the qualitative finding is credible, but the headline finetuning claim needs a reliability check on the manual coding. read the letter →

arxiv 2506.19095 v1 pith:M6BJ42EQ submitted 2025-06-23 cs.AI

Fien van Wetten , Aske Plaat , Max van Duijn This is my paper · ORCID
classification cs.AI
keywords BabaisYoulargelanguagemodelsdynamicrulechangesuse-mentiondistinctionchain-of-thoughtreasoningfinetuningpuzzlegamesLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Baba is You is a puzzle game where the rules of the level are written on tiles that the player can push around, so the same text block acts both as a symbol to move and as a rule that changes the game state. The paper uses this game to test whether large language models can reason under dynamic rule changes. It evaluates six LLMs across three prompt designs and finetunes two small models on game-specific data, manually coding the errors in each model's reasoning chain. The central finding is that even a state-of-the-art model like GPT-4o solves parts of the levels but systematically fails when rule-breaking or rule-creation is required, and that finetuning improves a model's level analysis without improving its ability to produce working solutions. The authors conclude that the use-mention distinction—understanding that pushing a text block is not the same as applying the rule it forms—is a bottleneck for current LLMs.

What carries the argument

The load-bearing mechanism is the game's two-level rule system, interpreted through the classical use-mention distinction. In Baba is You, a text block in a level grid is simultaneously an object to be pushed (mention) and a constituent of a rule when aligned with other blocks (use); the rule then changes object behaviors, such as turning a rock into a flag or making lava melt the player's avatar. The paper operationalizes this by encoding levels as ASCII grids, defining a four-step reasoning chain (level interpretation, problem statement, solution formulation, action formulation) with categorized error types, and comparing three prompt formats that progressively add active rules and allowed actions, plus LoRA finetuning on game and reasoning datasets. The error taxonomy is what turns the game into a measurable probe of reasoning: it lets the authors show where in the chain each model fails.

What would settle it

Present a frontier LLM with the same 14 levels but after each action feed it the updated active rules (oracle rule-state updates) and keep the prompt otherwise identical; if the model still fails the rule-creation and rule-breaking levels, then the limitation is not tracking dynamic rule changes but a more basic gap in applying rules to spatial states.

Watch

Extended reading notes

Core claim

The paper's central claim is that reasoning about dynamic rule changes is difficult for LLMs, specifically at the use-mention distinction: when a text block is part of a rule, the model must both recognize the block as a game object that can be pushed (mention) and compute the new game-state consequences of the rule it forms when aligned (use). Across 14 levels, GPT-4o and Gemini 1.5 Flash outperformed smaller models, but even they frequently proposed breaking unbreakable rules, overlooked spatial constraints, and failed to distinguish objects from text blocks. Finetuning Mistral 7B and OLMo 7B on game questions and solutions improved their ability to identify objects and state the problem, but did not significantly improve the final action sequences, which remained riddled with errors. The authors interpret this as evidence that neither scale nor in-domain finetuning is sufficient to acquire dynamic rule reasoning, and that the game offers a challenging, compact testbed for reasoning and reflection in LLMs.

Load-bearing premise

The quantitative claims rest on the manual categorization of each model's reasoning chain into four steps and many error subtypes; if that coding is not reproducible by other raters, the reported error frequencies and finetuning gains lose their precision.

Editorial extensions

If this is right

  • If the finding holds, dynamic-rule reasoning is a distinct capability that scale alone does not unlock: the largest and most capable model still fails at rule-breaking and rule-creation when spatial constraints matter.
  • Prompt augmentation that hands the model the active rules and the allowed actions improves problem analysis but does not reliably improve action generation, implying that richer prompting cannot substitute for the missing use-mention competence.
  • Finetuning on game-related textual and structural data improves a model's descriptive analysis of a level but not its constructive solution, suggesting that in-domain data helps recognition before it helps planning.
  • The grid is systematically misread as a one-dimensional or static space: models treat 'FLAG IS WIN' as proof a flag exists, and treat 'WIN' as an object; the paper's error analysis locates these failures in specific chain steps.
  • The game is proposed as a compact benchmark for reasoning and reflection in LLMs, suited to evaluating models' ability to track and apply changing rules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The failure pattern likely generalizes beyond games: any setting where a symbol is both an instruction and a manipulable object—such as code, where an identifier is both text and a reference—should exhibit the same use-mention bottleneck; this is testable by asking an LLM to edit its own rule text and predict the outcome before execution.
  • If the bottleneck is rule-state tracking, then providing the model with an explicit, externally updated rule set after each move (oracle feedback) should markedly improve performance; if it does not, the bottleneck lies deeper, in applying rules to spatial states.
  • The manual error taxonomy suggests a finer-grained test: measuring inter-rater agreement on the four steps would quantify how much of the reported improvement after finetuning is attributable to genuinely better reasoning rather than to labeler expectations.
  • One could push further: a hybrid agent that combines an LLM for natural-language planning with a symbolic simulator that enforces rule mechanics would likely solve these levels, which would show that the game tests a missing integration rather than an unreachable capability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates six large language models (GPT-4o, Gemini 1.5 Flash, Mistral 7B, Mixtral 8x7B, OLMo 7B, OLMo 13B) on puzzles from the game Baba is You, using three prompt types (simple, rule-extended, action-extended) and a zero-shot plan-and-solve protocol. In addition, Mistral 7B and OLMo 7B are finetuned with LoRA on a small dataset of game-mechanics questions and level solutions. The authors manually classify errors in model reasoning chains into four steps (level interpretation, problem-statement formulation, solution formulation, action formulation) and report error frequencies and correctness per step. The central qualitative finding is that all models, including state-of-the-art GPT-4o, struggle with dynamic rule changes, especially the use-mention distinction between moving text blocks and applying the rules they create; finetuning improves the analysis part of the reasoning chain but does not clearly improve solution generation.

Significance. If the core qualitative conclusion holds, the paper is a useful contribution: it identifies a concrete, language-heavy reasoning task in which current LLMs fail in a specific and interpretable way, offers publicly available prompts, finetuning datasets, and scripts, and frames the difficulty in terms of the use-mention distinction, which is a productive lens for future work. The qualitative trends (larger models do better, small models fail at object recognition, finetuning helps analysis) are plausible and supported by representative examples. However, the paper's quantitative claims rest on a manual error analysis with no inter-rater reliability, a small number of levels, and prompts tuned to GPT-4o; the quantitative precision therefore exceeds what the current measurement supports.

major comments (3)
  1. [§3.1, Table 1; Figures 4, 8, 9; Limitations] The load-bearing quantitative claims, in particular that finetuning improves 'analysis' (steps 1–2) but not 'solution formulation' (steps 3–4), and the error-frequency comparisons in Figures 4, 8, and 9, are generated from a manual coding procedure with no reported inter-rater reliability, no blinded coding, and no coding rubric beyond the category descriptions in Table 1. Several category boundaries are not objectively decidable from the text: for example, 'Transfer of errors' versus 'Wrong reasoning' requires causal attribution across steps, and 'Incomplete information' versus 'Hallucination' in level interpretation is a judgment call. The paper itself acknowledges in the Limitations that 'error analysis in this work was performed manually, introducing potential subjectivity.' Without reliability evidence (e.g., Cohen's kappa on a subset of chains coded by a second annotator) or at least a substantial set of fully worked example annotations, the quantitative precision of the finetuning conclusion is not supported. I recommend either adding such reliability analysis or reframing these claims as qualitative observations supported by examples rather than as quantitative findings.
  2. [§3.1, 'Prompts were constructed manually through iterative trial and error with GPT-4o'] The prompt-design process is explicitly tuned to GPT-4o. This confounds the cross-model comparison: the finding that GPT-4o 'performs relatively well' and that the action-extended prompt benefits it the most may reflect prompt overfitting to that model rather than a general property of the evaluated models or prompts. The central qualitative claim about dynamic rule changes is not circular, but the specific comparative claims (e.g., that small models 'do not consistently benefit from action-extended prompts') are weaker than presented because the action-extended and rule-extended prompts may be optimized for GPT-4o's strengths. At minimum, the paper should explicitly discuss this as a limitation; a stronger approach would be to evaluate with a held-out prompt designed independently of any target model, or to verify the main comparisons with one or two additional prompt formulations.
  3. [§3.1 'a solution was correct if it appeared in at least three runs'; §4.2 'does not significantly improve solution…] The evaluation uses 14 levels and a 3-of-5 majority criterion. For Mistral and OLMo, the authors state that repeated runs produce identical outputs, so for these models the five runs constitute a single effective trial per level, making the accuracy measure binary rather than a stable rate. The text in Section 4.2 uses the word 'significantly' ('does not significantly improve solution formulation') without any statistical test or confidence interval. Given the small level count and the effective sample size, statements of statistical significance either need a proper test (e.g., a paired test or a confidence interval) or should be reworded to 'we did not observe a clear improvement' to avoid implying more statistical power than the design provides.
minor comments (5)
  1. [Section 2, first paragraph] The name 'kasparov' should be capitalized as 'Kasparov'.
  2. [§3.1, description of OLMo] The OLMo models are cited to Jiang et al. 2023, but that reference is the Mistral 7B paper; OLMo is an AI2 model and should be cited to the appropriate OLMo documentation or paper.
  3. [Figure 11 (Appendix)] The simple prompt text contains a typo, 'Y ou', where 'You' is intended, and the grid representation in the prompt has an escaped ' ' sequence rather than a line break; the latter may affect how the model perceives the grid and should be clarified.
  4. [Table 4] The action description 'Breake_Rule[GOOP IS SINK]' appears to be a typo for 'Break_Rule'; please fix for consistency with the action names described in the paper.
  5. [§4.1, 'Simple prompt (1)'] The text says 'most outputs for the simple prompt were incorrect (results are not shown),' but it immediately references 'Figure 4, prompt 2' for object-recognition errors; it should be clarified whether Figure 4 reports prompt 1, prompt 2, or both, and why the simple-prompt results are not shown.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark study whose claims rest on external levels and model outputs, not on derived predictions.

full rationale

The paper contains no derivations in which an output is constructed from its own target. Its three prompt variants are fixed text templates evaluated on 14 levels drawn from the external Keke AI competition ('Most of these levels are demo levels of the Keke AI competition [Charity and Togelius, 2022], except level 14'), so the difficulty of dynamic rule changes is not baked in by a fitting procedure. The finetuning datasets are described as containing 'different levels of Baba is You' and are separate from the evaluation-level set; even if overlap existed, the central claim is a comparative performance observation, not a quantitative prediction forced by fitted parameters. The use-mention distinction is an interpretive frame applied after the fact, not an input that determines the measured error frequencies. The prompt tuning with GPT-4o ('The prompts were constructed manually through iterative trial and error with GPT-4o. Outputs were reviewed for improvement, refined, and resubmitted until a satisfactory version was achieved.') could inflate GPT-4o's absolute score, but it would only weaken the paper's central claim that GPT-4o still struggles with dynamic rule changes; it does not make the claim circular. The acknowledged manual coding limitation ('error analysis in this work was performed manually, introducing potential subjectivity') is a measurement-reliability concern, not a circularity concern. Self-citations to Plaat et al. appear only in contextual survey references and data-availability links, and none is load-bearing for the central result. No circular step can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper makes no quantitative derivations, so there are no fitted free parameters or invented entities. Its claims rest on several domain assumptions about the adequacy of the prompts, the representativeness of the levels, and the reliability of the manual error analysis, all of which are acknowledged in the Limitations.

assumptions (4)
  • domain assumption The simplified game representation and the three prompt templates adequately convey game mechanics to all models, so failures reflect reasoning ability rather than an ambiguous interface.
    The entire evaluation rests on this premise; Section 3.1 describes prompt construction but does not validate that the prompts are equally clear across models.
  • domain assumption The manual error categorization is valid and applied consistently.
    The quantitative results in Figures 4-9 depend entirely on this manual coding; no inter-rater reliability is reported.
  • domain assumption The 14 selected levels are representative of dynamic rule changes and are easy enough for humans that failures are meaningful.
    Section 3.1 says the levels are 'relatively easy for humans,' but no human baseline is reported to support that assumption.
  • domain assumption The finetuning dataset of 15 level examples and 289 mechanics questions is sufficient to test whether finetuning improves solution formulation.
    The claim that finetuning does not significantly improve solution formulation could be an artifact of the tiny dataset; the paper acknowledges this in Limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Baba is LLM: Reasoning in a Game with Dynamic Rules." pith.science (2026). https://pith.science/paper/M6BJ42EQ

@misc{pith2026250619095,
  author       = {Pith},
  title        = {Pith review of: Baba is LLM: Reasoning in a Game with Dynamic Rules},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6BJ42EQ}},
  note         = {Machine review of arXiv:2506.19095}
}
read the original abstract

Large language models (LLMs) are known to perform well on language tasks, but struggle with reasoning tasks. This paper explores the ability of LLMs to play the 2D puzzle game Baba is You, in which players manipulate rules by rearranging text blocks that define object properties. Given that this rule-manipulation relies on language abilities and reasoning, it is a compelling challenge for LLMs. Six LLMs are evaluated using different prompt types, including (1) simple, (2) rule-extended and (3) action-extended prompts. In addition, two models (Mistral, OLMo) are finetuned using textual and structural data from the game. Results show that while larger models (particularly GPT-4o) perform better in reasoning and puzzle solving, smaller unadapted models struggle to recognize game mechanics or apply rule changes. Finetuning improves the ability to analyze the game levels, but does not significantly improve solution formulation. We conclude that even for state-of-the-art and finetuned LLMs, reasoning about dynamic rule changes is difficult (specifically, understanding the use-mention distinction). The results provide insights into the applicability of LLMs to complex problem-solving tasks and highlight the suitability of games with dynamically changing rules for testing reasoning and reflection by LLMs.

Figures

Figures reproduced from arXiv: 2506.19095 by the authors.

Figure 1
Figure 1. ASCII representation of a level; Pictorial representation; By pushing [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Examples of different game mechanics using the flag as a win condition. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Levels used for the evaluation of the LLM models in playing [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Frequency of error step and subcategory (see Table [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Correctness frequency per step in the reasoning chain generated by the [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Correct steps per model across the 14 Baba is You levels with the rule￾extended prompt. The rule-extended prompt, which provides the active rules present in the level, improves performance across models but still highlights major differences in reasoning capabilities. …
Figure 7
Figure 7. Figure 7: Correct steps per model across the 14 Baba is You levels with the action￾extended prompt. The action-extended prompt, which provides additional details about possible actions, leads to notable improvements for some models, partic￾ularly Gemini 1.5 Flash and GPT-4o. How…
Figure 8
Figure 8. Figure 8: Frequency of errors per step and subcategory in the reasoning chains [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Correctness frequency per step in the reasoning chain generated by the [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Correct steps per model across the 14 Baba is You levels with the action [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Simple prompt: consisting only of a short game description and defi￾nitions of the characters and rules. Followed by a question to solve a level with at the end a sentence to activate zero-shot CoT. The Rule-extended prompt and the Action-extended prompt can be found …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 18 canonical work pages

  1. [1]

    GOOP IS SINK

    **Breake_Rule[GOOP IS SINK]:** Remove the rule "GOOP IS SINK." This will prevent the Goop from destroying itself

    work page

  2. [2]

    ROCK IS HOT

    **Make_Rule[ROCK IS HOT]:** Create the rule "ROCK IS HOT." This will make the Rocks hot

    work page

  3. [3]

    arXiv:2210.03350 [cs]

    URL http://arxiv.org/abs/2210.03350. arXiv:2210.03350 [cs]. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018. Paul Saka. Quotation and the use-mention distinction.Mind, 107(425):113–135, 1998. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Lauren...

    arXiv 2018

  4. [4]

    WALL IS STOP

    URL http://arxiv.org/abs/1706.03762. arXiv:1706.03762 [cs]. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought rea- soning by large language models.arXiv preprint arXiv:2305.04091, 2023a. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei...

    arXiv 2022

  5. [7]

    GOOP IS MELT

    **Make_Rule[GOOP IS MELT]:** Create the rule "GOOP IS MELT." This will make the Goop melt when it touches the hot Rocks. Table 4: Examples of error snippets in the reasoning chains of GPT-4o and Gemini 1.5 Flash LLM models, where the red-colored text indicates incorrect reasoning. These errors highlight challenges such as hallucinations, incorrect as- sum...

    work page

  6. [8]

    Make BABA IS YOU to BABA IS MOVE

    work page

  7. [9]

    Move BABA to the position of the SKULL

    work page

  8. [10]

    Level 4:Step 2: Extract the relevant objects, text blocks and rules - Objects: BABA, SKULL, FLAG, FLOOR, GRASS, ROCK, WALL, KEKE, GOOP, LOVE, and empty space

    Make SKULL IS YOU. Level 4:Step 2: Extract the relevant objects, text blocks and rules - Objects: BABA, SKULL, FLAG, FLOOR, GRASS, ROCK, WALL, KEKE, GOOP, LOVE, and empty space. - Text blocks: BABA, IS, YOU, FLAG, IS, WIN. - Rules: BABA IS YOU, FLAG IS WIN. Step 3: Devise a plan - Make BABA the object you control. - Make FLAG the object you need to reach ...

    work page

Show all 22 references
  1. [11]

    Move the BABA to the flag by pushing the blocks in the way

  2. [12]

    Make the BABA move towards the flag

  3. [13]

    Level 3:Plan:

    Make the BABA push the flag towards itself. Level 3:Plan:

  4. [14]

    Move BABA to the 1 text block

  5. [15]

    Change BABA into the flag (f) using the 1 text block

  6. [16]

    Level 9: The goal is to move the flag to the exit (position B12) without getting killed by the skull

    Move the flag to the exit (the empty space next to the 3 text block). Level 9: The goal is to move the flag to the exit (position B12) without getting killed by the skull. OLMo 13B Level 3:**Objective:** Reach the WIN block to win the level. Level 4:**Plan:**

  7. [17]

    Level 9:1

    **Make Baba movable:** Ensure Baba has the "MOVE" property. Level 9:1. **Neutralize SKULL:** - We cannot directly push SKULL because it will kill BABA. - We need to use text blocks to change SKULL’s properties. - **Make Rule:** SKULL IS STOP. This will make SKULL impassable an...

  8. [18]

    **Identify the Goal:** Move BABA to the FLAG (F13)

  9. [19]

    Level 2:2

    **Current Obstacles:** BABA starts at B12, and there are walls (W) and empty spaces (.) blocking the path. Level 2:2. **Initial Setup:** - BABA is currently next to a WALL (W) and a KEKE (K). - There’s a FLAG (F) adjacent to BABA. Level 9:### Plan:

  10. [20]

    **Move BABA away from SKULL:** To avoid immediate death, move BABA away from SKULL

  11. [21]

    **Create a path to FLAG:** Use the environment to create a path to FLAG

  12. [22]

    These errors highlight challenges such as hallucinations, incorrect assumptions, and wrong reasoning steps

    **Utilize GOOP:** GOOP can be used to block SKULL or create a bridge Table 5: Examples of error snippets in the reasoning chains of OLMo and Mistral LLM models, where the red-colored text indicates incorrect reasoning. These errors highlight challenges such as hallucinations, ...

  13. [2019]

    arXiv:1912.06680 [cs]

    URL http://arxiv.org/abs/1912.06680. arXiv:1912.06680 [cs]. Isaiah Bjorklund. cot-logic-reasoning, 2025. URL https://huggingface.co/datasets/ isaiahbjork/cot-logic-reasoning. Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professi...

    1912 arXiv

  14. [2023]

    arXiv:2310.06825 [cs]

    URL http://arxiv.org/abs/2310.06825. arXiv:2310.06825 [cs]. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, et al....

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.