{"id":"43ad00ef-73d3-4197-92a3-8c6c8a184e9f","arxiv_id":"2506.19095","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs, including finetuned models, can partially analyze Baba is You levels but struggle to reason about dynamic rule changes and solve puzzles.","lead":"This paper tests six large language models on a puzzle game where the rules can be changed by moving text blocks. Larger models like GPT-4o do better, but all models, including finetuned ones, struggle with the key reasoning step.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual coding of reasoning chains is the load-bearing assumption: without inter-rater reliability, the finetuning analysis-vs-solution distinction and error-frequency claims are not quantitatively grounded.","rationale":"The central claim is about LLM difficulty with dynamic rule changes, and the paper's examples and qualitative observations support that in broad strokes. However, the unique quantitative contribution is the before/after finetuning dissociation: analysis improves, solution formulation does not. This is precisely the sort of fine-grained claim that manual coding can artifactually produce if the rater knows which outputs are from finetuned models or if categories are ambiguous. The paper's own limitation statement flags exactly this. A concrete inter-rater reliability study would settle it. I agree with the reader's weakest_assumption; my recommendation is unchanged (conditional acceptance pending such a check).","tokens_in":15682,"tokens_out":4388,"duration_ms":44718,"concrete_test":"Have two independent annotators, blind to model identity and finetuning status, code a stratified sample (e.g., 150–200 reasoning chains covering all models, prompt types, levels, and pre/post finetuning) using Table 1's four steps and error subcategories. Compute Cohen's kappa (or Krippendorff's alpha) for each of the four step-level correctness labels and for each error subtype. If kappa is below 0.6 for any step that drives the finetuning comparison (analysis steps 1–2 vs solution steps 3–4), the quantitative claims in Figures 8–9 and the abstract's finetuning statement are not reliable; the paper should then be revised to present these as exploratory. If kappa is high (≥0.8), the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central qualitative claim (LLMs struggle with dynamic rule changes and the use-mention distinction) is likely robust, but the specific quantitative findings—especially that finetuning improves analysis (steps 1–2) but not solution formulation (steps 3–4), and the error-frequency comparisons in Figures 4, 8, and 9—are generated from a manual coding procedure (Section 3.1, Table 1) with no inter-rater reliability check. The authors acknowledge this in the Limitations: 'error analysis in this work was performed manually, introducing potential subjectivity.' The coding categories are not objectively decidable from the text: e.g., 'Transfer of errors' versus 'Wrong reasoning' requires causal attribution across steps; 'Incomplete information' versus 'Hallucination' in object identification can be a judgment call. Since the finetuning conclusion rests on comparing the proportion of 'correct steps' in analysis vs solution before/after finetuning, a systematic coder bias (e.g., knowing which model is finetuned) or random coding noise could produce exactly the reported pattern. Without reliability evidence, the quantitative precision of the paper's claims exceeds what the measurement supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates six large language models (GPT-4o, Gemini 1.5 Flash, Mistral 7B, Mixtral 8x7B, OLMo 7B, OLMo 13B) on puzzles from the game Baba is You, using three prompt types (simple, rule-extended, action-extended) and a zero-shot plan-and-solve protocol. In addition, Mistral 7B and OLMo 7B are finetuned with LoRA on a small dataset of game-mechanics questions and level solutions. The authors manually classify errors in model reasoning chains into four steps (level interpretation, problem-statement formulation, solution formulation, action formulation) and report error frequencies and correctness per step. The central qualitative finding is that all models, including state-of-the-art GPT-4o, struggle with dynamic rule changes, especially the use-mention distinction between moving text blocks and applying the rules they create; finetuning improves the analysis part of the reasoning chain but does not clearly improve solution generation.","tokens_in":15902,"tokens_out":3199,"duration_ms":32182,"significance":"If the core qualitative conclusion holds, the paper is a useful contribution: it identifies a concrete, language-heavy reasoning task in which current LLMs fail in a specific and interpretable way, offers publicly available prompts, finetuning datasets, and scripts, and frames the difficulty in terms of the use-mention distinction, which is a productive lens for future work. The qualitative trends (larger models do better, small models fail at object recognition, finetuning helps analysis) are plausible and supported by representative examples. However, the paper's quantitative claims rest on a manual error analysis with no inter-rater reliability, a small number of levels, and prompts tuned to GPT-4o; the quantitative precision therefore exceeds what the current measurement supports.","major_comments":[{"comment":"The load-bearing quantitative claims, in particular that finetuning improves 'analysis' (steps 1–2) but not 'solution formulation' (steps 3–4), and the error-frequency comparisons in Figures 4, 8, and 9, are generated from a manual coding procedure with no reported inter-rater reliability, no blinded coding, and no coding rubric beyond the category descriptions in Table 1. Several category boundaries are not objectively decidable from the text: for example, 'Transfer of errors' versus 'Wrong reasoning' requires causal attribution across steps, and 'Incomplete information' versus 'Hallucination' in level interpretation is a judgment call. The paper itself acknowledges in the Limitations that 'error analysis in this work was performed manually, introducing potential subjectivity.' Without reliability evidence (e.g., Cohen's kappa on a subset of chains coded by a second annotator) or at least a substantial set of fully worked example annotations, the quantitative precision of the finetuning conclusion is not supported. I recommend either adding such reliability analysis or reframing these claims as qualitative observations supported by examples rather than as quantitative findings.","section":"§3.1, Table 1; Figures 4, 8, 9; Limitations"},{"comment":"The prompt-design process is explicitly tuned to GPT-4o. This confounds the cross-model comparison: the finding that GPT-4o 'performs relatively well' and that the action-extended prompt benefits it the most may reflect prompt overfitting to that model rather than a general property of the evaluated models or prompts. The central qualitative claim about dynamic rule changes is not circular, but the specific comparative claims (e.g., that small models 'do not consistently benefit from action-extended prompts') are weaker than presented because the action-extended and rule-extended prompts may be optimized for GPT-4o's strengths. At minimum, the paper should explicitly discuss this as a limitation; a stronger approach would be to evaluate with a held-out prompt designed independently of any target model, or to verify the main comparisons with one or two additional prompt formulations.","section":"§3.1, 'Prompts were constructed manually through iterative trial and error with GPT-4o'"},{"comment":"The evaluation uses 14 levels and a 3-of-5 majority criterion. For Mistral and OLMo, the authors state that repeated runs produce identical outputs, so for these models the five runs constitute a single effective trial per level, making the accuracy measure binary rather than a stable rate. The text in Section 4.2 uses the word 'significantly' ('does not significantly improve solution formulation') without any statistical test or confidence interval. Given the small level count and the effective sample size, statements of statistical significance either need a proper test (e.g., a paired test or a confidence interval) or should be reworded to 'we did not observe a clear improvement' to avoid implying more statistical power than the design provides.","section":"§3.1 'a solution was correct if it appeared in at least three runs'; §4.2 'does not significantly improve solution…"}],"minor_comments":[{"comment":"The name 'kasparov' should be capitalized as 'Kasparov'.","section":"Section 2, first paragraph"},{"comment":"The OLMo models are cited to Jiang et al. 2023, but that reference is the Mistral 7B paper; OLMo is an AI2 model and should be cited to the appropriate OLMo documentation or paper.","section":"§3.1, description of OLMo"},{"comment":"The simple prompt text contains a typo, 'Y ou', where 'You' is intended, and the grid representation in the prompt has an escaped '\n' sequence rather than a line break; the latter may affect how the model perceives the grid and should be clarified.","section":"Figure 11 (Appendix)"},{"comment":"The action description 'Breake_Rule[GOOP IS SINK]' appears to be a typo for 'Break_Rule'; please fix for consistency with the action names described in the paper.","section":"Table 4"},{"comment":"The text says 'most outputs for the simple prompt were incorrect (results are not shown),' but it immediately references 'Figure 4, prompt 2' for object-recognition errors; it should be clarified whether Figure 4 reports prompt 1, prompt 2, or both, and why the simple-prompt results are not shown.","section":"§4.1, 'Simple prompt (1)'"}],"recommendation":"major_revision","confidential_remarks":"The paper's central qualitative claim is likely robust and the public release of prompts, datasets, and scripts is commendable. The main risk is that the quantitative narrative, particularly the finetuning 'analysis versus solution' distinction, is built on a manual coding procedure that lacks reliability evidence. If the authors add inter-rater reliability (or reframe the claims qualitatively) and address the prompt-overfitting concern, the paper could become suitable. I would not reject it; the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is a clean, reproducible extension of Cloos et al.'s 2024 result that LLMs struggle with Baba is You. The authors test six models, three prompt formats, and two LoRA-finetuned models, and they ship code, data, and model weights. The new finding that finetuning helps with level analysis (interpreting the grid, stating the problem) but not with full solution formulation is genuinely new and worth discussing. The use-mention framing gives the result a nice handle. The paper is also honest about its own limitations, which makes it easy to work with.\n\nThe soft spots are in the quantitative claims. The main analysis relies on manual coding of four reasoning steps into error subcategories (Table 1). That coding is the only source for the step-correctness frequencies in Figures 4–10, and the finetuning conclusion depends on comparing proportions of correct steps before and after finetuning. There is no inter-rater reliability check, and several category boundaries are genuinely hard to draw: is a wrong step a 'transfer of errors' or 'wrong reasoning'? Is a missing detail 'incomplete information' or a 'hallucination'? The authors acknowledge the subjectivity in the Limitations, but the claims are stated with more precision than the measurement supports. This is the main thing I would want fixed before publication.\n\nTwo smaller issues. First, the prompts were tuned iteratively with GPT-4o (Section 3.1), which biases the cross-model comparison in its favor. The qualitative conclusion (bigger models do better) is probably robust, but the exact ordering should be treated cautiously. Second, the finetuning dataset contains only 15 handwritten level solutions, and the paper never states whether those levels overlap with the 14 evaluation levels. If any do, the finetuning improvement is partly a memorization effect. At minimum this must be clarified. I also note that the described solution accuracy metric (five runs, majority vote) is not actually reported in the main text; the step-correctness figures carry the argument.\n\nNone of this destroys the paper's central qualitative claim: even state-of-the-art LLMs, and models finetuned on game data, have trouble reasoning about rule changes and the use-mention distinction. The examples in Tables 4 and 5 are enough to make that point. So the paper is a solid empirical contribution that needs revision, not a reject.\n\nMy recommendation: send it to peer review, but require at least a small reliability study for the manual coding, a human baseline, and a clear statement about train/eval level overlap. If the authors can do that, the finetuning analysis-vs-solution distinction becomes much more credible.","headline":"A useful empirical follow-up on LLMs and dynamic-rule games; the qualitative finding is credible, but the headline finetuning claim needs a reliability check on the manual coding.","tokens_in":16415,"tokens_out":4586,"would_cite":true,"duration_ms":40693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that large language models, including state-of-the-art GPT-4o and models finetuned on game data, still fail to reason about dynamic rule changes in the puzzle game Baba is You, with the core difficulty being the…","keywords":["Baba is You","large language models","dynamic rule changes","use-mention distinction","chain-of-thought reasoning","finetuning","puzzle games","LLM reasoning"],"falsifier":"Present a frontier LLM with the same 14 levels but after each action feed it the updated active rules (oracle rule-state updates) and keep the prompt otherwise identical; if the model still fails the rule-creation and rule-breaking levels, then the limitation is not tracking dynamic rule changes but a more basic gap in applying rules to spatial states.","tokens_in":15504,"feed_emoji":"🧩","tokens_out":9802,"duration_ms":86489,"temperature":0.7,"pith_summary":"Baba is You is a puzzle game where the rules of the level are written on tiles that the player can push around, so the same text block acts both as a symbol to move and as a rule that changes the game state. The paper uses this game to test whether large language models can reason under dynamic rule changes. It evaluates six LLMs across three prompt designs and finetunes two small models on game-specific data, manually coding the errors in each model's reasoning chain. The central finding is that even a state-of-the-art model like GPT-4o solves parts of the levels but systematically fails when rule-breaking or rule-creation is required, and that finetuning improves a model's level analysis without improving its ability to produce working solutions. The authors conclude that the use-mention distinction—understanding that pushing a text block is not the same as applying the rule it forms—is a bottleneck for current LLMs.","feed_headline":"Even top LLMs fail when pushing words rewrites game rules","feed_subtitle":"Pushing text blocks rewrites the rules, and even the largest models miss how the new rule applies.","key_machinery":"The load-bearing mechanism is the game's two-level rule system, interpreted through the classical use-mention distinction. In Baba is You, a text block in a level grid is simultaneously an object to be pushed (mention) and a constituent of a rule when aligned with other blocks (use); the rule then changes object behaviors, such as turning a rock into a flag or making lava melt the player's avatar. The paper operationalizes this by encoding levels as ASCII grids, defining a four-step reasoning chain (level interpretation, problem statement, solution formulation, action formulation) with categorized error types, and comparing three prompt formats that progressively add active rules and allowed actions, plus LoRA finetuning on game and reasoning datasets. The error taxonomy is what turns the game into a measurable probe of reasoning: it lets the authors show where in the chain each model fails.","core_discovery":"The paper's central claim is that reasoning about dynamic rule changes is difficult for LLMs, specifically at the use-mention distinction: when a text block is part of a rule, the model must both recognize the block as a game object that can be pushed (mention) and compute the new game-state consequences of the rule it forms when aligned (use). Across 14 levels, GPT-4o and Gemini 1.5 Flash outperformed smaller models, but even they frequently proposed breaking unbreakable rules, overlooked spatial constraints, and failed to distinguish objects from text blocks. Finetuning Mistral 7B and OLMo 7B on game questions and solutions improved their ability to identify objects and state the problem, but did not significantly improve the final action sequences, which remained riddled with errors. The authors interpret this as evidence that neither scale nor in-domain finetuning is sufficient to acquire dynamic rule reasoning, and that the game offers a challenging, compact testbed for reasoning and reflection in LLMs.","pith_inferences":["The failure pattern likely generalizes beyond games: any setting where a symbol is both an instruction and a manipulable object—such as code, where an identifier is both text and a reference—should exhibit the same use-mention bottleneck; this is testable by asking an LLM to edit its own rule text and predict the outcome before execution.","If the bottleneck is rule-state tracking, then providing the model with an explicit, externally updated rule set after each move (oracle feedback) should markedly improve performance; if it does not, the bottleneck lies deeper, in applying rules to spatial states.","The manual error taxonomy suggests a finer-grained test: measuring inter-rater agreement on the four steps would quantify how much of the reported improvement after finetuning is attributable to genuinely better reasoning rather than to labeler expectations.","One could push further: a hybrid agent that combines an LLM for natural-language planning with a symbolic simulator that enforces rule mechanics would likely solve these levels, which would show that the game tests a missing integration rather than an unreachable capability."],"forward_implications":["If the finding holds, dynamic-rule reasoning is a distinct capability that scale alone does not unlock: the largest and most capable model still fails at rule-breaking and rule-creation when spatial constraints matter.","Prompt augmentation that hands the model the active rules and the allowed actions improves problem analysis but does not reliably improve action generation, implying that richer prompting cannot substitute for the missing use-mention competence.","Finetuning on game-related textual and structural data improves a model's descriptive analysis of a level but not its constructive solution, suggesting that in-domain data helps recognition before it helps planning.","The grid is systematically misread as a one-dimensional or static space: models treat 'FLAG IS WIN' as proof a flag exists, and treat 'WIN' as an object; the paper's error analysis locates these failures in specific chain steps.","The game is proposed as a compact benchmark for reasoning and reflection in LLMs, suited to evaluating models' ability to track and apply changing rules."],"supporting_citations":[{"why":"Defines the game Baba is You and its rule-creation mechanics.","marker":"[Teikari, 2019]"},{"why":"Prior study that already found state-of-the-art LLMs struggle with the game's reasoning, which this work extends.","marker":"[Cloos et al., 2024]"},{"why":"Provides the simplified game module and mechanics used to encode levels as ASCII grids.","marker":"[Charity et al., 2020]"},{"why":"Supplies the Keke AI competition levels used for evaluation.","marker":"[Charity and Togelius, 2022]"},{"why":"Documents the GPT-4o model that is the main state-of-the-art comparator.","marker":"[Hurst et al., 2024]"},{"why":"Documents the Gemini 1.5 model used as the second strong model.","marker":"[Gemini, 2024]"},{"why":"Introduces Mistral 7B, one of the base models evaluated and finetuned.","marker":"[Jiang et al., 2023]"},{"why":"Introduces Plan-and-Solve prompting, the zero-shot CoT method used in all prompts.","marker":"[Wang et al., 2023b]"},{"why":"Introduces LoRA, the parameter-efficient finetuning method applied to Mistral and OLMo.","marker":"[Hu et al., 2021]"},{"why":"Provides the CoT-logic-reasoning dataset used alongside game-specific data in finetuning.","marker":"[Bjorklund, 2025]"}],"fun_headline_variants":["LLMs can't grasp when game rules change mid-play","Finetuning helps LLMs analyze game but not solve it","When pushing words, LLMs can't see rule vs object","Dynamic rules trip up even fine-tuned LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative claims rest on the manual categorization of each model's reasoning chain into four steps and many error subtypes; if that coding is not reproducible by other raters, the reported error frequencies and finetuning gains lose their precision.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't grasp when game rules change mid-play","Finetuning helps LLMs analyze game but not solve it","When pushing words, LLMs can't see rule vs object","Dynamic rules trip up even fine-tuned LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001034,"raw_usage":{"total_tokens":4368,"prompt_tokens":975,"completion_tokens":3393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":3325}},"tokens_in":591,"tokens_out":3393,"duration_ms":23939,"temperature":1.0,"reasoning_tokens":3325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:36:03.698608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present a frontier LLM with the same 14 levels but after each action feed it the updated active rules (oracle rule-state updates) and keep the prompt otherwise identical; if the model still fails the rule-creation and rule-breaking levels, then the limitation is not tracking dynamic rule changes but a more basic gap in applying rules to spatial states.","supporting_citations":[],"review_version":2}