REVIEW 4 major objections 5 minor 2 references
Haunted House: A text-based game for comparing the flexibility of mental models in humans and LLMs
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that current LLMs largely fail at an interactive clue-driven navigation task that about a third of humans solve, indicating weak active model-based reasoning.
desk verdict A useful new benchmark with a large human-LLM gap, but the mental-model interpretation is underdetermined without a chance-level baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Haunted House game itself: a 3x3 grid of nine rooms, a starting room C1 whose identity is hidden from the player, a key hidden in A1, and a ghost that remains stationary until the key is found and then moves according to deterministic rules. Players receive a verbal clue after each move—"nearby" means an adjacent room contains the key or ghost—and must use those clues to infer their location, find the key, return to the start, and navigate to a relocated exit at A3 without stepping into the ghost's room. The game is designed so that solving it requires building a mental model of the house from partial observations and updating that model as new clues arrive and the ghost moves; the paper measures model-based reasoning through sub-objective completion and error categories: illegal moves indicating self-location failure, random guessing before the key is found, and not tracking the ghost's movement afterward.
What would settle it
If a random-move baseline that simply chooses uniformly among legal directions until death or the 20-move limit matched the LLM pass rate and error distribution, then the pass-rate gap alone would not show that LLMs lack model-based reasoning. Likewise, if an LLM given a complete map and explicit ghost-movement rules still made illegal moves and lost track of the ghost at the same rate as without those aids, the mental-model explanation would be weakened.
Extended reading notes
Core claim
The central discovery is that a simple-looking, clue-driven navigation task that roughly a third of ordinary people solve almost entirely defeats current frontier LLMs. Across seven models and 140 attempts, only Claude 3 Opus completed the game once; even the strongest models typically failed before finding the key, and the few that located the key often could not retrace their steps or track the ghost. Later trials with newer models such as GPT o3-mini-high produced one successful escape in ten attempts, still below the human rate. Analysis of move logs shows the LLM failures are not just rare bad luck: they systematically include illegal moves, random guesses with a 50% chance of losing, and a failure to update the ghost's location after it begins moving. The authors conclude that current LLMs struggle with tasks demanding active model-based reasoning, in which partial information, dynamic variables, and verbal clues must be integrated into a coherent internal model.
Load-bearing premise
The load-bearing premise is that the error categories derived from moves—illegal moves, random guessing, and failing to track the ghost—diagnose failures to construct or update mental models, rather than instruction-following or interface artifacts.
Editorial extensions
If this is right
- The task provides an interactive benchmark where humans outperform frontier LLMs by a wide margin: 31.6% versus 1/140 across seven models.
- LLM failures concentrate early: most models never find the key, and the few that do often fail the simple sub-goal of retracing their steps to the starting room.
- Error analysis shows LLMs are prone to illegal moves, random high-risk guesses, and loss of the ghost's position, implicating failed mental-model construction rather than a lack of world knowledge alone.
- Modified instructions—explicitly stating the ghost is stationary and giving players coordinates—improve key-finding but not full-game escape, suggesting the deficit is in model updating, not just spatial mapping.
- Newer reasoning models show preliminary improvement, with GPT o3-mini-high completing the task in one of ten attempts, but still not at the human level.
Reading between the lines
- A targeted probe could separate navigation from updating: if an LLM is shown an explicit map after every move and still fails to avoid the ghost, the bottleneck would be specifically in updating a dynamic world model rather than in initial orientation.
- The 'nearby' clue structure converts the game into a graph-search problem with local gradient information; standard algorithms would solve it, so the LLM failures suggest a gap between linguistic output and a usable internal spatial representation.
- One cheap diagnostic derived from the paper's error taxonomy is the rate at which a player voluntarily returns to the starting room after a first move; human solvers do this, while LLMs often prefer a 50% guess, and this single behavioral metric could be used as a fast mental-model probe.
- If future LLMs close the pass-rate gap, the game could still serve as a measure of how well models handle partial information and dynamic state, complementing static benchmarks that reward recall rather than active inquiry.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces Haunted House, a text-based 3x3 grid navigation game in which players locate a key, return to the start, and reach a relocated exit while avoiding a ghost, using verbal clues updated after each move. In Study 1, 31 of 98 human participants (31.6%) completed the task, while across 140 attempts by seven LLMs only a single Claude 3 Opus attempt passed. Study 2 compares the move-level behavior of 29 additional humans with LLM logs using sub-objective completion and a three-category error taxonomy. Two follow-up instruction modifications ('Ghost' and 'Coordinates') are tested on selected models. The authors conclude that current LLMs have difficulty with tasks that demand active model-based reasoning.
Significance. If the empirical pass-rate gap holds, the task is a useful addition to the growing set of interactive benchmarks on which frontier LLMs underperform non-expert humans. The paper's strengths are the clearly specified game mechanics, the public playable implementation, the inclusion of human comparison samples in two studies, and the transparent reporting of instruction-related confounds and preliminary newer-model results. The main value is as a falsifiable behavioral benchmark; the construct-level interpretation, however, currently outruns the measurements because no no-model baseline or direct mental-model probe is provided.
major comments (4)
- [Abstract; Study 1 Results] The paper's central explanatory claim — that LLM failures reflect difficulty with active model-based reasoning — is not supported without a chance-level baseline under the same game rules. The authors report a 31.6% human pass rate versus 1/140 LLM attempts, but they never report the pass rate of a policy that makes legal moves uniformly at random without building or updating any map. Because the ghost remains at B2 until the key is found, the key is at A1, and the optimal route is short (the paper's Figure 2 illustrates a roughly fourteen-move solution), a memory-less random legal-move policy can in principle complete all sub-objectives by luck. If that baseline is comparable to or above the LLM pass rates, the poor LLM performance could be explained by output-formatting, instruction-following, or dialogue-interface failures rather than by absent mental models. Reporting this baseline, with confidence intervals, is necessary to justify the Abstract's causal wording and the General Discussion's claim that LLM failure is 'best explained by their inability to construct working mental models.'
- [Study 2 Method (error taxonomy)] The error taxonomy codifies the mental-model account rather than testing it. The three categories — failure to locate oneself, random guessing, and not tracking the ghost — are defined 'based on the moves alone' from the authors' normative solution path, so deviations are counted as mental-model failures by construction. For instance, moving from B1 to A1 after the clue 'The ghost is nearby. The key is nearby.' is classified as a random guess even though a risk-taking player could deliberately choose that gamble; conversely, a player could hold a correct mental model and still guess because the clues make the choice uncertain. Analyses such as 'LLMs frequently struggled with random and illogical moves' therefore inherit the assumption that the only rational strategy is the authors' safe backtracking plan. The paper needs either an independent measure of model-based reasoning (for example, state-reconstruction probes asking where the ghost or key is) or a more conservative framing of what the error categories establish.
- [General Discussion (third supportive evidence)] The third supportive evidence contains an internal contradiction. The Results section for the 'Coordinates' condition states that 'Both Claude 3 Opus and GPT o1-preview found the key in at least half of the attempts' and that only one of 20 attempts resulted in passing the entire game, but the General Discussion says the modification did not improve performance, 'with only one attempt out of 20 successfully finding the key.' Since finding the key improved markedly under coordinate instructions, the claim that a simplified version did not help LLMs is not supported by the reported data. This sentence should be corrected to refer to passing the game, and the interpretation updated accordingly.
- [Study 1 Results] The phrase 'significantly outperforming' appears without any inferential statistic. The authors report 31/98 human successes versus 1/140 LLM attempts, but no confidence interval, hypothesis test, or effect size is given for the human–LLM comparison or for the Study 2 sub-objective and error-rate comparisons. Adding exact binomial confidence intervals (and, if appropriate, a mixed-effects model accounting for 20 repeated attempts per model) would make the strength of the evidence assessable and would clarify statements such as 'o1-preview with a pass rate of 20%' in Table 1, which is based on 4 of 20 attempts.
minor comments (5)
- [Study 1 Method (Participants)] The reported sex counts sum to 99 (65 female and 34 male) while the stated n is 98; please reconcile the counts or the total.
- [Study 2 Method (Participants)] An 11-year-old participant is included without comment; if this is intentional, please justify the inclusion or exclude the participant for comparability with the adult samples.
- [Study 2 Results] There are grammar errors such as 'the model failed to locate themself'; the paper would benefit from a careful proofreading pass.
- [Figure 1 and Study 2 Procedure] Please state explicitly whether the human interface displayed a visual grid or only text; this matters because the LLM condition received only textual directions and clues, and the description in the Procedure section is ambiguous.
- [References and terminology] Model names are written inconsistently (e.g., 'GPT 4-o' versus 'GPT-4o'); standardize model names throughout.
Circularity Check
No significant circularity: the paper is an empirical benchmark whose main pass-rate result rests on raw game outcomes; the mental-model interpretation is an inference, not a definitional reduction.
full rationale
The paper contains no fitted parameters, no predictive equations, no uniqueness theorems, and no load-bearing self-citations. The central empirical claim—that 31.6% of humans solved the Haunted House task while only 1 of 140 LLM attempts succeeded—is a direct report of raw game outcomes, not a quantity derived from an input that already encodes the result. The task design and the modified-instruction controls are transparent about the experimental manipulations. The only potentially circular-looking passage is the Study 2 error taxonomy: categories such as “random guessing” and “failure to track the ghost” are coded from the authors’ normative solution path, and the General Discussion then cites these errors as support for the claim that LLM failures are “best explained by their inability to construct working mental models.” That is an interpretive validity concern—an alternative explanation such as interface effects or missing chance-level baselines is not ruled out—but it is not circularity in the stipulated sense, because the conclusion is not equivalent to the coding definition by construction. The error categories are behavioral descriptions, not restatements of the mental-model hypothesis, and the raw pass-rate gap stands independently of the error analysis. The absence of a random-move baseline is a completeness/control concern, not a circularity. Accordingly, the derivation chain is self-contained and receives a score of 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Verbal clues are processed as intended by both humans and LLMs; "nearby" means the key or ghost is in an adjacent room.
- domain assumption The original instructions give players enough information about ghost movement for fair comparison; the ambiguity noted by two participants did not invalidate the main result.
- ad hoc to paper The Study 2 error categories are valid operationalizations of mental-model failure.
- domain assumption One human attempt per participant and up to 20 LLM attempts per model are directly comparable measures of average ability.
invented entities (1)
-
Haunted House task environment
independent evidence
Cite this review
Pith. "Pith review of Haunted House: A text-based game for comparing the flexibility of mental models in humans and LLMs." pith.science (2026). https://pith.science/paper/3YK5LZB6
@misc{pith2026250316437,
author = {Pith},
title = {Pith review of: Haunted House: A text-based game for comparing the flexibility of mental models in humans and LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/3YK5LZB6}},
note = {Machine review of arXiv:2503.16437}
}
read the original abstract
This study introduces "Haunted House" a novel text-based game designed to compare the performance of humans and large language models (LLMs) in model-based reasoning. Players must escape from a house containing nine rooms in a 3x3 grid layout while avoiding the ghost. They are guided by verbal clues that they get each time they move. In Study 1, the results from 98 human participants revealed a success rate of 31.6%, significantly outperforming seven state-of-the-art LLMs tested. Out of 140 attempts across seven LLMs, only one attempt resulted in a pass by Claude 3 Opus. Preliminary results suggested that GPT o3-mini-high performance might be higher, but not at the human level. Further analysis of 29 human participants' moves in Study 2 indicated that LLMs frequently struggled with random and illogical moves, while humans exhibited such errors less frequently. Our findings suggest that current LLMs encounter difficulties in tasks that demand active model-based reasoning, offering inspiration for future benchmarks.
Reference graph
Works this paper leans on
-
[1]
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021). Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color. arXiv. https://doi.org/10.48550/arXiv.2109.06129 Aristotle. (2014). Nicomachean Ethics (R. Crisp, Trans.). Cambridge University Press. Borji, A. (2023). A Categorical Archive of C...
-
[2]
Marcus, G. (2020). The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence. arXiv. https://doi.org/10.48550/arXiv.2002.06177 Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review , 63 (2), 81–97. https://doi.org/10.1037/h0043158 Mitchell, M. (2023). ...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.