Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

TextQuests: How Good are LLMs at Text-Based Video Games?

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TextQuests shows that current LLMs, with no tools and only their own transcript, complete none of 25 Infocom games and reach at most 37.8% progress; adding the official hint booklets raises the best model to 70% but most games remain…

desk verdict TextQuests is a useful benchmark for long-horizon agent evaluation, but the headline progress numbers rest on unvalidated checkpoint labels and single runs. read the letter →

arxiv 2507.23701 v3 pith:HHX3QAVJ submitted 2025-07-31 cs.AI cs.CL

Long Phan , Mantas Mazeika , Andy Zou , Dan Hendrycks This is my paper · ORCID
classification cs.AIcs.CL
keywords TextQuestsinteractivefictiongamesLLMagentslong-contextreasoningexploratoryenvironmentsGameProgressmetricautosaveInvisiClues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing agent benchmarks over-rely on tools and scaffolding, so they miss a fundamental capability: sustained, self-directed reasoning in an exploratory environment. To test that capability directly, TextQuests drops an LLM into 25 classic Infocom text adventures with no external tools, no summarization, no notepads, and asks it to solve the game from its own growing transcript. The empirical claim is stark: across every evaluated frontier model, not one game was completed without clues, and the best average progress (GPT-5) was 37.8 percent. Providing the official InvisiClues hint booklets raises the best model to 70 percent average progress and five completions, but most games remain unfinished. If this benchmark is right, frontier models are still far short of the kind of long-context, trial-and-error problem solving that these games demand from humans.

What carries the argument

The load-bearing object is the Game Progress metric: for each game, a set of labeled checkpoints marking essential puzzles and milestones, with an agent's score defined as the maximum percentage of checkpoints it ever completes in a trajectory; the benchmark average is the mean over 25 games. The evaluation environment assembles this metric with a full-history interaction protocol (no truncation, no external tools), an autosave/restore mechanism that lets the agent backtrack to any earlier state, and an optional WITH CLUES condition that supplies the official InvisiClues booklets as tiered hints. The checkpoint metric does the argument's work: unlike built-in game scores, which reward exploration and can be decoupled from completion, the checkpoints are intended to track progress along the path to finishing each game. The trajectory data, cut off at 500 steps because longer runs yield little additional gain, is what supports the claim that capable models sustain progress longer but still plateau far below completion.

What would settle it

Re-label the 25 games' checkpoints with independent annotators and re-run each model several times per condition: if Game Progress scores move by more than the gaps between the listed models, or if reruns of the same model vary by tens of percent, the central comparison is measurement noise rather than capability.

Watch

Extended reading notes

Core claim

In its own terms, the paper's core discovery is that LLM agents make minimal progress on TextQuests without external assistance, despite frontier-level performance on static knowledge benchmarks. The benchmark consists of 25 Infocom interactive fiction games run through the original game runtime, with each model given the full ever-growing history of observations, its own reasoning, and its actions; at each turn the model must output one command. Game Progress is the maximum percentage of hand-labeled checkpoints marking essential puzzles and milestones that the agent reaches during a 500-step run. Across all models tested in NO CLUES mode, the completion count is zero out of 25; with the complete official clue booklets in context, at most five games are completed, and progress on the rest remains partial. The authors interpret this as evidence that current LLMs lack the intrinsic long-context reasoning and sustained planning needed for exploratory, stateful problem-solving, and that their trajectories show characteristic failures such as hallucinating past actions and looping on repeated behaviors.

Load-bearing premise

The central assumption is that the hand-labeled checkpoints, and the single 500-step run used for each model, give an unbiased picture of how much progress an agent actually makes on each game.

Editorial extensions

If this is right

  • If TextQuests measures what it claims, then frontier LLMs still lack a core component of autonomous agency: the ability to make steady, self-directed progress in a novel stateful environment using only internal reasoning.
  • Because the environment forbids tools, gains on TextQuests are attributable to the base model itself; this gives a cleaner signal of intrinsic long-context reasoning than tool-augmented agent benchmarks.
  • The 500-step saturation point means the benchmark can differentiate models without unbounded runs, but it also suggests that raw context length is not the bottleneck; the bottleneck is using the history to plan.
  • The gap between WITH CLUES and NO CLUES shows that these models can partially use structured external hints, yet still fail to integrate them into a complete solution.
  • Scaling appears to matter: the larger standard models outperform their mini variants by large margins on exploratory tasks, consistent with test-time-compute effects that plateau after a budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit is that the single unvalidated checkpoint set per game carries the whole ranking; independent re-labeling and repeated runs would show how much of the order is metric noise rather than capability.
  • The failure to reverse an ascent path in Wishbringer suggests the bottleneck may be constructing and updating a mental map from a transcript, which would predict correlations with other spatial or causal backtracking tasks that the paper does not test.
  • Because autosave makes all exploration reversible, TextQuests measures problem-solving without permanent consequences; introducing irreversible costs, limited resources, or time pressure could change both the absolute scores and the model ordering.
  • Since the puzzles come from 1980s games with parser conventions and cultural references, part of the gap could reflect unfamiliarity with the genre rather than pure reasoning; adding newly authored interactive fiction would help separate those factors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces TextQuests, a benchmark of 25 classic Infocom interactive fiction games, and evaluates a range of LLMs as agents in a no-tools, no-truncation, single-session setting. The agent receives the full growing history of observations, reasoning, and actions, and can use an autosave/restore mechanism. The central claim is that frontier LLMs make minimal progress without assistance, with the headline numbers being GPT-5 at 37.8% Game Progress in NO CLUES mode and 70.0% WITH CLUES, and zero games completed by any model in NO CLUES mode. The paper also reports harm scores and qualitative failure modes such as hallucinating prior actions and failing to reverse spatial paths, and it provides token-efficiency analyses. The empirical conclusion that current models are far from solving these exploratory games is plausible and partly supported by objective completion counts, but the quantitative Game Progress scores rest on unvalidated manual checkpoints and single runs per model.

Significance. If the quantitative metric is validated, TextQuests would fill a genuine gap: most agent benchmarks test tool use or structured tasks, while TextQuests targets sustained, self-directed reasoning over a long, growing context. The benchmark release, the autosave/restore design, the 500-to-800-step saturation analysis, and the detailed token-cost accounting are concrete contributions. The objective completion counts (0/25 in NO CLUES for every model, at most 5/25 WITH CLUES) are robust and independently support the qualitative finding that current models struggle. The qualitative trajectory analyses, especially the Wishbringer and Zork I examples, are informative and give the paper face validity. The main weaknesses are the unvalidated checkpoint-based progress metric and the lack of repeated runs, both of which affect the precision of the headline percentages.

major comments (3)
  1. [Section 3.2] The Game Progress metric is the load-bearing quantitative instrument: Tables 1 and 4 average P_g, the maximum fraction of manually labeled checkpoints reached. Yet the paper reports no annotation protocol, no inter-rater reliability, no checkpoint inventory, and no independent check that checkpoint density or ordering is comparable across the 25 games. If one game has a single early checkpoint while another has twenty late checkpoints, the same agent behavior would yield very different P_g values, so the averaged rankings and the magnitudes such as 37.8% and 70.0% are partly determined by labeler choices. Appendix F provides illustrative comparisons but not a systematic audit. This issue is fixable: the authors should release the checkpoint lists, describe the annotation guidelines, report inter-rater agreement, and provide a sensitivity analysis (for example, leave-one-game-out or an alternative checkpoint construction).
  2. [Section 3.1 / Table 1] The evaluation uses a single run per model per mode, with no repeated seeds, confidence intervals, or per-game variance. P_g is the maximum over a stochastic trajectory, and the qualitative examples show that one wrong restore or one hallucinated action can halt progress, so the point estimates may be highly variable. Claims such as the importance of model scale in Figure 3 and the ordering of close scores (for example, Gemini 2.5 Pro at 23.2% versus Claude Sonnet 4 at 24.7% in NO CLUES mode) are not statistically supported. The authors should report per-game scores, provide robust standard errors across the 25 games, and repeat at least the frontier-model evaluations with multiple seeds.
  3. [Section 4.2 / Figure 2] The paper interprets the trajectories as evidence of 'better long-horizon reasoning' in more capable models, but no controlled comparison isolates context length from general planning ability. A model with better short-horizon planning would show the same pattern of sustained progress, so the advertised specificity to 'intrinsic long-context reasoning' is not yet demonstrated. In addition, the 'Human Optimal' line in Figure 2 is not described with a protocol or a human baseline; it appears to be an optimal walkthrough. The claims should either be softened or supported by an ablation such as the same model with truncated or summarized history, or with an external memory.
minor comments (4)
  1. [Appendix D / Figure 8] Figure 8 is corrupted in the manuscript text, with many '/gid...' token-like strings replacing the intended diagram and JSON example; it should be regenerated as a clean figure.
  2. [References] The reference list contains formatting artifacts, including a split name entry ('Huanxu, Liu') and a duplicate 'Jay Paek' entry from the HLE author list; these should be cleaned.
  3. [Abstract / Appendix A.1.1] The abstract emphasizes 'precluding the use of external tools' and 'self-contained problem-solving,' but Appendix A.1.1 states that feelie text is provided in both NO CLUES and WITH CLUES modes. The paper should clarify that 'self-contained' means no external tools or lookups, not that no external information is ever placed in context.
  4. [Section 3.2 / Table 3] Because the Harm metric is cumulative, the raw 500-step versus 800-step comparisons in Table 3 partly reflect the longer trajectory; the paper acknowledges this and recommends normalization, but the table itself would be more informative if a per-step harm rate were also reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TextQuests measures models against externally labeled checkpoints, and the reported scores are empirical evaluations rather than quantities derived from or fitted to the benchmark's own claims.

full rationale

The paper contains no derivation chain in which a predicted quantity is constructed from the data used to test it. The central quantity, Average Game Progress, is defined in Section 3.2 as the average over games of P_g, 'the maximum percentage of checkpoints the agent completes.' The checkpoints are 'labeled checkpoints for essential puzzles and game milestones' (Section 2). The models' trajectories are external to the label construction: no model output or fitted parameter is used to define the checkpoints, and the headline 'minimal progress' conclusion follows from measuring the models against fixed labels. The 'WITH CLUES' comparison likewise evaluates models given fixed InvisiClues booklets. The benchmark builds on Hendrycks et al. (2021c), a prior paper with overlapping authors, for game files and annotations; this is a normal reuse of a published resource, and no load-bearing argument reduces to that citation. The lack of an annotation protocol or inter-rater reliability for the checkpoints is a validity and robustness concern, not a circularity: if the labels are biased or non-comparable across games, the numeric scores could misrepresent ability, but that is an empirical measurement issue, not a logical equivalence between inputs and outputs. The completion counts (0/25 in NO CLUES; up to 5/25 in WITH CLUES) are additionally objective and independent of label density. Therefore no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark introduces no fitted constants or invented physical entities. The central numbers rest on hand-built checkpoints and inherited annotations, plus design choices such as the 500-step cap; these are the real 'free parameters' of the evaluation.

free parameters (2)
  • Game Progress checkpoint sets = unreported, manual labels per game
    Section 3.2 defines P_g as the maximum percentage of labeled checkpoints; the labels are inherited and extended from Hendrycks et al. (2021c) without an audit trail, and they determine every headline progress number in Tables 1 and 4.
  • Maximum run length = 500 steps
    The 500-step cap is justified post hoc by observed saturation in the same experiments (Appendix B); different caps would change both progress and harm totals.
assumptions (3)
  • domain assumption Infocom interactive fiction games are a valid proxy for real-world exploratory, stateful agent tasks.
    Section 1 claims these games 'provide a compelling testbed' and uses them to draw conclusions about LLM agents in exploratory environments; this mapping is asserted, not validated against a real-world task.
  • domain assumption The official InvisiClues booklets do not amount to a walkthrough, so WITH CLUES measures interpretation rather than direct answer copying.
    Section 2 asserts clues are 'tiered, often cryptic', yet the quoted WishBringer clue ('To leave, go out the open gate at Creepy Corner') is explicit, so this assumption is not uniformly true.
  • domain assumption The game files and annotations from Hendrycks et al. (2021c) are complete and correctly inherited.
    Appendix A.1 builds the benchmark on that collection; any gaps or inaccuracies in the inherited annotations propagate directly into progress and harm scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TextQuests: How Good are LLMs at Text-Based Video Games?." pith.science (2026). https://pith.science/paper/HHX3QAVJ

@misc{pith2026250723701,
  author       = {Pith},
  title        = {Pith review of: TextQuests: How Good are LLMs at Text-Based Video Games?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHX3QAVJ}},
  note         = {Machine review of arXiv:2507.23701}
}
read the original abstract

Evaluating AI agents within complex, interactive environments that mirror real-world challenges is critical for understanding their practical capabilities. While existing agent benchmarks effectively assess skills like tool use or performance on structured tasks, they often do not fully capture an agent's ability to operate autonomously in exploratory environments that demand sustained, self-directed reasoning over a long and growing context. To enable a more accurate assessment of AI agents in challenging exploratory environments, we introduce TextQuests, a benchmark based on the Infocom suite of interactive fiction games. These text-based adventures, which can take human players over 30 hours and require hundreds of precise actions to solve, serve as an effective proxy for evaluating AI agents on focused, stateful tasks. The benchmark is specifically designed to assess an LLM agent's capacity for self-contained problem-solving by precluding the use of external tools, thereby focusing on intrinsic long-context reasoning capabilities in an exploratory environment characterized by the need for trial-and-error learning and sustained problem-solving within a single interactive session. We release TextQuests at https://textquests.ai.

Figures

Figures reproduced from arXiv: 2507.23701 by the authors.

Figure 1
Figure 1. Examples showing the diverse reasoning challenges in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Game progress for various model scales versus an optimal human walkthrough. Capable models sustain [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Examples of long context reasoning failures in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: Comparing mini and standard models from differ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 5
Figure 5. Figure 5: A comparison of output and reasoning token efficiency across state-of-the-art LLMs on [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Adding an AutoSave mechanism to the game environment improves the agent’s exploration efficiency. Left: An example of evaluated LLMs makes use of the autosave and restore features to experiment with different approaches to solve an in-game puzzle. Right: As LLMs’ capab…
Figure 7
Figure 7. Figure 7: Game progress trajectories on TEXTQUESTS for selected frontier LLMs evaluated up to 800 steps. The visualization shows game progress saturating after approximately 500 steps. For detailed metrics, see [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: An illustration of an agent’s turn. From the model’s output, the brief reasoning and action are extracted and [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: A comparison of our Game Progress metric against the in-game Game Score. Left: The trajectory for an optimal walkthrough of a sample game shows that our Game Progress provides a more representative signal of advancement than the built-in score. Right: The final scores …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rushes: A Human Preference Dataset for Pluralistic Alignment

    cs.CL 2026-07 conditional novelty 6.0 of 10

    In six AI-written interactive games, players' 44,226 logged choices are predicted better by a simple SVD recommender (37.7%) and a popularity rule (36.4%) than by GPT-5 with the player's history (34.2%).

  2. SysAdmin: Measuring Instrumental Power-Seeking in Frontier AI

    cs.AI 2026-04 conditional novelty 6.0 of 10

    Seven frontier LLMs showed little spontaneous power-seeking in a Linux sysadmin sandbox (bias-corrected rates roughly 0-5%), but showed more specification gaming and resistance to goal modification.

  3. Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisions

    cs.AI 2025-11 conditional novelty 6.0 of 10

    MAPs is a new amusement-park simulator benchmark on which frontier LLM agents score 7–15% of human performance, exposing persistent gaps in long-horizon planning, active learning, spatial reasoning, and handling stoch...

Reference graph

Works this paper leans on

16 extracted references · 3 canonical work pages · cited by 3 Pith papers

  1. [2]

    This is a fake!

    Ballyhoo Planetfall Sherlock Border Zone Plundered Hearts Sorcerer Cutthroats Seastalker Spellbreaker Deadline Starcross Stationfall Enchanter Suspect The Hitchhiker’s Guide to the Galaxy Hollywood Hijinx The Lurking Horror The Witness Infidel Trinity Wishbringer Moonmist Zork I Zork II Zork III Table 2: List of the 25 Infocom text adventure games include...

    work page 2021

  2. [4]

    Gemini Team

    URL https://arxiv.org/abs/2410.07095. Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next genera- tion agentic capabilities. Technical report, Google DeepMind, June

    arXiv

  3. [5]

    URL https://arxiv.org/abs/1909.05398. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. Multi-if: Benchmarking llms on multi-turn and multilingual i...

    arXiv 1909

  4. [6]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    URL https://arxiv.org/abs/2410.15553. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://arxiv.org/abs/2009.03300. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Me...

    arXiv

  5. [8]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R

    URL https://arxiv.org/abs/2501.14249. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark,

    arXiv

  6. [9]

    Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing

    URL https: //arxiv.org/abs/2311.12022. Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms,

    arXiv

  7. [10]

    Susan Smetale

    URL https://arxiv.org/abs/2501.17399. Susan Smetale. Through the zorking glass. The Washington Post , December

    arXiv

  8. [12]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese

    URL https://arxiv.org/abs/2504.01848. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents,

    arXiv

Show all 16 references
  1. [13]

    Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan

    URL https://arxiv.org/abs/2504.12516. Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games,

    arXiv

  2. [14]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan

    URL https://arxiv.org/abs/2010.02903. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    2010 arXiv

  3. [15]

    10 A T EXT QUESTS Environments A.1 Environments TEXT QUESTS consists of 25 classic Infocom games

    URL https://arxiv.org/abs/2406.12045. 10 A T EXT QUESTS Environments A.1 Environments TEXT QUESTS consists of 25 classic Infocom games. Our benchmark is built upon the game files and annotations collected by Hendrycks et al. (2021c), using the Jericho interface (Hausknecht et al.,

    2021 arXiv

  4. [1983]

    URL https: //www.washingtonpost.com/archive/lifestyle/1983/12/22/through-the-zorking-glass/ 8f6fc376-0942-4e66-abb9-06f66a05165c/ . Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, ...

    1983

  5. [2020]

    Anthropic

    URL https://arxiv.org/abs/2001.08837. Anthropic. Claude’s extended thinking. Research blog post, Anthropic, February

    2001 arXiv

  6. [2023]

    Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A

    URL https://arxiv.org/abs/2311.12983. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. Nolima: Long-context evaluation beyond literal matching,

    arXiv

  7. [2024]

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M ˛ adry

    URL https://arxiv.org/abs/2308.14508. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M ˛ adry. Mle-bench: Evaluating machine learning agents on machine le...

    arXiv

  8. [2025]

    Prithviraj Ammanabrolu and Matthew Hausknecht

    URL https://arxiv.org/abs/2504.11900. Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.