Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This comment argues that the reported 'accuracy collapse' of large reasoning models on planning puzzles is an experimental artifact, not a fundamental reasoning failure.

desk verdict A useful caution about benchmark validation, but the Section 5 probe changes the task and cannot carry the claim that the accuracy collapse is an artifact. read the letter →

arxiv 2506.09250 v2 pith:UTEDE6PP submitted 2025-06-10 cs.AI cs.LG

A. Lawsen This is my paper
classification cs.AIcs.LG
keywords largereasoningmodelsaccuracycollapseTowerofHanoiRiverCrossingrecursiveoutputrepresentationevaluationartifactstokenlimitsplanningbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This comment challenges a widely reported result that large reasoning models stop being able to solve planning puzzles once the puzzles pass a complexity threshold. It argues that the reported accuracy collapse mostly comes from how the test was built: the Tower of Hanoi task demands exponentially long move lists, models truncate those lists or hit output limits, and some River Crossing benchmark instances are mathematically impossible to solve. When models are allowed to answer with a recursive function rather than a full move list, the comment reports very high accuracy on a 15-disk Tower of Hanoi problem that the original study had scored as a total failure. The point the author is trying to establish is that an apparent failure of reasoning can be an artifact of requiring exhaustive enumeration.

What carries the argument

The central device is a representation switch: rather than require the model to emit a list of all $2^N - 1$ moves, a demand that grows exponentially and invites truncation, the prompt asks for a short recursive Lua function that prints the solution when called. That isolates planning competence from output-length constraints. The other load-bearing piece is a solvability check: using the known impossibility of River Crossing variants with $N > 5$ actors and boat capacity $3$, the comment shows that some benchmark items had no correct answer at all. A token-count estimate of about 10 tokens per move locates where exhaustive output would exceed practical limits and explains why failures concentrate at large $N$.

What would settle it

Because the comment's own Lua-function recovery is labeled preliminary, the decisive experiment is a powered rerun of the original Tower of Hanoi prompts at the sizes where collapse was reported, but with an output budget large enough for the full $2^N - 1$ move list. If accuracy remains near zero, truncation does not fully explain the collapse; if accuracy rises sharply, the original evaluation was scoring output decisions as reasoning failures.

Watch

Extended reading notes

Core claim

The paper's central claim is that the zero-accuracy collapse reported for reasoning models on planning puzzles is primarily an experimental artifact. It identifies three mechanisms: models explicitly acknowledge stopping because their output would be too long; the automated checker scores truncation and refusal the same way it scores wrong reasoning; and River Crossing instances with more than five actors and boat capacity three have no valid solution, so models are penalized for recognizing unsolvability. As a positive control, the comment asks models to output a recursive Lua function that prints the Tower of Hanoi solution for 15 disks, and reports very high accuracy across several models in under 5,000 tokens. This is taken as evidence that algorithmic understanding survives when the task no longer requires exhaustive enumeration.

Load-bearing premise

The load-bearing premise is that writing a correct recursive Lua function that prints a Tower of Hanoi solution exercises the same planning ability the original test was measuring with an exhaustive move list; if those are different tasks, the restored accuracy does not by itself prove the original failures were format artifacts.

Editorial extensions

If this is right

  • If the recursive-output control is representative, prior zero scores on large Tower of Hanoi instances mainly measure a format mismatch, not the absence of planning.
  • Benchmark users should treat unsolvable generated instances, such as River Crossing with more than five actors and boat capacity three, as invalid items instead of model failures.
  • Evaluation pipelines need a distinct category for truncation and refusal, so that choosing not to enumerate is not scored identically to failing to reason.
  • Solution length is a weak proxy for difficulty: Tower of Hanoi has an $O(1)$ per-move rule despite its exponential move count, whereas optimal Blocks World planning is NP-hard.
  • When a prompt instructs optimality, the checker must verify optimality; otherwise models that faithfully seek the minimum are judged against a weaker standard.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The artifact view predicts a format gradient: the same model should solve the same Tower of Hanoi instance under a compact, code-like output while truncating under a verbose, step-by-step format; this can be tested within one model family.
  • If truncation is a deliberate calibration decision, models may systematically underestimate their available context, producing spurious collapse in any benchmark whose output length grows with problem size.
  • The solvability-check lesson generalizes: automatically generated benchmark suites should verify that each instance has a solution before using it as a failure case, since one unsolvable batch can invert a comparative finding.
  • The comment's Section 5 recovery run is explicitly preliminary and underpowered, so the strongest form of its claim needs a powered multi-model, multi-trial replication before the artifact explanation fully replaces the original collapse narrative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This comment paper challenges Shojaee et al.'s report of 'accuracy collapse' in large reasoning models on planning puzzles. The authors argue that the apparent failure is primarily an experimental artifact, citing three issues: models truncate long Tower of Hanoi outputs due to token limits, automated evaluation misclassifies such truncation as reasoning failure, and River Crossing benchmarks contain unsolvable instances for N ≥ 6. They support the token-limit claim with a token-count estimate and a Twitter screenshot, and they report preliminary results that models can write correct recursive Lua functions for Tower of Hanoi with 15 disks. The paper concludes that the original findings reflect evaluation design limitations rather than fundamental reasoning deficits.

Significance. If the central claim were established, it would substantially revise the interpretation of Shojaee et al. and would caution the community about overgeneralizing 'accuracy collapse' results. The paper does identify a potentially important issue: verifying that benchmark instances are actually solvable is a necessary check, and the River Crossing impossibility point, if it applies to the original task, is a genuine flaw. The paper also usefully distinguishes between solution length and computational difficulty. However, the evidence presented is not sufficient to support the strong claim that the original results are primarily artifacts; the key experiment substitutes a different task and is explicitly preliminary.

major comments (4)
  1. [Section 5] The probe in Section 5 replaces the original exhaustive-move-list task with a request to write a recursive Lua function for Tower of Hanoi N=15. This is a task swap: producing a recursive function demonstrates algorithmic comprehension, whereas the original evaluation required the model to generate and maintain a complete concrete sequence of 2^N−1 moves. The claim that 'intact reasoning capabilities' explain the original failures is therefore unsupported, because the probe removes exactly the exhaustive-execution requirement that the original benchmark tests.
  2. [Section 4, Eqs. (2)–(4)] The paper's own token estimates show that for N=9 the required output is only about 5,000 tokens, far below the 64k/100k context limits. Thus actual output-token truncation cannot explain the reported collapse near N=9. The paper's fallback explanation—that models 'choose to stop prematurely' due to poor calibration—is speculative and rests on a single Twitter screenshot, which is anecdotal and not a systematic sample of model outputs.
  3. [Section 5, footnote 2] The 'very high accuracy' result is not quantified: no trial counts, no standard deviations, no sample outputs, and no comparison to the original evaluation are reported. The paper itself states that 'full experimental validation remains as future work,' so the central empirical claim is explicitly deferred. A comment that makes a strong 'artifact' claim needs data, not just a preliminary anecdote.
  4. [Section 3] The River Crossing impossibility claim relies on an external theorem [4], but the paper does not reproduce or quote the original benchmark definition from Shojaee et al. to confirm that their instances use the same rules and boat capacity as the theorem. Without this verification, the impossibility argument may not apply to the exact task that was evaluated, and the severity of the flaw in the original study remains unclear.
minor comments (5)
  1. [Section 2.1] Equation (1) assumes that per-token errors are independent and identically distributed, which is not justified and is not used in the original Shojaee et al. evaluation; this statistical digression is distracting.
  2. [Section 4] The reliance on a Twitter post as the sole evidence for model behavior is not appropriate for a formal journal; at minimum, the authors should provide direct quotes from multiple model outputs or a systematic analysis.
  3. [Section 1 and Acknowledgments] The paper's informal self-referential passages, including the contribution statement and the acknowledgments, are unprofessional for a formal venue and contain a typo ('it's life' instead of 'its life').
  4. [Table 1] The 'Computational Complexity' column entries are not well-defined; 'O(1) per move' for Tower of Hanoi is a trivial statement and does not capture the complexity of producing the full move sequence, which is the actual requirement in the original evaluation.
  5. [Section 6.1] The discussion of Blocks World optimality is speculative: the paper does not show that Shojaee et al. instructed models to find provably optimal plans, nor that models actually attempted this, so the argument that models were penalized for seeking optimality is not supported.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the comment's claims rest on external references and an independent, self-flagged preliminary probe rather than on derivation steps that reduce to their own conclusions.

full rationale

The paper does not exhibit a circular derivation chain. Its three main arguments are: (1) Tower of Hanoi outputs risk hitting token limits, supported by a Twitter screenshot and token-count estimates in Eqs. (2)-(4); (2) River Crossing instances with N >= 6 and boat capacity 3 are impossible, cited to an external source [4]; and (3) requesting a Lua function instead of an exhaustive move list yields high accuracy, an independent albeit preliminary probe reported in Section 5. None of these steps fits a fitted parameter later renamed as a prediction, a load-bearing self-citation, an author-imported uniqueness theorem, an ansatz smuggled in via citation, or a redefinition of the target result. The token estimates are rough and use an assumed 10 tokens per move, but they are not fitted to the observed collapse; indeed the paper notes that the theoretical maximum (N = 12-13) is larger than the reported collapse threshold (N < 9), so the estimate is not used to force the conclusion. The closest concern is construct validity rather than circularity: Section 5 assumes that generating a correct recursive Lua function demonstrates the same capability that the original benchmark requires for producing an exact exhaustive move list. That is an evidential gap, and the paper itself labels the experiments 'preliminary' and says 'Full experimental validation remains as future work' in the Section 5 footnote. The same footnote and the corrected Appendix A also admit prior errors and missing trials. These limitations affect how much weight the Section 5 probe can carry, but they do not make the paper's reasoning circular: the probe is an external, not-yet-validated observation, not a restatement of its own assumptions. No specific reduction of the central claim to its inputs can be quoted from the paper, so under the hard rules no circularity step is recorded. The score of 1 reflects the absence of circularity while acknowledging that the paper's central positive evidence is explicitly preliminary and not yet independently validated.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's argument rests on two external facts (river crossing unsolvability, tokenizer estimates) and two interpretive assumptions (models deliberately truncate; writing a function proves planning). The token-per-move constant is a free parameter in the Nmax calculation. No new physical or conceptual entities are introduced.

free parameters (2)
  • tokens_per_move = 10
    Estimated from the OpenAI tokenizer and used in Equations 2-4 to argue that token limits should not bind below N=12-13. The exact value strongly affects Nmax, and the paper earlier used an inconsistent value before correcting itself.
  • overhead_tokens_C = not specified
    Added to Equation 2 for problem setup and formatting but never quantified. It shifts the predicted maximum solvable size and is not independently justified.
assumptions (3)
  • standard math River Crossing with N>=6 and boat capacity b=3 is unsolvable.
    Cited to Efimova (2018) and used to score Shojaee et al.'s instances as impossible. The theorem is external to this paper and is plausible, but the paper does not prove it.
  • domain assumption Models deliberately decide to truncate output because of token limits.
    Inferred from a single Twitter example in Section 2. The central interpretation that apparent failures are 'reasonable engineering decisions' rests on this unproven behavioral attribution.
  • domain assumption Generating a correct recursive Lua function for Tower of Hanoi demonstrates the same reasoning capability needed for move-list execution.
    Used in Section 5 to conclude that failures are format constraints, not reasoning failures. The equivalence is asserted, not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity." pith.science (2026). https://pith.science/paper/UTEDE6PP

@misc{pith2026250609250,
  author       = {Pith},
  title        = {Pith review of: Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UTEDE6PP}},
  note         = {Machine review of arXiv:2506.09250}
}
read the original abstract

Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit "accuracy collapse" on planning puzzles beyond certain complexity thresholds. We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Our analysis reveals three critical issues: (1) Tower of Hanoi experiments risk exceeding model output token limits, with models explicitly acknowledging these constraints in their outputs; (2) The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities; (3) Most concerningly, their River Crossing benchmarks include mathematically impossible instances for N > 5 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems. When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures. These findings highlight the importance of careful experimental design when evaluating AI reasoning capabilities.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Protoreasoning in Tiny Transformers

    cs.CL 2026-08 conditional novelty 6.0 of 10

    On two Dyck-bracket tasks, protoreasoning traces let ~1M-parameter transformers generalize out of distribution much better than vanilla training, and the effect comes from trace content rather than extra tokens.

  2. To Trade or Not to Trade: An Agentic Approach to Estimating Market Risk Improves Trading Decisions

    q-fin.ST 2025-07 conditional novelty 6.0 of 10

    LLM-discovered stochastic models of price paths provide risk metrics that improve trader-agent decisions, raising average Sharpe ratios from 0.88 to 1.40 in the paper's backtests.

  3. A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A comment claiming the LRM reasoning cliff is caused by the restrictive text-only interface, with a case study showing tool-enabled models outperform their tool-less versions.

  4. CoreThink: A Symbolic Reasoning Layer to reason over Long Horizon Tasks with LLMs

    cs.AI 2025-08 reject novelty 3.0 of 10

    The paper claims a symbolic orchestration layer, CoreThink, achieves state-of-the-art results on seven coding and reasoning benchmarks with no training, but provides no verifiable implementation or method details.

Reference graph

Works this paper leans on

5 extracted references · 3 canonical work pages · cited by 4 Pith papers

  1. [4]

    Efimova, E. A. (2018). River Crossing Problems: Algebraic Approach . arXiv:1802.09369

    arXiv 2018

  2. [1]

    Shojaee, P., Mirzadeh, I., Alizadeh, K., et al. (2025). The Illusion of Thinking: Understand- ing the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity . arXiv:2501.12948

    arXiv 2025

  3. [2]

    @scaling01. (2025). Twitter thread on LRM replication. https://x.com/scaling01/status/ 1931817022926839909/photo/1

    work page 2025

  4. [3]

    Dziri, N., Lu, X., Sclar, M., et al. (2023). Faith and fate: Limits of transformers on composi- tionality. Advances in Neural Information Processing Systems , 36

    work page 2023

  5. [5]

    and Thi´ ebaux, S

    Slaney, J. and Thi´ ebaux, S. (2001). Blocks World revisited.Artificial Intelligence, 125(1-2):119- 153. A Original Token Limit Analysis Our initial analysis considered two possible interpretations of the evaluation requirements in Shojaee et al. We present both here for completeness. Note: The original version of this paper contained an error. While we s...

    work page 2001

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.