REVIEW 4 major objections 6 minor 1 cited by
TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Large reasoning models that ace hard math benchmarks often struggle on simple Tic-Tac-Toe-style games, and this paper backs that claim with a new verifiable benchmark whose answer keys come from a Win/Blocked/Fork heuristic.
desk verdict Useful benchmark, but the headline gap rests on an unverified answer-key heuristic that is also not reproducible from the printed pseudo-code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the programmatic question generator and its scoring heuristic (Algorithm 1). Starting from all reachable game states after N moves, the pipeline keeps only states in which neither player has already won and the current player has not already established a 'winning fork' — two or more immediate winning lines one move short of completion. For each kept state, the heuristic labels the next best moves with one of three verdicts: 'Win' (a move that immediately wins), 'Blocked' (a move that is the sole way to stop the opponent's immediate win), or 'Fork' (a move creating two or more such threats, guaranteeing a win whatever the opponent does). This three-way taxonomy makes the 412 questions verifiable without full game-tree search, and the benchmark's validity rests on the assumption that these labels cover every optimal move.
What would settle it
Run an exhaustive minimax search over every board state in TTT-Bench and compare the set of moves it classifies as optimal — wins, or draws when no win exists — against the 'Win', 'Blocked', and 'Fork' labels produced by Algorithm 1. If any labeled move is not actually optimal under perfect play, or any unlabeled move is equally good, the answer keys and the measured accuracy gaps are not a faithful measure of reasoning ability.
Extended reading notes
Core claim
The central discovery is a disconnect between mathematical competence and game-strategic competence in large reasoning models. On the four TTT-Bench games — ordinary TTT, double TTT, cube TTT, and square TTT — models that excel on MATH 500 and AIME 2024 frequently fail to find the next best move, scoring on average 41% and 5% lower than on those math benchmarks. The gap is largest on the three novel games and on questions whose correct move is labeled 'Blocked' or 'Fork', which require anticipating the opponent or planning two moves ahead; immediate 'Win' moves are found far more reliably. On top of the accuracy drop, the models use as many or more thinking tokens on these simple puzzles as on olympiad-level math, often producing circular and repetitive reasoning instead of the short, direct solution a human would give.
Load-bearing premise
The answer keys assume that the heuristic in Algorithm 1 labels every optimal next move, so any move outside its sets is scored wrong even if full game-tree search would show it wins or draws; if the heuristic misses optimal moves, the reported performance gap could be an artifact of the scoring rather than a genuine reasoning failure.
Editorial extensions
If this is right
- If the claim is correct, current reasoning models cannot be trusted for tasks that combine spatial layout, opponent modeling, and multi-step planning, even when every individual step is elementary.
- Math performance is not a reliable proxy for general reasoning: benchmark suites should include simple, novel, verifiable game states alongside STEM problems to expose this gap.
- TTT-Bench's heuristic-based pipeline can generate unlimited fresh questions for any finite, deterministic two-player game, enabling contamination-free evaluation as models improve.
- Long chain-of-thought is not helping on these tasks: models that produce shorter traces on TTT-Bench tend to score higher, suggesting overthinking is actively harmful for simple problems.
Reading between the lines
- One plausible explanation the paper leaves implicit is that reinforcement-learning training on formal, step-by-step math may be teaching a narrow prover-like skill rather than general competence; TTT-Bench could test this by fine-tuning a small model on Fork/Blocked cases and checking whether gains transfer to other spatial-strategy tasks.
- Because the tasks are text-only, the gap could partly reflect mental-visualization load rather than strategy; giving models rendered images of the boards (a testable variant the paper itself notes) would separate these two causes.
- The Fork category specifically requires holding two simultaneous threats in mind; a synthetic probe varying the number of simultaneous threats could test whether failures track working-memory-style load, which would make the deficit more mechanistic than 'strategic reasoning' implies.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces TTT-Bench, a benchmark of four two-player Tic-Tac-Toe-style games (oTTT, dTTT, cTTT, sTTT) with programmatically generated questions that ask for the next best move. Answers are labeled by a heuristic (Algorithm 1) into Win, Blocked, or Fork verdicts. The authors evaluate a wide range of open LRMs plus o3-mini-medium and DeepSeek-R1 on TTT-Bench and compare with MATH 500 and AIME 2024, reporting that models score on average 41% lower than MATH 500 and 5% lower than AIME 2024, with the largest drops on Fork-type questions. They also report that models use longer chains of thought on TTT-Bench than on MATH 500, and conclude that LRMs that excel at hard math frequently fail at simple strategic and spatial reasoning tasks.
Significance. If the answer keys are correct, TTT-Bench is a valuable and contamination-resistant addition to reasoning evaluation: it is simple, scalable, and covers a domain (strategic and spatial reasoning) that is underrepresented in math-centric benchmarks. The paper's strengths include releasing the benchmark, evaluating a broad set of open models, and reporting per-verdict results that yield a falsifiable difficulty ordering. The central claim, however, rests on the correctness of the heuristic-generated labels and on the assertion that the games are trivial for humans; neither is currently established. The reported performance gaps could be artifacts of mislabeled optimal moves or of protocol differences, so benchmark validation is needed before the main conclusion can be accepted.
major comments (4)
- [Appendix C, Algorithm 1, get_solution] As printed, the `else return None` branch after the Blocked check means that `get_solution` returns whenever the opponent has zero or more than one immediate winning move. Consequently, `get_forks` is unreachable and no Fork verdict can ever be generated, which directly contradicts Figure 2 and Table 4, both of which report substantial numbers of Fork questions. This is a load-bearing inconsistency: the answer-key generation is not reproducible from the paper alone. Please correct the pseudocode or provide the exact implementation used, and confirm that the released benchmark actually contains Fork-labeled items.
- [Appendix C, Algorithm 1, get_forks; Section 3.1] The paper asserts that the generated solutions are 'optimal next-best moves', but Algorithm 1 is a heuristic and no minimax or exhaustive game-tree verification is provided. The fork test counts any two winning lines with n-1 current-player stones and no opponent stones; the printed pseudocode does not explicitly require the two threats to share the empty square played. In the reachable state space, after `get_wins` has returned no winning moves, this condition may be automatic, but the paper does not demonstrate this, and the control-flow error above makes the procedure unreproducible. Without an independent check that the Win/Blocked/Fork labels equal the game-theoretically optimal move sets, a model that selects a different winning or drawing move can be scored as wrong, so the reported Pass@1 gaps and the headline 41%/5% numbers could be artifacts. Please provide executable generator code, a minimax-based validation of all labels, and a clear statement of any disagreements.
- [Abstract; Section 1; Section 3] The paper repeatedly claims that these games are trivial for humans ('humans can effortlessly solve from a young age', 'easy for humans to play even from a young age'), but no human evaluation is reported. This is load-bearing because the main conclusion contrasts model failures with human ease. Please include a small human baseline (even a few participants on a sample of questions) or explicitly reframe the triviality claim as an assumption rather than an established fact.
- [Section 4.1; Table 2] Pass@1 is computed with k=16 responses for open models, while the frontier models (o3-mini-medium and DeepSeek-R1) are evaluated with a single response per TTT-Bench test sample and their math scores are taken from published results with unknown sampling protocols. This makes the head-to-head comparison in Table 2 and the statement that frontier models find TTT-Bench 'as difficult as the standard math benchmarks' not apples-to-apples. Please either rerun the frontier models under the same protocol or report the comparison with explicit caveats; confidence intervals for all reported Pass@1 values would also help assess whether the smaller AIME-2024 gaps (e.g., 5%) are meaningful.
minor comments (6)
- [Section 4.1, Eq. (1)] The quantity computed is the sample mean of correctness over k=16 responses, not the usual Pass@1 estimator; the name is misleading and no confidence intervals are reported.
- [Appendix C, Algorithm 1] There are typos in the pseudocode ('Wining States' should be 'Winning States', 'wining moves' should be 'winning moves'), and the variables `W_current` and `W_opponent` are defined with notation that is hard to parse; a cleaner, runnable version would improve reproducibility.
- [Section 3.1] The phrase 'Winning Fork' is defined informally and is used both to filter generated states and to define the Fork label; please align the informal definition with the exact condition implemented in Algorithm 1.
- [Section 4.2] The difficulty ordering 'oTTT<dTTT<sTTT<cTTT' is introduced with the < symbol but not defined; the claim that this ordering is 'consistent across' both MATH 500 and AIME 2024 comparisons would benefit from statistical support rather than visual inspection of Figure 3.
- [Table 2 and footnotes] The published math results for frontier models are cited only in footnotes; please include full references with access dates in the bibliography.
- [Introduction] The text refers to 'Appendix 1' when describing the solution-generation algorithm; the algorithm is actually in Appendix C.
Circularity Check
No significant circularity; heuristic-defined answer keys are a validity concern, not a circular reduction.
full rationale
TTT-Bench is a benchmark paper, not a derivation with fitted parameters or a prediction loop. The answer keys are produced by the authors' Algorithm 1, and model Pass@1 scores are computed against those keys; this is the normal operational definition of a benchmark, not a circular reduction. The paper's central claim (LRMs perform worse on TTT-Bench than on MATH 500/AIME 2024) is an empirical comparison against external math benchmarks and is not equivalent by construction to any fitted input. There are no self-citations used as load-bearing evidence: the related-work and evaluation sections cite external sources for models and benchmarks, and the only internal reference (Appendix C's algorithm) is the generator itself. The unverified game-theoretic optimality of the heuristic and the internal inconsistency in the printed get_solution (the 'else return None' branch makes the subsequent get_forks call unreachable, even though Figure 2 reports Fork verdicts) are serious correctness and reproducibility concerns that could invalidate the empirical conclusions, but they are not circularity: a flawed ground-truth generator does not make the evaluation a restatement of its inputs. Similarly, the 'trivial for humans' claim is unsupported by a human baseline, but that is an external-validity gap, not a self-definitional loop. The Limitations section acknowledges domain, text-only, and sampling constraints, but not the missing minimax or human validation; that omission is a correctness risk, not a circular step. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (2)
- Move-count ranges N per game =
oTTT {4,5}, dTTT {6,7}, cTTT {5,6,7}, sTTT {6,7}
- Fork threat threshold =
2 (threat_count >= 2)
assumptions (4)
- ad hoc to paper The Win/Blocked/Fork heuristic in Algorithm 1 yields the complete set of optimal next-best moves for each generated state.
- domain assumption All TTT-Bench tasks are effortless for human players from a young age.
- domain assumption Pass@1 computed with k=16 responses for open models is comparable to k=1 for frontier models and to published math benchmark scores.
- domain assumption The text-only representation of spatially structured boards does not systematically disadvantage models relative to human performance.
Cite this review
Pith. "Pith review of TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games." pith.science (2026). https://pith.science/paper/YJEAZ3DR
@misc{pith2026250610209,
author = {Pith},
title = {Pith review of: TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games},
year = {2026},
howpublished = {\url{https://pith.science/paper/YJEAZ3DR}},
note = {Machine review of arXiv:2506.10209}
}
abstract
Large reasoning models (LRMs) have demonstrated impressive reasoning capabilities across a broad range of tasks including Olympiad-level mathematical problems, indicating evidence of their complex reasoning abilities. While many reasoning benchmarks focus on the STEM domain, the ability of LRMs to reason correctly in broader task domains remains underexplored. In this work, we introduce \textbf{TTT-Bench}, a new benchmark that is designed to evaluate basic strategic, spatial, and logical reasoning abilities in LRMs through a suite of four two-player Tic-Tac-Toe-style games that humans can effortlessly solve from a young age. We propose a simple yet scalable programmatic approach for generating verifiable two-player game problems for TTT-Bench. Although these games are trivial for humans, they require reasoning about the intentions of the opponent, as well as the game board's spatial configurations, to ensure a win. We evaluate a diverse set of state-of-the-art LRMs, and \textbf{discover that the models that excel at hard math problems frequently fail at these simple reasoning games}. Further testing reveals that our evaluated reasoning models score on average $\downarrow$ 41\% \& $\downarrow$ 5\% lower on TTT-Bench compared to MATH 500 \& AIME 2024 respectively, with larger models achieving higher performance using shorter reasoning traces, where most of the models struggle on long-term strategic reasoning situations on simple and new TTT-Bench tasks.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play
Supervised fine-tuning collapses LLM action diversity in board-game play beyond what the accuracy–diversity tradeoff requires; augmenting SFT data with all optimal actions per state partially prevents this.
Reference graph
Works this paper leans on
-
[1]
oTTT test sample example in Figure 1 Question: Alice and Bob are playing a game on a 3x3 grid
For the solutions, we also provide the verdict and the states contributing to the corresponding verdict. oTTT test sample example in Figure 1 Question: Alice and Bob are playing a game on a 3x3 grid. The points on the grid are labeled top to bottom, left to right, as A,B,C,D,E,F,G,H,I. Alice plays white. Bob plays black. At each turn, the player places a ...
-
[2]
To form a line, she needs a third stone in a row, column, or diagonal
**Alice’s Potential Threats**: - In **Grid1**, Alice has D and I. To form a line, she needs a third stone in a row, column, or diagonal. - In **Grid2**, Alice has J, M, and I. These positions do not form a line yet, but she could aim for a diagonal or column
-
[3]
If she takes C and F, she can form a vertical line
**Critical Blocking Moves**: - **Grid1’s Column3 (C, F, I)**: Alice has I. If she takes C and F, she can form a vertical line. Bob can block this by taking F. - **Grid1’s Row2 (D, E, F)**: Alice has D. Taking F would block her from completing this row. - **Grid2’s Diagonal (K-L-I)**: If Alice takes L, she could form a diagonal with I and L. However, Bob’s...
-
[4]
- Taking F also affects **Grid2’s column1**, preventing Alice from forming a vertical line there
**Strategic Considerations**: - Taking F in **Grid1** blocks both the vertical column3 and the horizontal row2 threats. - Taking F also affects **Grid2’s column1**, preventing Alice from forming a vertical line there. After evaluating all possibilities, the optimal move for Bob is to place a stone at **F**, which blocks Alice’s potential threats in both g...
-
[5]
InAdvances in Neural Information Processing Systems, volume 36, pages 53728–53741
Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level...
arXiv 2024
-
[6]
Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599. NovaSky Team. 2025a. Sky-t1: Train your own o1 preview model within $450. https://novasky- ai.github.io/posts/sky-t1. Accessed: 2025-01-09. OpenThoughts Team. 2025b. Open Thoughts. https://open-thoughts.ai. Qwen Team. 2025c. Qwq-32b: Embracing the power of reinforcement l...
arXiv 2025
-
[7]
The vertices involved are A, B, C, D, E, F, G, H, I, J, K, L
Alice places a white sticker at K. The vertices involved are A, B, C, D, E, F, G, H, I, J, K, L. The shared vertices are B, C, F, G. Alice’s current stickers are at B, G, J, K. Bob’s current stickers are at A, C, E. The remaining vertices are D, H, I, L, F. To determine where Bob should place his next sticker, we need to consider the potential planes wher...
-
[8]
There are 25 intersection points between the 5 horizontal lines and 5 vertical lines. These 25 points are labeled from top to bottom, left to right, as A, B, C, D, . . . , Y. Alice plays white. Bob plays black. At each turn, the player places a stone of the corresponding color onto one of the 25 points that have not been occupied. Whoever has four stones ...
Show all 24 references
-
[9]
**Current Positions**: - **Grid1**: - Alice: I (bottom-right), D (middle-left) - Bob: G (bottom-left), A (top-left) - **Grid2**: - Alice: J (top-middle), M (middle-right), I (bottom-left) - Bob: K (top-right)
-
[13]
Alice places a white sticker at B
-
[14]
Bob places a black sticker at A
-
[15]
Alice places a white sticker at G
-
[16]
Bob places a black sticker at C
-
[17]
Alice places a white sticker at J
-
[18]
Bob places a black sticker at E
-
[20]
diagonal square
There are 25 intersection points between the 5 horizontal lines and 5 vertical lines. These 25 points are labeled from top to bottom, left to right, as A, B, C, D, . . . , Y. Alice plays white. Bob plays black. At each turn, the player places a stone of the corresponding color...
-
[21]
**Alice’s stones**: A(0,0), B(0,1), C(0,2), W(4,2)
-
[22]
**Bob’s stones**: E(0,4), I(1,3), O(2,4)
-
[23]
The critical points for such squares include G(1,1) and H(1,2)
**Threat analysis**: Alice’s top-row stones (A, B, C) could form part of a unit square if combined with stones below. The critical points for such squares include G(1,1) and H(1,2)
-
[24]
Win"; end opp_w_moves←get_wins(P o,A)▷Check if current player has blocking moves. iflen(opp_w_moves) == 1then returnopp_w_moves,
**Blocking potential squares**: The unit square involving A, B, and needing F(1,0) and G(1,1) is a key threat. Since Alice has A and B, blocking G(1,1) would prevent her from completing this square in future moves. After evaluating all potential squares and critical points, th...
2024
-
[2021]
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav
Training verifiers to solve math word prob- lems.Preprint, arXiv:2110.14168. Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. 2024. Gamebench: Evaluating strategic reasoning abilities of ll...
2024 arXiv
-
[2023]
A survey on evaluation of large language mod- els.Preprint, arXiv:2307.03109. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman
-
[2024]
InThe Twelfth Inter- national Conference on Learning Representations
Let’s verify step by step. InThe Twelfth Inter- national Conference on Learning Representations. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian- guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025a. Wizardmath: Empowering ma...
2024 arXiv
-
[2025]
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun
rstar-math: Small llms can master math rea- soning with self-evolved deep thinking.Preprint, arXiv:2501.04519. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun....
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.