REVIEW 3 major objections 4 minor 28 references
Otter: A Time-Aware, History-Conditioned Human Chess AI
T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Otter predicts human chess moves 55.23% of the time by conditioning on the last 20 moves and the clock, surpassing a larger position-only model.
desk verdict Otter's internal ablations are clean and its history/time conditioning is genuinely new, but the Maia2 comparison is uncontrolled; the paper deserves review with a shared evaluation required. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two conditioning streams carry the argument: a move-history encoder that embeds the last 20 moves with a small Transformer and produces both token-level representations and a pooled summary, and a time module that turns remaining clock fraction and increment into features through a two-layer MLP. These are fused with Elo buckets into one 640-dimensional conditioning vector that is projected into the queries of every attention block, so board tokens can cross-attend to history tokens with player identity and time pressure shaping what they look for. This conditioned-query attention is the mechanism that lets a small model exploit path dependence and clock state.
What would settle it
Run the prior model and Otter on the same 1.1-million-position February 2025 test set with identical legal-move masking and Elo bucketing; if the prior model's top-1 accuracy reaches or exceeds 55.23%, the paper's headline comparison fails even though its internal ablations would remain valid.
Extended reading notes
Core claim
The core discovery claim is that the position-only Markov assumption is the main bottleneck in human chess move prediction. Adding a Transformer that encodes the last 20 moves and feeding clock pressure into every attention layer lifts top-1 accuracy from 47.61% to 55.23% on identical test positions, with every one of 11 Elo rating brackets gaining at least +7.22 points. The history contribution saturates quickly, with K=5 already capturing 84% of the total history gain, and the time module contributes uniformly even in the lowest rating bracket, where the paper argues clock management is least deliberate. On the paper's own comparison, Otter's 55.23% exceeds the prior model's reported 53.25% with 34% fewer parameters and 31% less training data.
Load-bearing premise
The load-bearing assumption is that the prior model's publicly reported accuracy, measured on its own evaluation set and training distribution, is directly comparable to Otter's accuracy on a balanced February 2025 test set selected by the authors; Section 5.2 and Table 7 explicitly say this comparison is not fully controlled for training data distribution.
Editorial extensions
If this is right
- If the central claim holds, position-only human chess models are leaving a large, cheaply accessible signal unused, and future human move models should include move history and clock context.
- The window-size curve implies most of the benefit, 84 percent, is already available with just the last 5 moves, so near-full accuracy can be reached with less compute at inference time.
- The uniform gain across every rating bracket, including the lowest, says time pressure shapes move choice generally rather than only for strong players.
- In the paper's framing, stronger players become more predictable once behavioral context is provided, which suggests skill-conditioned models gain most from history when they also know the player's level.
- Concrete applications such as personalized coaching, human-like AI opponents, and cheat detection follow directly if human move prediction at this accuracy is reproducible.
Reading between the lines
- Editorial inference: the paper trains only on rapid games, so the time-conditioning story is untested for bullet and blitz; the same normalized-clock features would plausibly carry over, but that is not in the paper.
- Editorial inference: because the code and models are released, an independent researcher could settle the headline comparison by running the prior model on the identical 1.1-million-position test set; the paper's own note that the comparison is not fully controlled for training distribution is a warning the two numbers may not be apples to apples.
- Editorial inference: the acknowledged under-prediction of blunders points to a separate research direction, namely using a tail-weighted or uncertainty-aware loss to predict rare high-cost errors, rather than a defect the history and time context were designed to fix.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Otter, a 15.3M-parameter neural network for predicting human moves in chess, trained on 117M Lichess rapid games. Otter augments a position-only convolutional board encoder with two conditioning signals: a Transformer encoder over the last 20 moves, and a time-control/clock module encoding game format and remaining time. The paper reports 55.23% top-1 accuracy on a balanced February 2025 test set of 1.1M positions, claims to surpass Maia2 (53.25%), and presents ablations isolating +5.24pp from history and +2.38pp from time, with gains uniform across Elo brackets and game phases. Code, models, and training logs are released.
Significance. If the internal ablations are reliable, the paper provides a clear demonstration that move-history and time-pressure conditioning improve human move prediction over a position-only baseline of the same architecture, with a relatively compact model. The release of code, models, and training logs is a strength. However, the headline SOTA claim rests on a comparison with Maia2 using published aggregate numbers from a different evaluation setup; the paper itself acknowledges this in Section 5.2. The central contribution is therefore the ablation study and the architecture, not the cross-dataset superiority claim as currently stated.
major comments (3)
- [§5.2, Table 7] The claim that Otter surpasses Maia2 is not supported by the presented evidence because the two accuracy numbers are measured on different evaluation sets with different rating-bracket composition and time periods. Maia2's 53.25% is a macro-average over three Skilled/Advanced/Master cohorts from Tang et al.; Otter's 55.23% is a macro-average over 11 equally weighted Elo buckets from February 2025 Lichess games selected by the authors. Section 5.2 itself acknowledges the comparison is 'not fully controlled for training data distribution,' yet the abstract and Section 1 present the SOTA claim as a headline result. Since the paper's own Base model scores 47.61%, far below Maia2's 53.25%, the attribution of the entire margin to history/time conditioning requires commensurability that is not established. The authors should either run Maia2 (or a Maia2 reimplementation) on the same February 2025 test set, or at minimum report Otter's accuracy on Maia2's evaluation cohorts and test distribution, and adjust the claims accordingly.
- [§5.6] The history-window sensitivity analysis does not test the design choice of K=20. Truncating the context of a model trained with K=20 (by zeroing the oldest tokens) at inference time measures the model's robustness to missing context, not the effect of training with a shorter window. A model trained with K=5 might behave differently, for example by allocating capacity differently. The conclusion that 'K=5 already retains 84% of the total history advantage' is therefore not a valid justification for the chosen architecture. Please train separate models with K=5, 10, and 20, or rephrase the result as an inference-time truncation analysis.
- [§5.1–§5.2] No uncertainty quantification is provided for any of the reported accuracies. At n=1.1M, the sampling error on 55.23% is about ±0.05 percentage points (95% CI), so the internal ablation gaps of 5–8pp are clearly significant under a binomial model. However, for the cross-dataset Maia2 comparison, the uncertainty in Maia2's published number is unknown and the comparison is confounded by evaluation-set differences. The authors should report confidence intervals (e.g., bootstrap) for the main numbers, and ideally train each ablation variant with multiple seeds to rule out seed-dependent conclusions.
minor comments (4)
- [Throughout] The paper uses 'Maia2' and 'Maia 2' inconsistently; pick one form and use it consistently across the abstract, body, and tables.
- [§4.3] The text states that the value loss 'does not decrease significantly' yet the value head is retained as a regularizer; a sentence on whether the regularization actually improves policy accuracy (e.g., an ablation without the value head) would strengthen this claim.
- [Figure 4] The x-axis labels in Figure 4 are cramped (e.g., '1100 1199' appears compressed); consider rotating the labels or adjusting the layout for legibility.
- [§5.2] The sentence 'confirming that architectural differences alone do not explain the improvement' is odd because the Base model is 5.64pp below Maia2, which suggests the base is much weaker; the intended meaning is that the full model's gain comes from the added conditioning, not from the architecture. Please rephrase for clarity.
Circularity Check
No significant circularity: all reported accuracies are measured on held-out positions, ablations share the same evaluation, and the Maia 2 comparison uses an externally published benchmark.
full rationale
The paper's central derivation chain is empirical rather than circular. Otter is trained by supervised learning on human moves, and its top-1 and top-5 accuracies are measured on a held-out February 2025 test set that is disjoint from the 2024 training period. The ablations compare three variants (base, history-only, full) on identical test positions, so the reported +5.24 pp from history and +2.38 pp from time are measured effects of adding those input signals, not quantities reconstructed from fitted constants or redefinitions of the inputs. The Maia 2 comparison cites the publicly reported 53.25% figure from Tang et al., an external paper, and the paper itself explicitly notes in Section 5.2 that the comparison is not fully controlled for training data distribution; that is an evaluation-validity caveat, not a circularity. No self-citations are load-bearing, no uniqueness theorem is imported, and no known result is merely renamed as a new coordinate system. The paper does not recycle any fitted parameter as a prediction, and no equation in the paper equates an output to an input by construction. The history and time features are conditional inputs to the model, and the accuracy gain is the empirical outcome of training, so the derivation chain is self-contained with respect to circularity concerns.
Assumptions & free parameters
free parameters (4)
- History window K =
20
- Loss weights (policy/value/aux) =
1.0 / 0.25 / 0.5
- Elo bucket boundaries =
11 buckets: <1100, 100-point bins, >=2000
- Time control buckets =
5 buckets: bullet, blitz, rapid, classical, other
assumptions (6)
- domain assumption The last 20 moves, padded and canonicalized, capture sufficient behavioral context; longer histories would not materially change predictions.
- domain assumption Maia2's published aggregate accuracy is directly comparable to Otter's balanced February 2025 test set despite different evaluation protocols.
- domain assumption The clock features f1 and f2, normalized by base time, capture the relevant time-pressure signal across formats.
- domain assumption The time-control bucket provides useful game-format conditioning even though the training set contains only rapid games.
- domain assumption Supervised cross-entropy on human moves is the correct objective for human move prediction.
- standard math Standard transformer and convolution operations behave as documented in public frameworks.
Cite this review
Pith. "Pith review of Otter: A Time-Aware, History-Conditioned Human Chess AI." pith.science (2026). https://pith.science/paper/GSLBTIZX
@misc{pith2026260805206,
author = {Pith},
title = {Pith review of: Otter: A Time-Aware, History-Conditioned Human Chess AI},
year = {2026},
howpublished = {\url{https://pith.science/paper/GSLBTIZX}},
note = {Machine review of arXiv:2608.05206}
}
read the original abstract
Otter is a 15.3M-parameter human chess AI that predicts human move selection by modeling play as a time-aware, sequential process rather than treating each position in isolation. It combines two conditioning signals: (1) a move history encoder that conditions predictions on the last 20 moves, capturing opening preferences, positional drift, and intra-game behavioral tendencies; and (2) a time control module that modulates predictions based on clock pressure. Otter is trained on 6.1 billion positions from 117 million Lichess rapid games over 30 days on a single T4 GPU. Otter achieves 55.23% top-1 and 90.95% top-5 move-prediction accuracy, surpassing the prior state-of-the-art human chess model, Maia 2, with far fewer parameters and less training data. Across 11 Elo brackets (<1100 to >=2000), accuracy peaks at 57.38% in the 1900-1999 bracket. These results show that modeling chess as a time-aware, sequential activity yields more human-accurate move prediction than position-only approaches, using a smaller model. Code, trained models, and complete training logs are publicly released.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Aligning superhuman ai with human behavior: Chess as a model system
Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhuman ai with human behavior: Chess as a model system. InPro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), KDD ’20, pages 1677–1687. ACM, August 2020
work page 2020
-
[2]
Maia-2: A unified model for human-ai alignment in chess
Zhenwei Tang, Difan Jiao, Reid McIlroy-Young, Jon Kleinberg, Siddhartha Sen, and Ashton Anderson. Maia-2: A unified model for human-ai alignment in chess. InAdvances in Neural Information Processing Systems (NeurIPS), 2024
work page 2024
-
[3]
Stockfish: A strong open source chess engine
The Stockfish developers. Stockfish: A strong open source chess engine. https://stockfishchess. org, 2024
work page 2024
- [4]
-
[5]
Michael L. Littman. Markov games as a frame- work for multi-agent reinforcement learning. In William W. Cohen and Haym Hirsh, editors,Ma- chine Learning Proceedings 1994, pages 157–163. Morgan Kaufmann, San Francisco (CA), 1994
work page 1994
-
[6]
Detecting indi- vidual decision-making style: Exploring behavioral stylometry in chess
Reid McIlroy-Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Detecting indi- vidual decision-making style: Exploring behavioral stylometry in chess. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P. S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 24482–24497. Curran As- so...
work page 2021
-
[7]
Uwe Sunde, Dainis Zegners, and Anthony Strittmat- ter. Speed, quality, and the optimal timing of com- plex decisions: Field evidence.arXiv preprint arXiv:2201.10808, 2022
arXiv 2022
-
[8]
Johannes Carow and Niklas M. Witzig. Time pressure and strategic risk-taking in professional chess.Journal of Economic Behavior & Organi- zation, 238:107218, 2025
work page 2025
Show all 28 references
-
[9]
A general reinforcement learn- ing algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learn- ing algorithm that masters chess...
2018
-
[10]
Learn- ing models of individual behavior in chess
Reid McIlroy-Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learn- ing models of individual behavior in chess. InPro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, pages 1253–1263. ACM, August 2022
2022
-
[11]
How time pressure modulates individual differ- ences in the functional connectivity of chunk memory in chess games.Neuroscience, 552:39–46, 2024
Chantat Leong, Yuwen Lin, Juan Zhang, and Zhen Yuan. How time pressure modulates individual differ- ences in the functional connectivity of chunk memory in chess games.Neuroscience, 552:39–46, 2024
2024
-
[12]
Chase and Herbert A
William G. Chase and Herbert A. Simon. Percep- tion in chess.Cognitive Psychology, 4(1):55–81, 1973
1973
-
[13]
Learning to play the chess variant crazyhouse above world cham- pion level with deep neural networks and human data
Johannes Czech, Moritz Willig, Alena Beyer, Kris- tian Kersting, and Johannes Fürnkranz. Learning to play the chess variant crazyhouse above world cham- pion level with deep neural networks and human data. Frontiers in Artificial Intelligence, 3, April 2020
2020
-
[14]
Chalmers
Daniel Monroe and Philip A. Chalmers. Master- ing chess with a transformer model.arXiv preprint arXiv:2409.12272, 2024
2024 arXiv
-
[15]
Chess as a testbed for language model state tracking
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as a testbed for language model state tracking. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11385–11393, 2022
2022
-
[16]
Lewis, Joel Veness, and Tim Genewein
Anian Ruoss, Grégoire Delétang, Sourabh Meda- pati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess. In A. Glober- son, L. Mackey, D. Belgrave, ...
2024
-
[17]
Predicting human chess moves: An ai assisted analysis of chess games using skill-group specific n-gram language models.arXiv preprint arXiv:2512.01880, 2025
Daren Zhong, Dingcheng Huang, and Clayton Greenberg. Predicting human chess moves: An ai assisted analysis of chess games using skill-group specific n-gram language models.arXiv preprint arXiv:2512.01880, 2025
2025
-
[18]
Batch nor- malization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy. Batch nor- malization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors,Proceedings of the 32nd Interna- tional Conference on Machine Learning, volume 37 ofProceedings of Machin...
2015
-
[19]
Deep residual learning for image recog- nition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
2016
-
[20]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017
2017
-
[21]
Scalable diffu- sion models with transformers
William Peebles and Saining Xie. Scalable diffu- sion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023
2023
-
[22]
Gaus- sian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016
Dan Hendrycks and Kevin Gimpel. Gaus- sian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016
2016 arXiv
-
[23]
Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014
1929
-
[24]
Lichess open database
Lichess.org. Lichess open database. https:// database.lichess.org, 2024
2024
-
[25]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Confer- ence on Learning Representations, 2019
2019
-
[26]
SGDR: Stochas- tic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter. SGDR: Stochas- tic gradient descent with warm restarts. InInter- national Conference on Learning Representations, 2017
2017
-
[27]
Mixed precision training
Paulius Micikevicius, Sharan Narang, Jonah Al- ben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. InInternational Conference on Learning Representations, 2018
2018
-
[28]
Pytorch: An im- perative style, high-performance deep learning li- brary
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, L...
2019
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.