REVIEW 3 major objections 6 minor 48 references
Communicating Chess Strategies in Natural Language
T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5
Pith's one-line read Natural language can convey chess engine strategies well enough to improve both human and LLM play while staying far more compact than raw strategy trees.
desk verdict Solid methods paper: strategy verbalization + off-main-line utility eval actually works; engine-as-truth is the known soft limit, not a hidden collapse. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Partial strategy tree construction (keep only the top-kstrat engine-ranked opponent moves under engine-optimal solver play) plus LLM verbalization, evaluated by Tree-Expanded Puzzle Playing: at each opponent turn sample top-keval engine moves, roll out one solver reply, and expand the branch that is worst for the solver, scoring the final position by engine win probability.
What would settle it
On held-out puzzles, if players given the verbalized descriptions show no gain over the no-strategy baseline under Tree-Expanded Puzzle Playing as keval grows, or if humans systematically score lower with the verbalizations than with the same-size JSON trees, the claim that language is an effective compact medium would fail.
Extended reading notes
Core claim
Natural language is a workable, compact medium for communicating engine strategies: descriptions of pruned partial strategy trees raise puzzle performance for both LLM and human players above no-strategy baselines, while remaining far smaller than the full JSON strategy representation as opponent branching increases.
Load-bearing premise
Top engine-ranked opponent moves are treated as the right branches both to keep in the strategy and to stress-test during evaluation, so the reported scores only upper-bound true worst-case play if those rankings miss the lines that actually break the player.
Editorial extensions
If this is right
- Strategy explanations should be judged by how well players perform with them, including off the main line, not only by fluency or concept coverage.
- High-level chess concepts alone rarely define an executable plan; concrete move contingencies are needed.
- Natural language can compress strategy trees more gracefully than structured JSON as branching factor grows.
- LLM-generated descriptions can help LLM solvers more than human solvers (and vice versa for human-written text), so audience matters.
- Self-reflection helps only when the verbalizer can already spot strategic omissions; weaker models can make descriptions worse.
Reading between the lines
- The same prune-then-verbalize pattern could apply to other perfect-information games that already have strong engines but weak human-facing explanations.
- Coaching tools may need separate generation targets for human readers versus LLM proxies, given the preference misalignment the paper reports.
- Limited worst-branch tree expansion is a practical stand-in for full exploitability when querying humans or large models is expensive.
- Concept keywords may work better as compression aids after the concrete tree is known, rather than as a substitute for it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces chess strategy verbalization: converting engine-derived (partial) strategies into natural-language descriptions that help a solving player recover and execute the strategy on chess puzzles. It proposes (i) a pipeline that builds a kstrat-pruned partial strategy tree from Stockfish and verbalizes it with LLMs under strategy/concept guidance and optional self-reflection, and (ii) Tree-Expanded Puzzle Playing, which scores descriptions by downstream engine win-probability after probing up to keval opponent replies and expanding the worst branch for the solver. Experiments on 100 Lichess puzzles (LLM solvers o3 and gpt-oss-120b), ablations on kstrat/keval, a Xiangqi transfer study, and a 30-person human study with balanced assignment show that verbalized strategies improve over no-strategy baselines for both LLMs and humans, that concept-only guidance is weak, that NL is more compact than JSON as branching grows, and that LLM vs human preferences for description style can misalign—especially off the main line.
Significance. If the results hold, the work cleanly carves out a pedagogically motivated task distinct from post-hoc move commentary and abstract concept extraction, and supplies a utility-based evaluation that is more principled than surface metrics (BLEU, G-Eval). Strengths include controlled baselines (No Strategy, C, S, R, Optimal), multi-run reporting with standard deviations, explicit ablations on branching factors, a human study with Elo-stratified participants and round-robin strategy assignment, and an honest limitations section on player confounding and LLM–human misalignment. The compactness result (Fig. 9) and the demonstration that pure concepts do not yield executable strategies are useful contributions for interpretable game AI and for NLG evaluation via downstream task performance.
major comments (3)
- [§5, Tree-Expanded Puzzle Playing; Tables 2–3; Fig. 8] Stockfish rankings define both the partial strategy tree (top-kstrat opponent moves, §4.1) and the off-main-line probe (top-keval then worst branch, §5). The authors correctly note that Tree-Expanded Puzzle Playing only upper-bounds exploitability and assumes near-optimal solver play when ranking branches. This is load-bearing for insight (a) and for the claim of a “robust” evaluation framework. The manuscript should more explicitly quantify how method rankings change between keval=1 and keval=3 (Tables 2–3 already hint at a flip: LLM descriptions look stronger on the main line; human-written ones better for humans off-main-line) and discuss the risk that engine-suboptimal but human-plausible opponent moves are systematically under-sampled. A short sensitivity experiment (e.g., sampling some non-top engine moves, or reporting rank correlation of methods across keval) would make the centr
- [§6.3; Appendix D; Tables 2–3] Human evaluation (§6.3, Appendix D) terminates early when the participant’s move on the expanded branch is not the engine-optimal move, so reported scores are upper bounds and off-main-line expansion is incomplete. The authors apply the same modified procedure to the LLM player for fairness, which is appropriate, but this weakens the strength of the claim that human-written descriptions better communicate off-main-line variations (Table 2, keval=3). The paper should state more clearly what fraction of puzzles terminate early under each condition and whether the Human vs LLM gap survives when restricted to fully expanded trajectories, or treat the human off-main-line comparison as suggestive rather than conclusive.
- [§6.2, Table 1] Table 1 reports means ± one std over five runs but no significance tests or confidence intervals on ΔBase / pairwise method differences. Several key contrasts have overlapping intervals (e.g., o3 as solver: V-S 0.813±0.037 vs S 0.851±0.009; RV-S vs S). Given that the headline is that verbalization is “beneficial but lossy” relative to JSON and better than no strategy, the paper should either run paired tests (or bootstrap over puzzles) or temper language where differences are not clearly separated. This is fixable without new experiments beyond re-analysis of existing runs.
minor comments (6)
- [Abstract; Figure 1; §5] Figure 1 and the abstract promise “objective evaluation,” but evaluation still depends on the solving player class (§8). A one-sentence clarification in the abstract or intro that utility is always relative to a player class would avoid overclaim.
- [§4.2; §6.2] Concept keywords are taken as gold from Lichess player votes (§4.2, §6.1). A brief note on noise/coverage of those tags would help readers interpret the weak V-C / C results.
- [Appendix B; §6.4] Xiangqi results (Appendix B, Table 4) are interesting but under-discussed in the main text; one sentence in §6.4 or the conclusion on what the larger S–Optimal gap implies for “JSON as instructions vs hints” would strengthen the generalization claim.
- [Appendix C; Table 8] Qualitative examples (Appendix C) are valuable; labeling illegal or hallucinated moves more systematically (as in the d5 knight/pawn case in Table 8) would make the failure modes easier to scan.
- [Abstract; §1; §4–6] Typo/consistency: “chessstrategy verbalization” appears without a space in the abstract and §1; standardize to “chess strategy verbalization.” Also “kstrat” / “keval” formatting is inconsistent in places (subscripts vs inline).
- [§2; Appendix A] Related work on human–AI alignment in chess (Maia / Maia-2) is cited; a short explicit contrast—verbalizing engine strategy vs modeling human style—would situate the pedagogical goal more sharply.
Circularity Check
No circular derivation: verbalization quality is scored by independent play outcomes, not by re-reading the description or by construction from fitted inputs.
full rationale
This is an empirical systems paper, not a first-principles derivation. The load-bearing chain is: (i) build a pruned partial engine strategy tree (top-kstrat opponent moves), (ii) verbalize it with an LLM under optional concept/self-reflection guidance, (iii) score the description by how well a solving player (LLM or human) plays under Tree-Expanded Puzzle Playing, converting final Stockfish centipawns to win probability via the fixed Lichess map and reporting ΔBase / ΔOpt. None of these steps is equivalent to its inputs by construction. Verbalization is explicitly lossy (NL typically underperforms raw JSON S; concept-only V-C is weak; self-reflection is mixed). Evaluation is not self-definitional: success is not “does the text match the tree” or “does an LLM judge the text fluent,” but whether a separate player, given only the text, reaches better engine-evaluated terminal states than a no-strategy baseline, including off-main-line branches. Shared use of Stockfish for tree construction and terminal scoring is a disclosed ground-truth assumption (§3, §5), not a fitted parameter renamed as a prediction, and not a self-citation uniqueness theorem. There is no self-citation load-bearing chain, no uniqueness imported from the authors’ prior work, and no ansatz smuggled in via citation. Comparative claims (verbalized > no-strategy; NL more compact than JSON as kstrat grows; human–LLM misalignment) remain independently falsifiable under the stated protocol. Score 0.
Assumptions & free parameters
free parameters (5)
- kstrat (partial strategy tree opponent branching)
- keval (Tree-Expanded Puzzle Playing opponent samples)
- self-reflection rounds
- Lichess puzzle evaluation subset size
- Xiangqi fixed puzzle length
assumptions (5)
- domain assumption Stockfish (and Fairy-Stockfish) best moves and centipawn/win-probability evaluations may be treated as ground-truth strategy and position quality for puzzles.
- domain assumption Top engine-ranked opponent moves are a reasonable heuristic for strategically important and exploitable branches.
- domain assumption LLMs have enough chess knowledge to verbalize strategies and act as scalable solving-player proxies.
- domain assumption Lichess formula score = (1 + e^{-0.00368·cp})^{-1} is an appropriate utility metric for puzzle outcomes.
- standard math Standard game-tree and partial-strategy formalisms (mapping states to moves; ply-limited puzzles).
invented entities (3)
-
Chess strategy verbalization (task)
-
Tree-Expanded Puzzle Playing
-
Partial strategy tree with kstrat opponent pruning
Cite this review
Pith. "Pith review of Communicating Chess Strategies in Natural Language." pith.science (2026). https://pith.science/paper/F7VMVXET
@misc{pith2026260711486,
author = {Pith},
title = {Pith review of: Communicating Chess Strategies in Natural Language},
year = {2026},
howpublished = {\url{https://pith.science/paper/F7VMVXET}},
note = {Machine review of arXiv:2607.11486}
}
read the original abstract
Chess engines have long achieved superhuman playing strength. However, the underlying strategy behind their move suggestions is difficult for human players, even skilled ones, to comprehend. Motivated by this, we propose the task of chess strategy verbalization, which is to describe chess strategies in natural language. We design (i) a pipeline for verbalizing strategies and (ii) an evaluation framework for objective evaluation of generated strategy descriptions. Our experiments show that natural language is a promising and interpretable medium for communicating strategic information to both human and LLM players. We glean additional interesting insights, including (a) the importance of evaluating strategies beyond the main line, (b) the limitations of pure concept-based descriptions, and (c) the limitations of relying on LLMs rather than humans for evaluation.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Science , volume =
David Silver and Thomas Hubert and Julian Schrittwieser and Ioannis Antonoglou and Matthew Lai and Arthur Guez and Marc Lanctot and Laurent Sifre and Dharshan Kumaran and Thore Graepel and Timothy Lillicrap and Karen Simonyan and Demis Hassabis , title =. Science , volume =
-
[2]
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=
Bridging the gap between expert and language models: Concept-guided chess commentary generation and evaluation , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=
2025
-
[3]
Learning to Generate Move-by-Move Commentary for Chess Games from Large-Scale Social Forum Data
Jhamtani, Harsh and Gangal, Varun and Hovy, Eduard and Neubig, Graham and Berg-Kirkpatrick, Taylor. Learning to Generate Move-by-Move Commentary for Chess Games from Large-Scale Social Forum Data. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018
2018
-
[4]
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=
Automated chess commentator powered by neural chess engine , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=
-
[5]
2002 , booktitle =
Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing , title =. 2002 , booktitle =
2002
-
[6]
Feng, Xidong and Luo, Yicheng and Wang, Ziyan and Tang, Hongrui and Yang, Mengyue and Shao, Kun and Mguni, David and Du, Yali and Wang, Jun , journal=. Chess
-
[7]
Complete chess games enable
Zhang, Yinqi and Han, Xintian and Li, Haolong and Chen, Kedi and Lin, Shaohui , booktitle=. Complete chess games enable
-
[8]
Kolasani, Sai and Saplin, Maxim and Crispino, Nicholas and Montgomery, Kyle and Davis, Jared Quincy and Zaharia, Matei and Wang, Chi and Wang, Chenguang , journal=
Show all 48 references
-
[9]
arXiv preprint arXiv:2412.12119 , year=
Mastering board games by external and internal planning with language models , author=. arXiv preprint arXiv:2412.12119 , year=
-
[10]
arXiv preprint arXiv:2212.08195 , year=
Improving chess commentaries by combining language models with symbolic reasoning engines , author=. arXiv preprint arXiv:2212.08195 , year=
-
[11]
2008 , publisher=
The Rating of Chessplayers: Past and Present , author=. 2008 , publisher=
2008
-
[12]
Campbell, Murray and Hoane Jr, A Joseph and Hsu, Feng-hsiung , journal=. Deep. 2002 , publisher=
2002
-
[13]
Stockfish , author =
-
[14]
Luo, Qinyu and Ye, Yining and Liang, Shihao and Zhang, Zhong and Qin, Yujia and Lu, Yaxi and Wu, Yesai and Cong, Xin and Lin, Yankai and Zhang, Yingli and others , booktitle=. Repo
-
[15]
Yang, Dayu and Simoulin, Antoine and Qian, Xin and Liu, Xiaoyi and Cao, Yuwei and Teng, Zhaopu and Yang, Grey , booktitle=. Doc
-
[16]
Summarizing Source Code using a Neural Attention Model
Iyer, Srinivasan and Konstas, Ioannis and Cheung, Alvin and Zettlemoyer, Luke. Summarizing Source Code using a Neural Attention Model. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016
2016
-
[17]
2018 , booktitle =
Hu, Xing and Li, Ge and Xia, Xin and Lo, David and Jin, Zhi , title =. 2018 , booktitle =
2018
-
[18]
Feng, Zhangyin and Guo, Daya and Tang, Duyu and Duan, Nan and Feng, Xiaocheng and Gong, Ming and Shou, Linjun and Qin, Bing and Liu, Ting and Jiang, Daxin and others , booktitle=. Code
-
[19]
Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and others , journal=. Self-
-
[20]
Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle=. G-
-
[21]
arXiv preprint arXiv:2508.10925 , year=
gpt-oss-120b & gpt-oss-20b model card , author=. arXiv preprint arXiv:2508.10925 , year=
-
[22]
Gonzalez and Hao Zhang and Ion Stoica , booktitle=
Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica , booktitle=. Efficient Memory Management for Large Language Model Serving with
-
[23]
2003 , publisher=
Chess Tactics for Kids , author=. 2003 , publisher=
2003
-
[24]
Xu, Jiannan and Li, Gujie and Jiang, Jane Yi , booktitle=. A
-
[25]
2026 , note =
Wait, Theo and Anand, Viswanathan , title =. 2026 , note =
2026
-
[26]
Proceedings of the International Joint Conference on Artificial Intelligence , pages=
Approximate Exploitability: Learning a Best Response , author=. Proceedings of the International Joint Conference on Artificial Intelligence , pages=
-
[27]
Advances in neural information processing systems , volume=
A unified game-theoretic approach to multiagent reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[28]
Scientific American , volume=
A grandmaster chess machine , author=. Scientific American , volume=. 1990 , publisher=
1990
-
[29]
Fichter, Fabian and The Fairy-Stockfish Community , url =. Fairy-
-
[30]
Xiangqi-r1: Enhancing spatial strategic reasoning in
Chen, Yuhao and Liu, Shuochen and Lyu, Yuanjie and Zhang, Chao and Shi, Jiayao and Xu, Tong , journal=. Xiangqi-r1: Enhancing spatial strategic reasoning in
-
[31]
Superhuman
Brown, Noam and Sandholm, Tuomas , journal=. Superhuman
-
[32]
Science , volume=
Human-level play in the game of diplomacy by combining language models with strategic reasoning , author=. Science , volume=. 2022 , publisher=
2022
-
[33]
Exploring large language models for communication games: An empirical study on
Xu, Yuzhuang and Wang, Shuo and Li, Peng and Luo, Fuwen and Wang, Xiaolong and Liu, Weidong and Liu, Yang , journal=. Exploring large language models for communication games: An empirical study on
-
[34]
Light, Jonathan and Cai, Min and Shen, Sheng and Hu, Ziniu , journal=. Avalon
-
[35]
Hidden-role games:
Carminati, Luca and Zhang, Brian Hu and Farina, Gabriele and Gatti, Nicola and Sandholm, Tuomas , journal=. Hidden-role games:
-
[36]
Maia-2: A unified model for human-
Tang, Zhenwei and Jiao, Difan and McIlroy-Young, Reid and Kleinberg, Jon and Sen, Siddhartha and Anderson, Ashton , journal=. Maia-2: A unified model for human-
-
[37]
Evaluation of human-
Siu, Ho Chit and Pe. Evaluation of human-. Advances in Neural Information Processing Systems , volume=
-
[38]
Aligning superhuman
McIlroy-Young, Reid and Sen, Siddhartha and Kleinberg, Jon and Anderson, Ashton , booktitle=. Aligning superhuman
-
[39]
On the critical role of conventions in adaptive human-
Shih, Andy and Sawhney, Arjun and Kondic, Jovana and Ermon, Stefano and Sadigh, Dorsa , journal=. On the critical role of conventions in adaptive human-
-
[40]
Acquisition of chess knowledge in
McGrath, Thomas and Kapishnikov, Andrei and Toma. Acquisition of chess knowledge in. Proceedings of the National Academy of Sciences , volume=. 2022 , publisher=
2022
-
[41]
arXiv preprint arXiv:1912.12191 , year=
Explain your move: Understanding agent actions using specific and relevant feature attribution , author=. arXiv preprint arXiv:1912.12191 , year=
1912 arXiv
-
[42]
2021 IEEE Conference on Games (CoG) , pages=
Some chess-specific improvements for perturbation-based saliency maps , author=. 2021 IEEE Conference on Games (CoG) , pages=. 2021 , organization=
2021
-
[43]
Towards Piece-by-Piece Explanations for Chess Positions with
Spinnato, Francesco , journal=. Towards Piece-by-Piece Explanations for Chess Positions with
-
[44]
Bridging the human--
Schut, Lisa and Toma. Bridging the human--. Proceedings of the National Academy of Sciences , volume=. 2025 , publisher=
2025
-
[45]
, author=
Unveiling Concepts Learned by a World-Class Chess-Playing Agent. , author=. Proceedings of the International Joint Conference on Artificial Intelligence , pages=
-
[46]
Extracting Search Trees from
Chen, Sixing and Li, Ji-An and Cakir, Saner and Akcali, Sinan and Lee, Kayla and Mattar, Marcelo G , journal=. Extracting Search Trees from
-
[47]
Proceedings of the International Joint Conference on Artificial Intelligence , pages=
Accelerating best response calculation in large extensive games , author=. Proceedings of the International Joint Conference on Artificial Intelligence , pages=
-
[48]
2026 , howpublished =
2026
Reviewed July 14, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.