Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that RL fine-tuning on chess puzzles lifts LLMs only to 25-30% accuracy because the models lack chess knowledge that reinforcement learning cannot create from scratch.

desk verdict A clean, honest RLVR-on-chess study with a plausible but under-supported negative result; worth reviewing, but the causal claim needs multi-seed and longer-training evidence. read the letter →

arxiv 2507.00726 v3 pith:A4LV6Y6H submitted 2025-07-01 cs.AI cs.LG

Dongyoon Hwang , Hojoon Lee , Jaegul Choo , Dongmin Park , Jongho Park This is my paper
classification cs.AIcs.LG
keywords LLMchessreasoningreinforcementlearningwithverifiablerewardsGRPOdenserewardviaaction-valuenetworkknowledgedistillationpretrainingdeficitstrategicLichesspuzzles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether reinforcement learning with verifiable rewards (RLVR) can give large language models strategic reasoning, using chess as the testbed. The authors fine-tune Qwen2.5 and Llama3.1 models with GRPO on 19.2k Lichess puzzle positions, rewarding moves with win-probability scores from a chess-pretrained action-value network instead of a binary correct/incorrect signal. Dense rewards beat sparse rewards, but every trained model plateaus at 25-30% puzzle accuracy, well below the 66.5% of expert-level chess models; adding distilled OpenAI o3 reasoning traces before RL does not break the ceiling. The paper concludes that the bottleneck is the base models' sparse internal chess knowledge—RL amplifies what pretraining already supplied and cannot create the missing domain understanding from scratch. A sympathetic reader would care because this sets a clear boundary on what post-training RL can do and redirects the recipe for new domains toward pretraining coverage.

What carries the argument

The load-bearing object is a chess-pretrained action-value network $Q_\theta(s,a)$, a 270M-parameter 16-layer transformer trained on 15B Stockfish-annotated state-action pairs with HL-Gauss loss, which outputs post-move win probability in $[0,1]$. It serves as a dense reward model, converting every candidate move—including suboptimal ones—into graded feedback to the LLM, a form of knowledge distillation. The paper pairs this with a FEN plus legal-moves prompt with <think>/<answer> tags, GRPO rollouts, and two diagnostics (FEN state-transition prediction and MATE binary move choice) that probe whether the base model can internally simulate chess.

What would settle it

Run the same dense-reward GRPO recipe on Qwen2.5-7B for several times longer (e.g., 600-1,500 steps, or with process-level rewards along the puzzle line) and measure held-out 10K-puzzle accuracy; if accuracy climbs beyond roughly 35% toward the 66.5% expert mark, the claim that RL cannot overcome the pretraining deficit is falsified. Likewise, a base model scoring near 0% on FEN tracking that reaches expert-level accuracy after pure RL would contradict the diagnosis.

Watch

Extended reading notes

Core claim

Post-training with expert dense rewards improves LLM chess tactics, but not enough: across Qwen2.5-3B/7B and Llama3.1-8B, GRPO with a 2299-Elo action-value critic as reward reaches only 25-30% held-out puzzle accuracy, relative to 66.5% for a 1800-Elo chess model, with sparse binary rewards failing outright on two of the three base models. Reasoning SFT on 1,000 o3-generated chess analyses produces much more articulate reasoning yet the same plateau, and in Llama3.1-8B it lowers accuracy. On diagnostic tasks, the models score 0% on FEN board-state tracking and 12.7-53.7% on MATE tactic selection, showing they lack faithful internal chess simulators; from this the paper argues that RL alone cannot compensate for chess knowledge missing from pretraining, and that RLVR's success in math is an amplification of knowledge that was already there.

Load-bearing premise

The conclusion depends on treating 150 GRPO steps on 19.2k puzzle samples, with every configuration run once, as sufficient evidence that no amount of additional or differently shaped RL can push past the 25-30% plateau.

Editorial extensions

If this is right

  • Dense, expert-graded rewards consistently outperform sparse correct/incorrect rewards for chess puzzle accuracy, and sparse rewards can completely fail on some base models.
  • RL fine-tuning beats supervised fine-tuning for tactical move quality, yet the 25-30% ceiling holds across model families and scales, so expert-level play is not reachable with this recipe.
  • Distilling high-quality reasoning traces (o3) before RL does not lift the plateau and can degrade Llama3.1-8B, separating reasoning form from chess content.
  • The plateau's root cause is diagnostic: base LLMs cannot reliably track board states or spot basic tactics, implying RLVR amplifies only what pretraining already encodes.
  • The contrast with RLVR success in math suggests pretraining domain coverage decides whether post-training RL can elicit expert-level reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the diagnosis is right, RLVR scale-ups in any domain with thin pretraining coverage—negotiation, novel games, specialized simulations—should show a similar ceiling; a cheap pretest is whether the base model can simulate the domain's state transitions before spending RL compute.
  • The plateau might partly reflect the 150-step, single-run training budget the paper used; longer training, a larger or more diverse puzzle pool, or process-level rewards is an untested alternative that, if it broke 30%, would falsify the strongest reading of the claim.
  • A direct testable extension: pretrain or heavily SFT the same LLMs on chess game corpora (as chess-specialized models do) and then run the identical dense-reward RL; if expert-level accuracy appears, the pretraining-deficit account is confirmed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper asks whether LLMs can develop strategic reasoning through reinforcement learning with verifiable rewards, using chess as a testbed. The authors train Qwen2.5-3B/7B and Llama3.1-8B with GRPO on 19.2k position-action pairs from Lichess puzzles, comparing sparse binary rewards against dense rewards produced by a pretrained chess action-value network. They report three main findings: dense rewards usually outperform sparse rewards; all models plateau at roughly 25-30% puzzle accuracy, far below the 66.5% accuracy of the expert critic; and additional reasoning SFT from OpenAI o3 traces does not lift the plateau. Diagnostic experiments on board-state comprehension and mate detection show near-zero or weak performance, which the authors interpret as evidence that base LLMs lack internal chess knowledge and that RL alone may not overcome this pretraining deficit. The paper also includes ablations on prompt format, move notation, board representation, and reward type.

Significance. If the central claim holds, the paper is a valuable negative result for RLVR: it identifies a domain where reward optimization amplifies existing knowledge but does not create de novo strategic competence, and it provides a concrete diagnostic framework (board-state comprehension, mate detection, legal-move sensitivity) for testing pretraining knowledge. The dense-reward distillation from a chess critic is a useful methodological contribution, and the ablations on SAN vs. UCI notation and on the necessity of legal moves are informative. The paper also ships code and uses held-out ground-truth puzzle evaluation, which strengthens reproducibility. However, the main conclusion is currently under-supported because the evidence is limited to a fixed, short training budget and single runs per configuration, so the significance depends on whether the plateau is a true convergence result rather than an artifact of the training setup.

major comments (3)
  1. [§3.1, Fig. 2, Table 2] The central plateau claim rests on exactly 150 GRPO training steps, a single run per model/reward configuration, with no seeds, no error bars, and no convergence analysis. In RLVR settings for mathematical reasoning, performance often continues to improve well beyond 150 steps, so the reader cannot distinguish a genuine ceiling from an undertrained policy. The authors should provide either (i) training curves extended until evaluation accuracy is flat for a sustained number of steps, (ii) multiple seeds with variance estimates, or (iii) an explicit comparison at larger step budgets and larger training datasets, before claiming that RL alone cannot overcome the deficit.
  2. [§3.2, Fig. 3] The decline of Llama3.1-8B after o3-SFT followed by RL is interpreted as evidence against the value of reasoning traces, but the same curve is also consistent with optimization instability, KL collapse, or reward hacking in the dense-reward critic. The paper does not report reward curves, KL divergence, format-compliance rates, or the accuracy of the final policy on the SFT corpus. Reporting these quantities would clarify whether the decline reflects a knowledge limit or a training failure.
  3. [§3.3, Table 1] The comparison against 'expert-level chess models' achieving 66.5% puzzle accuracy is not apples-to-apples: the 66.5% figure comes from a 270M-parameter action-value network trained for 10M optimization steps on 15B Stockfish-annotated state-action pairs, whereas the LLMs are general-purpose pretrained models fine-tuned for 150 GRPO steps on 19.2k samples. The plateau could therefore reflect the RL recipe, the reward formulation, the prompt interface, or the amount of chess-specific data, rather than a fundamental inability of pretrained LLMs to learn chess. The authors should either provide a matched comparison (e.g., an LLM trained on comparable chess-specific data) or weaken the conclusion to 'RLVR with this budget and this reward formulation does not overcome the deficit in these models.'
minor comments (4)
  1. [§2.1, §3.1] The evaluation metric 'puzzle accuracy' is defined as exact match of the complete action sequence; since the model outputs only one move per position, it would help to clarify how the full sequence is obtained during evaluation and whether any rollout search or greedy decoding is used.
  2. [§3.3, Table 1] The diagnostic results are reported as single numbers without confidence intervals or sample sizes. Given that some entries are exactly 0.0%, adding the number of evaluation examples and any error bars would make the failure analysis more convincing.
  3. [Appendix C.2] The comparison between direct win-rate feedback and normalized rank feedback is useful, but the claim that 'both methods converge to similar final performance' is based on the same single-run issue; please state the number of runs used for these ablations.
  4. [Appendix B.2] The reasoning SFT corpus is only 1,000 examples, and the paper does not report how many o3 traces were filtered for correctness or formatting. A sentence describing trace-quality filtering and the number of rejected traces would help the reader assess the corpus.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central plateau claim is benchmarked against external held-out ground truth and an external teacher critic, not against any quantity fitted in this paper.

full rationale

The paper's central negative result—that GRPO-based RLVR plateaus at 25–30% puzzle accuracy and that this reflects a pretraining knowledge deficit—is not derived from the paper's own fitted quantities. The reward critic Qθ is an externally pretrained 270M-parameter action-value network from Ruoss et al. (2024), trained on 15B Stockfish-annotated state-action pairs; its parameters are not fit in this paper. Puzzle accuracy is measured against 10K held-out Lichess puzzle ground-truth solutions, an external benchmark, and the 66.5% expert comparison is quoted from Ruoss et al. (2024). The reasoning-SFT experiment uses OpenAI o3 traces, but the subsequent RL evaluation is again against the same held-out ground truth, and the o3 data is not used to define the success metric. The failure analysis in Section 3.3 uses independent diagnostics (FEN state-transition prediction and MATE move selection) that are not constructed from the RL training data. No equation in the paper defines its conclusion in terms of its inputs: the dense reward is Qθ(s,a) from an external critic, the sparse reward is a ground-truth match indicator, and both are optimized by GRPO with standard hyperparameters listed in Table 2. The only self-citation involving the authors (Park et al., 2025, cited for examples of strategic-reasoning settings) is contextual and is not load-bearing for any result. The manuscript also states its own limitation: Figures 2–3 and Table 2 report a single 150-step training budget with one run per configuration, which is a generalizability concern rather than a circularity. Accordingly, no circular step meets the evidentiary bar of equation-to-equation reduction or fitted-input-as-prediction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on a few hand-chosen experimental parameters (reward coefficients, training steps, SFT corpus size, rollout count) and on domain assumptions about the validity of puzzle accuracy, the reliability of the external critic, the absence of chess knowledge in base models, the suitability of GRPO, and the correctness of o3 traces. No new physical or conceptual entities are postulated.

free parameters (4)
  • Reward coefficients = λsparse/λdense = 1/0 or 0/1; λfmt = 0.1; λlang = 0.1
    Hand-chosen coefficients define the sparse vs dense reward comparison; the central comparison sets the opposite reward coefficient to zero.
  • Training steps = 150
    The plateau conclusion is defined by this fixed step count; single runs per configuration determine whether the curve has truly saturated.
  • SFT reasoning corpus size = 1,000
    The reasoning SFT ablation uses only 1,000 o3 traces, so the null result may depend on corpus size or quality.
  • Rollout count = 8
    GRPO samples 8 rollouts per prompt; this exploration budget shapes the reward estimates and could affect the observed plateau.
assumptions (5)
  • domain assumption Lichess puzzle accuracy is a valid measure of chess strategic reasoning and expert-level play.
    The paper equates expert-level chess understanding with roughly 60-80% puzzle accuracy and uses full-solution exact match as the evaluation metric. Puzzle solving is a narrow tactical skill, so this may not capture all strategic reasoning. Invoked in Section 3.1.
  • domain assumption The pretrained action-value network Qθ from Ruoss et al. is a reliable oracle for move quality.
    Dense rewards are taken from a 270M-parameter critic with 2299 Elo and 95.4% puzzle accuracy. If this critic is biased, the dense reward rankings propagate that bias. Invoked in Section 2.2.
  • domain assumption Qwen2.5 and Llama3.1 base models have insufficient chess-specific pretraining exposure.
    The paper infers a pretraining deficit from diagnostic task performance rather than measuring chess-token prevalence in pretraining corpora. This assumed cause is not directly measured. Discussed in Sections 3.3 and 5.
  • domain assumption GRPO is a valid policy optimization method that does not itself cause the plateau.
    All RL results use the same GRPO implementation and hyperparameters; no alternative RL algorithm is tested, so the plateau is attributed to knowledge limits rather than to a policy optimization artifact. Invoked in Section 2.2.
  • domain assumption OpenAI o3 reasoning traces are high-quality and suitable distillation targets.
    The reasoning SFT corpus is generated by a proprietary model without systematic verification of correctness beyond qualitative examples. Invoked in Section B.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess." pith.science (2026). https://pith.science/paper/A4LV6Y6H

@misc{pith2026250700726,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4LV6Y6H}},
  note         = {Machine review of arXiv:2507.00726}
}
read the original abstract

While reinforcement learning (RL) for large language models (LLMs) has shown promise in mathematical reasoning, strategic reasoning for LLMs using RL remains largely unexplored. We investigate whether LLMs can develop strategic reasoning capabilities through RL in chess. To this end, we leverage a chess-pretrained action-value network to provide dense reward on the LLM's output move quality, which can be seen as a form of knowledge distillation. Our experiments show that our distillation-based dense rewards often outperform sparse binary rewards. However, surprisingly, all models plateau far below expert levels. We provide SFT and RL ablations on chess reasoning training and find evidence that this limitation stems from a deficit in the pretrained models' internal understanding of chess-a deficit which RL alone may not be able to fully overcome. The code is available at https://github.com/krafton-ai/Chess-R1.

Figures

Figures reproduced from arXiv: 2507.00726 by the authors.

Figure 1
Figure 1. Overview: (a) A data sample from the Lichess puzzle dataset is formatted into a prompt that includes the current board state and the set of legal moves. (b) At each GRPO step, the policy model generates multiple rollouts of predicted actions. A reward model evaluates these rollouts with dense feedback, including sub-optimal actions. Optionally, reasoning traces from OpenAI o3 are fine-tuned into the base model befor… view at source ↗
Figure 2
Figure 2. Evaluation performance comparison of RL fine-tuned models. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Evaluation performance of models trained [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Input prompt format for chess reasoning tasks with FEN for board state and SAN [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Token length distribution of o3 reasoning data per each tokenizer type. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Evaluation performance comparison with and without legal moves in the input [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Evaluation performance comparison across move notations (SAN vs. UCI) for [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Evaluation performance comparison across board state representations (FEN vs. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Evaluation performance comparison between direct win-rate feedback and nor [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Training performance comparison between Qwen2.5-7B Base and Instruct model. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Qualitative reasoning example from Qwen2.5-3B after RL fine-tuning with sparse [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Qualitative reasoning example from Qwen2.5-3B after RL fine-tuning with dense [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Qualitative reasoning example from Qwen2.5-7B after RL fine-tuning with sparse [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Qualitative reasoning example from Qwen2.5-7B after RL fine-tuning with dense [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Qualitative reasoning example from Qwen2.5-7B after RL fine-tuning with dense [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Qualitative reasoning example from Llama3.1-8B after RL fine-tuning with sparse [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Qualitative reasoning example from Llama3.1-8B after RL fine-tuning with sparse [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Qualitative reasoning example from Llama3.1-8B after RL fine-tuning with dense [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Qualitative reasoning example from Qwen2.5-7B after reasoning SFT (using [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Qualitative reasoning example from Qwen2.5-7B after reasoning SFT (using [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: Qualitative reasoning example from Llama3.1-8B after reasoning SFT (using [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 22
Figure 22. Figure 22: Qualitative reasoning example from Llama3.1-8B after reasoning SFT (using [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Example reasoning trace generated by OpenAI [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Example reasoning trace generated by OpenAI [PITH_FULL_IMAGE:figures/full_fig_p028_24.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In a chess latent-reasoning model, replacing or removing the silent thought vectors barely changes moves, so the RL improvement appears to be encoded in the weights, not in a consulted scratchpad.

Reference graph

Works this paper leans on

34 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Steven J. Edwards. Standard: Portable game notation specification and implementation guide, 1994. URL https://ia802908.us.archive.org/26/items/pgn-standard-1994-03-12/PGN_standard_1994-03-12.txt

    work page 1994

  3. [3]

    Chessgpt: Bridging policy learning and language modeling

    Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. Advances in Neural Information Processing Systems, 36: 0 7216--7262, 2023

    work page 2023

  4. [4]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  6. [6]

    Learning to reason for long-form story generation

    Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation. arXiv preprint arXiv:2503.22828, 2025

    arXiv 2025

  7. [7]

    Improving regression performance with distributional losses

    Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International conference on machine learning, pp.\ 2157--2166. PMLR, 2018

    work page 2018

  8. [8]

    Bridging the gap between expert and language models: Concept-guided chess commentary generation and evaluation

    Jaechang Kim, Jinmin Goh, Inseok Hwang, Jaewoong Cho, and Jungseul Ok. Bridging the gap between expert and language models: Concept-guided chess commentary generation and evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pap...

    work page 2025

Show all 34 references
  1. [9]

    T " ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  2. [10]

    Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 2025

    Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 2025

    2025 arXiv

  3. [11]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025 a

    2025 arXiv

  4. [12]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025 b

    2025 arXiv

  5. [13]

    Orak: A foundational benchmark for training and evaluating llm agents on diverse video games

    Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, et al. Orak: A foundational benchmark for training and evaluating llm agents on diverse video games. arXiv preprint arXiv:2506.03610, 2025

    2025 arXiv

  6. [14]

    Qwen2.5 technical report

    Qwen:, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le...

    2025 arXiv

  7. [15]

    Amortized planning with large-scale transformers: A case study on chess

    Anian Ruoss, Gr \'e goire Del \'e tang, Sourabh Medapati, Jordi Grau-Moya, Li K Wenliang, Elliot Catt, John Reid, Cannada A Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess. Advances in Neural Information Processing ...

    2024

  8. [16]

    Spurious rewards: Rethinking training signals in rlvr

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947, 2025

    2025 arXiv

  9. [17]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  10. [18]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  11. [19]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:17...

    2017 arXiv

  12. [20]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  13. [21]

    Explore the reasoning capability of llms in the chess testbed

    Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. Explore the reasoning capability of llms in the chess testbed. arXiv preprint arXiv:2411.06655, 2024

    2024 arXiv

  14. [22]

    Reinforcement learning for reasoning in large language models with one training example

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025

    2025 arXiv

  15. [23]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  16. [24]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  17. [25]

    Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning

    Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning. arXiv preprint arXiv:2502.19655, 2025 a

    2025 arXiv

  18. [26]

    Llm as a mastermind: A survey of strategic reasoning with large language models

    Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm as a mastermind: A survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230, 2024

    2024 arXiv

  19. [27]

    Complete chess games enable LLM become a chess master

    Yinqi Zhang, Xintian Han, Haolong Li, Kedi Chen, and Shaohui Lin. Complete chess games enable LLM become a chess master. Association for Computational Linguistics, 2025 b

    2025

  20. [28]

    Distill not only data but also rewards: Can smaller language models surpass larger ones? arXiv preprint arXiv:2502.19557, 2025 c

    Yudi Zhang, Lu Wang, Meng Fang, Yali Du, Chenghua Huang, Jun Wang, Qingwei Lin, Mykola Pechenizkiy, Dongmei Zhang, Saravan Rajmohan, et al. Distill not only data but also rewards: Can smaller language models surpass larger ones? arXiv preprint arXiv:2502.19557, 2025 c

    2025 arXiv

  21. [29]

    Absolute zero: Reinforced self-play reasoning with zero data

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025 a

    2025 arXiv

  22. [30]

    Echo chamber: Rl post-training amplifies behaviors learned in pretraining

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025 b

    2025 arXiv

  23. [31]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024 arXiv

  24. [32]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  25. [33]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  26. [34]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.