Pith. sign in

REVIEW 4 major objections 4 minor 63 references

Silent expert systems — game solvers, planners, verifiers — can teach LLMs to reason by keeping only the explanations that measurably improve the model's own actions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:27 UTC pith:HBDBYXI4

load-bearing objection The core idea is real and most of the empirics are careful, but the Flop Hold'em comparison is internally contradictory about the supervision target, so the headline +60 mbb/g is not yet interpretable. the 4 major comments →

arxiv 2607.21856 v1 pith:HBDBYXI4 submitted 2026-07-23 cs.LG cs.AIcs.CL

LeAct: Learning to Reason from Expert Actions

Ziran Yang , Chengshuai Shi , Raj Ghugare , Benjamin Eysenbach , Karthik Narasimhan , Chi Jin This is my paper
classification cs.LG cs.AIcs.CL
keywords chain-of-thoughtlatent variable reasoningexpert systemsimportance-weighted autoencoderexpert iterationimperfect-information gamesimitation learningreasoning supervision
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that any system that routinely commits to near-optimal actions without writing down its reasoning — a poker solver, a classical planner, a theorem prover — can be turned into a teacher of natural-language reasoning for language models. Its method, LeAct, treats the missing chain of thought as a latent variable: for each expert action, the student samples several candidate explanations and keeps only those that increase the student's own probability of recovering that action. The retained explanations are then used as supervised fine-tuning data, with the expert's full action distribution as the target rather than the model's own guess. Across imperfect-information games at several scales and a simulated robotics benchmark, a model trained this way matches the solver floor on small games, beats behavior-cloning and expert-iteration baselines on large games, and is the only recipe that improves on direct imitation in the robotics setting. If the claim holds, expert systems become a categorically new source of reasoning supervision, independent of human annotation or distillation from stronger LLMs.

Core claim

The central claim is that a 'silent' action oracle can be converted into a chain-of-thought teacher by closing the loop on the student's own likelihood. LeAct samples candidate explanations conditioned on the oracle's action (backward generation), scores each by the log-density gain Δ(z;x,y) = log p(y|x,z) − log p(y|x) — how much that explanation raises the student's probability of the expert action — and keeps only candidates with positive, top-K deltas. The retained traces are trained with the oracle's full action distribution as the supervised target, a choice the paper derives from an importance-weighted autoencoder bound and calls expert-policy forcing. The paper's theoretical contribut

What carries the argument

The load-bearing object is the factored importance weight in the IWAE bound: each candidate weight separates into an action-recovery ratio exp(Δ) and a proposal ratio ρ that the method drops. LeAct's selection rule is a hard top-K filter on positive Δ, computed through a parsed textual policy proxy, and the M-step is expert-policy forcing — training on the oracle's full action distribution rather than the model's own decoded action. Δ, defined in one phrase as how much a candidate explanation raises the student's probability of the expert action, is the named identity that carries the argument: it turns an unobserved reasoning trace into a graded, trainable signal.

Load-bearing premise

The derivation rests on dropping an intractable term from the importance weights — the ratio of how likely the explanation is under the student's own model versus under the backwards generator — and assuming it does not correlate with the action-recovery term across candidates; the paper defends this with a conditional argument and a single-domain audit, so the theoretical grounding collapses if that correlation appears elsewhere.

What would settle it

Compute the full importance weights (including the dropped proposal ratio) using token-level log-likelihoods on a new domain with low-probability oracle actions, and check whether the top-K explanation set changes; a systematic ranking mismatch would falsify the rho-drop. Independently, running LeAct with no LLM-written coldstart — random CoT seeds plus the forward-delta filter — would settle whether the claimed non-LLM supervision truly avoids LLM seeding.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Game engines, classical planners, theorem provers, and verifiers that only output actions become a scalable source of chain-of-thought supervision for LLM training.
  • Reasoning data can be graded by whether it helps the student's own decision-making, replacing plausibility filtering as the selection criterion.
  • The recipe works where the optimal answer is a mixed strategy rather than a single token, extending supervision to settings where exact correctness is undefined.
  • A student trained this way generalizes beyond demonstrated states: it beats expert-iteration baselines on held-out poker infosets and on out-of-distribution robotics tasks.
  • The oracle need not be exact: an approximate regret-minimization poker policy and successful frontier-model trajectories both work without modifying the recipe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to proof checkers and verifiers whose 'actions' are certificates: the same Δ filter would grade candidate proof sketches by whether they raise the model's probability of emitting the verified step.
  • The rho-drop makes a concrete prediction: domains with rare but decisive low-probability actions, where the proposal ratio varies widely across candidates, should show degraded top-K selection; an audit like the paper's token-level agreement study would reveal it.
  • The current recipe still relies on an LLM-generated coldstart to seed the first round of candidates; a version that bootstraps from random or rule-based CoT seeds would test the stronger 'non-LLM source' claim directly.
  • Because the filter scores explanations against the student's own likelihood, the method is self-referential by design; this suggests LeAct could be composed with online RL, using the oracle's action distribution as a process-level baseline for credit assignment — an extension the paper names but does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces LeAct, a training recipe that converts silent action-only experts (CFR solvers, DeepCFR policies, frontier-model trajectory pools) into natural-language chain-of-thought supervision. The student samples candidate CoTs conditioned on the expert action, scores each by the increase in its own probability of recovering that action (forward delta), retains a top-K positive-delta subset, and fine-tunes with expert-policy forcing. The authors derive the procedure as an approximate IWAE M-step, and report experiments on Leduc variants, Liar's Dice, 3-player Leduc, Flop Hold'em, and BuilderBench, claiming that LeAct matches the solver on small games, beats behavior cloning and expert-iteration baselines on held-out states, and is the only recipe improving on direct imitation in the robotics setting.

Significance. If the claims hold, LeAct would open a genuinely new supervision channel for reasoning models: silent structured experts could provide natural-language reasoning data without human annotation or stronger-LLM distillation. The paper's strengths are real: a formal latent-variable framing that is honestly labeled as approximate; explicit ablations showing the delta filter is more informative than random ranking; a selection-fidelity audit of the parsed-policy proxy against token-level log-likelihoods (89–98% top-1 agreement); and a broad evaluation spanning several game families and a robotics benchmark with held-out generalization splits. The main weaknesses concern internal consistency of the Flop Hold'em comparison and the scope of the 'non-LLM source' claim.

major comments (4)
  1. [§4 vs. Appendix D.3] There is a direct contradiction about the COT EXIT baseline used in the headline Flop Hold'em (FHP) result. Section 4 defines COT EXIT as using the model's own parsed action distribution as the SFT target ('not π*'), i.e., no expert-policy forcing. Appendix D.3 states that 'Both methods compared use expert-policy forcing (matched supervision target)' and that 'both targets are the oracle Nash policy.' These cannot both be true. The issue is consequential: Table 3 shows a strong method×target interaction (LeAct+EPF 0.157 vs. LeAct+no-EPF 1.057; COT EXIT+EPF 1.028 vs. COT EXIT+no-EPF 0.443). If the FHP +60 mbb/g and 5× KL-gap comparisons used the EPF variant of COT EXIT, then the main-text baseline does not match the reported comparison, and the gain may be attributable to the supervision target rather than backward-delta selection. If they used the no-EPF variant, Appendix D.3's matching
  2. [§4, §7, App. H.3] The claim that LeAct provides a 'categorically new source of reasoning teachers' that 'does not require a stronger LLM' is qualified, and arguably contradicted, by the method's coldstart requirement. Section 4 states that the base Qwen3-8B model lacks the competence to seed reasoning traces directly, so both iterative methods 'initialise from a frontier-LLM-generated coldstart corpus.' Appendix H.3 confirms that the coldstart corpus is generated by Claude/Gemini/GPT-5 agents for the poker domains and by frontier-model trajectory pools for BuilderBench. Thus the pipeline as evaluated does depend on stronger LLM supervision in its first stage; the non-LLM oracle only enters after this warm start. The conclusion's 'does not require a stronger LLM' is therefore too strong as written. The authors should either present a coldstart-free variant or reframe the claim as 'reduces dependence on str
  3. [App. A.1, Step 3, Eqs. (10)–(12)] The theoretical grounding rests on dropping the rho factor in the IWAE weight and replacing the softmax with a hard top-K filter. The appendix is honest that this is an approximation and gives a conditional argument for when the rho-drop preserves ranking: rho must not correlate with Delta across candidates. However, the paper provides no direct test of that correlation; the one-domain proxy audit (Table 5) tests parsed-proxy vs. token-level Delta, not the rho-Delta correlation. The random-ranking ablation (§6) shows that a Delta-based filter is informative, but it does not establish that the retained CoTs are the ones that would have high IWAE weight under the true posterior. This is load-bearing for the claim that LeAct is an 'approximate IWAE M-step' rather than a heuristic filter. Please add a concrete diagnostic for the rho-Delta correlation (e.g., estimating rho on a subset of cand
  4. [Abstract, App. F.2, §5.2] The FHP scale and evaluation coverage are overstated relative to what is actually tested. The abstract and Section 5.2 describe Flop Hold'em as '~10^9 infosets,' and the abstract's '5× closer to the solver' and '+60 mbb/g' are headline claims. Appendix F.2 states that the DeepCFR teacher table covers only 2,343,732 unique infosets along policy-induced trajectories, and that KL evaluation is computed exclusively on the held-out subset of those 2.3M infosets, not on the full ~10^9 tree. The chip-outcome matrix (Table 2b) is also restricted to trained-vs-trained comparisons among three methods. The paper should state clearly that FHP results are on the teacher's 2.3M-infoset coverage and that the ~10^9 figure is the full-tree estimate, not the evaluation domain. This is a scope limitation, not an error, but it affects how a reader interprets the advertised scale.
minor comments (4)
  1. [§4, Table 8] The paper selects the best round out of 3 and the best epoch across checkpoints for each method and setting. This is a matched protocol across methods, but it inflates absolute headline numbers. Please report the final-round/final-epoch numbers as well, or clarify that 'best-of-rounds' is the intended comparison and note the selection rule in captions.
  2. [Table 8 / App. D.1] Error bars are computed over only 3 inference-rollout seeds of a single trained policy per cell. This is a small sample for statements like 'lands within solver-evaluation noise.' Please either add more seeds for at least the headline comparisons or soften language accordingly.
  3. [App. D.3] The phrase 'Both methods compared use expert-policy forcing' is ambiguous even apart from the contradiction with §4: Table 2b includes three methods (NOCOT BC, COT EXIT, LeAct), and the main text says NOCOT BC uses no CoT channel. Please clarify exactly which rows and columns of Table 2b share which supervision target.
  4. [Fig. 2 caption] The caption shows 'gpt-5.5' in the first panel as an anchor, but the legend in the figure appears to list it as a method. Please align the caption and legend, and state whether gpt-5.5 is zero-shot or coldstart-initialized.

Circularity Check

0 steps flagged

No significant circularity: LeAct's derivation is an externally anchored approximate-EM M-step with disclosed approximations; the FHP supervision-target ambiguity is a confound, not a definitional reduction.

full rationale

LeAct's derivation chain is not circular. The objective J(θ)=E_x E_{y~π*} log pθ(y|x) is an external likelihood of expert actions, and the IWAE bound (Eq. 2), the Bayes-factorized weight (Eq. 4), and the M-step (Eq. 5) follow standard variational-EM algebra from Burda et al. [7]. The two approximations — dropping the ρ factor (App. A.1 Step 3) and replacing softmax IWAE weights with a hard top-K filter — are explicitly disclosed and empirically audited (proxy-vs-token agreement 89–98%; random-ranking ablation in §6). They do not smuggle the conclusion into the input. The Δ filter is self-referential in that it uses the student's own likelihoods to choose CoTs, but that is the intended E-step of the algorithm, and the supervision target y~π* is fixed by an external oracle rather than by the trained model. All headline comparisons are on held-out states and actions (FHP 20K hold-out, Leduc rank-split, BuilderBench OOD), so the loop is anchored outside the fitted values. Self-citations (e.g., BuilderBench [9], Goedel-Prover [23,26]) are benchmarks or related-work references and are not load-bearing theorems; none is invoked to forbid alternatives or to justify the core objective. The App. D.3 vs §4 discrepancy about whether COT EXIT used expert-policy forcing at FHP is a genuine methodological ambiguity that affects interpretation of the +60 mbb/g result, but it is a target-choice confound rather than a definitional reduction: the result does not become equal to its input by construction. The frontier-LLM coldstart similarly weakens the strength of the 'non-LLM source' novelty claim without making the derivation circular. Overall, no step reduces to its own inputs, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No invented entities are introduced. The ledger shows the algorithm contributes a filter and an EPF target while assuming the heavy lifting as input: an accurate oracle, a competent coldstart, and unverified conditions on the rho-drop. All headline results also depend on hand-set selection hyperparameters K, N, M, and the positive-Delta threshold.

free parameters (5)
  • Top-K selection count K = 2
    K=2 is standardized across the paper. App. B shows K=1 underperforms while K=2 and K=4 are comparable, so K=2 is a hand-chosen compute/performance trade-off that affects all reported results.
  • Candidate count N = 8 (8-32 in some settings)
    N=8 backward CoT candidates per state is the default; Table 11 varies N by setting. The size of the candidate pool directly controls how much useful reasoning the filter can find.
  • Baseline samples M = 4
    The baseline log p_theta(y|x) in Delta is estimated by Monte Carlo from M=4 forward decodings. A small M makes the selection baseline noisy.
  • Positive-Delta threshold = 0
    Only candidates with average Delta > 0 enter the SFT pool. This admission criterion is load-bearing and is set to zero by construction.
  • BuilderBench heuristic reward weights and distance threshold = 1, 0.1, max(0, 1 - distance); 1 cm
    The single-action robotics oracle is scored by a hand-designed action-match reward with arbitrary partial credit weights and a 1 cm position threshold. This changes which CoTs are selected in the robotics domain.
axioms (5)
  • standard math The IWAE bound and its per-sample gradient hold for a fixed LLM proposal q_bwd.
    Used to derive Eqs. (2)-(3); taken from Burda et al. [7, Eq. 8].
  • domain assumption Dropping the rho factor preserves the ranking induced by the full IWAE weight.
    App. A.1 Step 3 assumes rho varies approximately constantly for high-probability oracle actions and does not systematically correlate with Delta. Only a conditional argument and a one-domain proxy audit support this.
  • domain assumption After coldstart, the backward proposal covers enough positive-Delta CoT candidates.
    Sec. 4 states the base Qwen3-8B lacks the competence to seed reasoning traces directly, so a frontier-LLM coldstart is required. If the proposal contains no useful traces, top-K selection has nothing to select.
  • domain assumption The DeepCFR policy is a sufficiently accurate proxy for Nash at Flop Hold'em.
    App. F.2 admits DeepCFR provides an approximate Nash policy, not an exact one, and KL evaluation is computed only on its 2,343,732 policy-visited infosets.
  • domain assumption The textual policy line faithfully represents the decoder's action distribution for Delta.
    App. A.4 describes the parsed-policy proxy as a self-report; agreement with a token-level estimator is audited at 89-98% top-1 in one domain only.

pith-pipeline@v1.3.0-alltime-deepseek · 28285 in / 17515 out tokens · 190981 ms · 2026-08-01T06:27:09.186968+00:00 · methodology

0 comments
read the original abstract

Modern reasoning models depend on reasoning data, today sourced from human annotations or distilled from stronger LLMs. However, a rich and largely untapped source of supervision lies in expert systems (e.g., game engines, classical planners, theorem provers), which routinely produce near-optimal actions across diverse domains. But these experts are silent: they commit to an action without writing down the chain of thought (CoT) behind it. Recovering that CoT as natural-language reasoning would distill expert knowledge into a student that generalizes beyond the demonstrated actions. We treat it as a latent variable and study how to recover it from the action alone. Our approach, LeAct (Learning to reason from Actions), optimizes this latent variable: the student samples candidate CoTs for each expert action, and we retain those that measurably improve its own probability of recovering the action. Across imperfect-information games at multiple scales and a simulated robotics benchmark, LeAct reaches the solver's numerical floor on small enumerable games. At larger scale, it is $5\times$ closer to the solver than the strongest expert-iteration baseline. At Flop Hold'em ($\sim 10^9$ infosets), LeAct wins head-to-head by $+60$ mbb/g, and on the robotics probe it is the only training recipe that improves on direct imitation. We present a principled framework and the result: expert systems become a categorically new source of reasoning teachers for foundation models.

Figures

Figures reproduced from arXiv: 2607.21856 by Benjamin Eysenbach, Chengshuai Shi, Chi Jin, Karthik Narasimhan, Raj Ghugare, Ziran Yang.

Figure 1
Figure 1. Figure 1: LeAct turns silent action experts into reasoning teachers. For each state x, the student samples candidate explanations conditioned on the oracle’s action π ∗ (· | x). We keep those that raise the student’s predicted probability of the action and use them as CoT to fine-tune the student. sampling reasoning that recovers it, and training on the reasoning that survived the filter. The recipe accepts any acti… view at source ↗
Figure 2
Figure 2. Figure 2: Imitation quality: LeAct matches NOCOT BC where memorisation fits, and dominates where it doesn’t. Five game settings (4K–80K infosets, log y-axis, lower better, BON uses N=8) and BuilderBench in-domain 26 tasks (linear y-axis, higher better; N=64). Numbers in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Two falsifiers of the shortcut hypothesis for backward-delta selection. (a) Forward-delta vs. random ranking on Single and BON (Leduc 10r4s). (b) Mean +∆ by verbatim-π ∗ status. Method Regime Single (↓) BON (↓) Coldstart — 1.6701 0.2824 COT EXIT EPF 1.0278 0.6191 COT EXIT no-EPF 0.4432 0.0599 LeAct EPF 0.1566 0.0019 LeAct no-EPF 1.0565 0.1940 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 37 linked inside Pith

  1. [1]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  2. [2]

    Thinking fast and slow with deep learning and tree search

    Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  3. [3]

    Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales.Fundamenta Mathematicae, 3(1):133–181, 1922

    Stefan Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales.Fundamenta Mathematicae, 3(1):133–181, 1922

    1922

  4. [4]

    Reweighted wake-sleep

    Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. InInternational Conference on Learning Representations (ICLR), 2015. arXiv:1406.2751

    Pith/arXiv arXiv 2015

  5. [5]

    Superhuman AI for multiplayer poker.Science, 365 (6456):885–890, 2019

    Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker.Science, 365 (6456):885–890, 2019

    2019

  6. [6]

    Deep counterfactual regret minimization

    Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. InProceedings of the 36th International Conference on Machine Learning (ICML), volume 97 ofPMLR, pages 793–802, 2019

    2019

  7. [7]

    Importance weighted autoencoders

    Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference on Learning Representations, 2016

    2016

  8. [8]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  9. [9]

    BuilderBench: The building blocks of intelligent agents,

    Raj Ghugare, Roger Creus Castanyer, Catherine Ji, Kathryn Wantlin, Jin Schofield, Karthik Narasimhan, and Benjamin Eysenbach. BuilderBench: The building blocks of intelligent agents,

  10. [10]

    Reinforced self-training (ReST) for language modeling.arXiv preprint arXiv:2308.08998, 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (ReST) for language modeling.arXiv preprint arXiv:2308.08998, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning.Nature, 645(8081):633–638, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning.Nature, 645(8081):633–638, 2025. doi: 10.1038/s41586-025-09422-z. arXiv:2501.12948

    Pith/arXiv arXiv 2025

  12. [12]

    Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992, 2023

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Teaching large language models to reason with reinforcement learning.arXiv preprint arXiv:2403.04642, 2024

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning.arXiv preprint arXiv:2403.04642, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Large language models are reasoning teachers

    Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    V-STaR: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Olympiad-level formal mathemati- cal reasoning with reinforcement learning.Nature, 651:607–613, 2025

    Thomas Hubert, Rishi Mehta, Laurent Sartran, et al. Olympiad-level formal mathemati- cal reasoning with reinforcement learning.Nature, 651:607–613, 2025. doi: 10.1038/ s41586-025-09833-y. 11

    2025

  17. [17]

    Large language models are zero-shot reasoners.Advances in Neural Information Processing Systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in Neural Information Processing Systems, 35:22199–22213, 2022

    2022

  18. [18]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    A unified game-theoretic approach to multiagent reinforcement learning

    Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. InAdvances in Neural Information Processing Systems, volume 30, pages 4191–4204, 2017

    2017

  21. [21]

    OpenSpiel: A framework for reinforcement learning in games.arXiv preprint arXiv:1908.09453, 2019

    Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. OpenSpiel: A framework for reinforcement learning in games.arXiv preprint arXiv:1908.09453, 2019

    Pith/arXiv arXiv 1908

  22. [22]

    Measuring faithfulness in chain-of-thought reasoning.arXiv preprint arXiv:2307.13702, 2023

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning.arXiv preprint arXiv:2307.13702, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Goedel-Code-Prover: Hierarchical proof search for open state-of-the-art code verification.arXiv preprint arXiv:2603.19329, 2026

    Zenan Li, Ziran Yang, Deyuan He, Haoyu Zhao, Andrew Zhao, Shange Tang, Kaiyu Yang, Aarti Gupta, Zhendong Su, and Chi Jin. Goedel-Code-Prover: Hierarchical proof search for open state-of-the-art code verification.arXiv preprint arXiv:2603.19329, 2026

    arXiv 2026

  24. [24]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050

    Pith/arXiv arXiv 2024

  25. [25]

    Goedel-Prover: A frontier model for open-source automated theorem proving.arXiv preprint arXiv:2502.07640, 2025

    Yong Lin, Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang, Jia Li, Mengzhou Xia, Danqi Chen, Sanjeev Arora, and Chi Jin. Goedel-Prover: A frontier model for open-source automated theorem proving.arXiv preprint arXiv:2502.07640, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Goedel-Prover-V2: Scaling formal theorem proving with scaffolded data synthesis and self-correction.arXiv preprint arXiv:2508.03613, 2025

    Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, Jiayun Wu, et al. Goedel-Prover-V2: Scaling formal theorem proving with scaffolded data synthesis and self-correction.arXiv preprint arXiv:2508.03613, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Chain of hindsight aligns language mod- els with feedback

    Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language mod- els with feedback. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2302.02676

    Pith/arXiv arXiv 2024

  28. [28]

    ReFT: Reasoning with reinforced fine-tuning

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2401.08967

    Pith/arXiv arXiv 2024

  29. [29]

    Evolving diverse red-team language models in multi-round multi-agent games.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    Chengdong Ma, Ziran Yang, Hai Ci, Jun Gao, Minquan Gao, Xuehai Pan, and Yaodong Yang. Evolving diverse red-team language models in multi-round multi-agent games.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. doi: 10.1109/TPAMI.2026. 3694367

    doi:10.1109/tpami.2026 2026

  30. [30]

    Teaching small language models to reason

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781,

  31. [31]

    Human-level play in the game of Diplomacy by combining language models with strategic reasoning.Science, 378(6624): 1067–1074, 2022

    Meta Fundamental AI Research Diplomacy Team (FAIR). Human-level play in the game of Diplomacy by combining language models with strategic reasoning.Science, 378(6624): 1067–1074, 2022. 12

    2022

  32. [32]

    DeepStack: Expert-level artificial intelligence in heads-up no-limit poker.Science, 356(6337):508–513, 2017

    Matej Moravˇcík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. DeepStack: Expert-level artificial intelligence in heads-up no-limit poker.Science, 356(6337):508–513, 2017

    2017

  33. [33]

    OpenAI o1 system card, 2024

    OpenAI. OpenAI o1 system card, 2024. URLhttps://arxiv.org/abs/2412.16720

    Pith/arXiv arXiv 2024

  34. [34]

    Generative language modeling for automated theorem proving

    Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020

    Pith/arXiv arXiv 2009

  35. [35]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Sticking the landing: Simple, lower- variance gradient estimators for variational inference

    Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lower- variance gradient estimators for variational inference. InAdvances in Neural Information Processing Systems, 2017

    2017

  37. [37]

    Reasoning to learn from latent thoughts.arXiv preprint arXiv:2503.18866, 2025

    Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts.arXiv preprint arXiv:2503.18866, 2025

    arXiv 2025

  38. [38]

    Rewarding progress: Scaling automated process verifiers for LLM reasoning.arXiv preprint arXiv:2410.08146, 2024

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning.arXiv preprint arXiv:2410.08146, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y .K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Odysseus: Scaling VLMs to 100+ turn decision-making in games via reinforcement learning

    Chengshuai Shi, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten, Ziran Yang, Zihan Ding, Gabriel Sarch, Danqi Chen, Karthik Narasimhan, and Chi Jin. Odysseus: Scaling VLMs to 100+ turn decision-making in games via reinforcement learning. arXiv preprint arXiv:2605.00347, 2026

    Pith/arXiv arXiv 2026

  41. [41]

    A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.Science, 362(6419):1140–1144, 2018

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.Science, 362(6419):1140–1144, 2018

    2018

  42. [42]

    Beyond human data: Scaling self-training for problem-solving with language models.Transactions on Machine Learning Research, 2024

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models.Transactions on Machine Learning Research, 2024

    2024

  43. [43]

    Scaling LLM test-time compute op- timally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute op- timally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Bayes’ bluff: Opponent modelling in poker

    Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes’ bluff: Opponent modelling in poker. InProceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pages 550–558, 2005

    2005

  45. [45]

    Solving large imperfect information games using CFR+.arXiv preprint arXiv:1407.5042, 2014

    Oskari Tammelin. Solving large imperfect information games using CFR+.arXiv preprint arXiv:1407.5042, 2014

    Pith/arXiv arXiv 2014

  46. [46]

    Logic-geometric programming: An optimization-based approach to combined task and motion planning

    Marc Toussaint. Logic-geometric programming: An optimization-based approach to combined task and motion planning. InProceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), pages 1930–1936, 2015

    1930

  47. [47]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Trinh, Yuhuai Wu, Quoc V

    Trieu H. Trinh, Yuhuai Wu, Quoc V . Le, He He, and Thang Luong. Solving olympiad ge- ometry without human demonstrations.Nature, 625(7995):476–482, 2024. doi: 10.1038/ s41586-023-06747-5. 13

    2024

  49. [49]

    Doubly reparameterized gradient estimators for Monte Carlo objectives

    George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J Maddison. Doubly reparameterized gradient estimators for Monte Carlo objectives. InInternational Conference on Learning Representations, 2019

    2019

  50. [50]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.04388

    Pith/arXiv arXiv 2023

  51. [51]

    Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022

    Pith/arXiv arXiv 2022

  52. [52]

    Offline reinforcement learning for LLM multi-step reasoning.arXiv preprint arXiv:2412.16145, 2024

    Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for LLM multi-step reasoning.arXiv preprint arXiv:2412.16145, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 9426–9439, 2024. arXiv:2312.08935

    Pith/arXiv arXiv 2024

  54. [54]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  56. [56]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023. arXiv:2210.03629

    Pith/arXiv arXiv 2023

  57. [57]

    Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    STaR: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. STaR: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  59. [59]

    Quiet-STaR: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629, 2024

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-STaR: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    ReST-MCTS*: LLM self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST-MCTS*: LLM self-training via process reward guided tree search. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.03816

    Pith/arXiv arXiv 2024

  61. [61]

    Veeravalli, Aarti Gupta, and Sanjeev Arora

    Haoyu Zhao, Ziran Yang, Jiawei Li, Deyuan He, Zenan Li, Chi Jin, Venugopal V . Veeravalli, Aarti Gupta, and Sanjeev Arora. AlgoVeri: An aligned benchmark for verified code generation on classical algorithms.arXiv preprint arXiv:2602.09464, 2026

    Pith/arXiv arXiv 2026

  62. [62]

    approximate IW AE M-step

    Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret mini- mization in games with incomplete information. InAdvances in Neural Information Processing Systems, volume 20, 2007. 14 A Formulation: Derivations This appendix gives the derivations summarised in §3.2: (i) the IW AE bound, per-sample importance- weight factorisation, ...

    arXiv 2007

  63. [2025]

    URLhttps://arxiv.org/abs/2510.06288

    Pith/arXiv arXiv