Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PAG claims that a single LLM trained by multi-turn RL to act alternately as solver and self-verifier, revising only when it flags an error, outperforms always-revise self-correction baselines and makes verifier-based best-of-N selection…

desk verdict PAG is a real method paper with a plausible selective-revision mechanism, but its strongest claim is contradicted by its own Table 2 and the evidence base lacks uncertainty quantification. read the letter →

arxiv 2506.10406 v1 pith:QIJHMMIU submitted 2025-06-12 cs.CL cs.AIcs.LG

Yuhua Jiang , Yuwen Xiong , Yufeng Yuan , Chao Xin , Wenyuan Xu , Yu Yue , Qianchuan Zhao , Lin Yan This is my paper
classification cs.CLcs.AIcs.LG
keywords self-correctionlargelanguagemodelsmulti-turnreinforcementlearninggenerativeverifierselectiverevisionmathematicalreasoningself-verificationbest-of-N
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single instruction-tuned LLM can learn to verify and correct its own mathematical reasoning in one multi-turn reinforcement-learning pass, without a separate verifier model or a two-stage training pipeline. Its central move is to gate revision on self-verification: the model produces a second attempt only when its own generative check explicitly says the first attempt is wrong. If the claim is right, self-correction no longer requires expensive multi-stage curricula, and a model's self-verification probability becomes a usable signal for choosing among its own sampled answers. The paper reports the highest final-answer accuracy among its compared baselines on MATH500, MinervaMath, AIME2024, and AIME2025, and reports that its verifier-based best-of-N selection beats majority voting.

What carries the argument

The load-bearing mechanism is the selective revision loop embedded in a multi-turn RL objective over alternating policy and verifier turns: the policy produces attempt y_i, the verifier checks it, and a revision is generated only when the check concludes the attempt is wrong. Training rewards both correct attempts and accurate verifications, with a turn-independent optimization that stops later-turn advantages from contaminating earlier turns, plus RoleAdvNorm, which normalizes advantages separately for the two roles. This design prevents the verifier from learning to always trigger or always suppress revision, and it is what lets the same model function as both solver and judge.

What would settle it

Run both self-verify best-of-N and majority voting at N=2, 4, 8, 16, and 32 across many datasets and random seeds; if majority voting wins on average, or the self-verify advantage disappears once variance is accounted for, the best-of-N claim is false. A direct calibration check would bin the model's outputs by its reported 'correct' probability and test whether higher probabilities are monotonically associated with higher true accuracy; if not, the ranking assumption behind verifier-based selection fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the failure mode of multi-turn RL self-correction, known as model collapse, is better addressed by role alternation and selective revision than by the two-stage training used in prior work. The model is trained to alternate between a policy role that writes solutions and a generative verifier role that checks the previous solution, and it is rewarded only when it revises after a genuine error signal and only when its verification judgment is accurate. Under this verify-then-revise regime, the paper reports that PAG achieves the best Acc@final among all baselines across the evaluated math benchmarks with Qwen2.5-7B-Instruct, improves verifier accuracy by large margins, and makes self-verification-based best-of-N selection outperform majority voting, a result prior work had suggested was not achievable.

Load-bearing premise

The headline that self-verification beats majority voting assumes that the probability the model assigns to 'the answer is correct' reliably ranks its own candidate answers, and the reported evidence is a small average advantage on just two datasets with no significance testing.

Editorial extensions

If this is right

  • Self-correction can be trained directly from an instruction-tuned model in one multi-turn RL stage, without SFT warm-up or a two-stage curriculum.
  • Selective revision keeps the answer-change ratio high during training and prevents model collapse, while always-revise training degenerates toward non-correction.
  • PAG-trained verifiers reach 81.7% and 90.7% verifier accuracy on self-generated MATH500 responses for the 1.5B and 7B models, and 78.5 and 86.6 on RewardBench mathprm.
  • Self-verify best-of-N outperforms majority voting by roughly 1% on MATH500 and 1.5% on AIME2024, reversing the earlier finding that self-verification cannot beat self-consistency.
  • Sequential self-correction with K=8 samples beats parallel sampling with K=32 samples under majority voting, a claimed 4x compute-efficiency gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether the verifier's probability ranking would remain a reliable selection rule outside math, for example in code generation, agent trajectories, or open-ended reasoning, where the same best-of-N selection logic could be applied.
  • A natural extension is to use the trained verifier as a cheap early-exit signal: because the model already decides whether to spend revision tokens, the same check could decide when to stop additional sampling or when to escalate to a more expensive solver.
  • The appendix's standalone-verifier results suggest that the verification skill is learned mainly from on-policy data, which implies the method could be adapted to domains that can generate their own correction pairs when no ground-truth reward is available.
  • The framework implies a test-time compute budget rule: spend revision tokens only when the model's own check says the attempt is wrong, which could be compared against uncertainty-threshold or learned early-exit policies on the same check outputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Policy as Generative Verifier (PAG), a multi-turn reinforcement learning framework in which a single instruction-tuned LLM alternates between a policy role (generating solutions) and a generative verifier role (checking its own previous solution). Revision is selective: a second attempt is generated only when the self-verification step identifies the first attempt as wrong. Training uses PPO extended to multi-turn trajectories with turn-independent optimization, a policy reward-shaping bonus for correcting previous mistakes, and role-specific advantage normalization (RoleAdvNorm). The authors claim that PAG improves both direct generation and self-correction accuracy across Qwen2.5-1.5B/7B-Instruct and Llama3-8B-Instruct on MATH500, MinervaMath, AIME2024, and AIME2025, and that its self-verification best-of-N selection outperforms majority voting. The paper also reports strong verifier accuracy on RewardBench math-prm and ablations showing the importance of selective revision, turn-independent optimization, and per-turn rewards.

Significance. If the reported results hold, PAG is a valuable contribution: it offers a conceptually simple, single-model alternative to multi-stage self-correction pipelines such as SCoRe, directly addressing the model-collapse phenomenon through a gated revision mechanism. The training curves and ablation studies are internally coherent and point to a real effect of the selective-revision design. The evaluation is anchored to external ground-truth labels (MATH answers, RewardBench labels), so there is no evident circularity between training and evaluation. The main significance is moderated, however, by the lack of released code and data, the absence of error bars or significance tests, and the fact that one of the central claims is contradicted by the paper's own Table 2. The self-verification BoN claim, in particular, rests on a small and unquantified margin over majority voting, with no calibration evidence for the verifier probabilities used in selection.

major comments (3)
  1. [Section 4.1, Table 2] The claim that 'PAG achieves the highest self-correction performance (Acc.@final) on Qwen2.5-7B-Instruct across all evaluated datasets, consistently outperforming all baselines' is not supported by Table 2 as printed. On MinervaMath, SCoRe's Acc.@final is 37.7% while PAG's is 37.2%; only the average over four datasets favors PAG. Furthermore, the margins on AIME24 (18.4 vs. 16.6) and AIME25 (15.1 vs. 14.0) correspond to one or two items on these very small test sets, and no error bars, seeds, or significance tests are reported. The passage should be revised to state the comparison accurately (for example, 'best average' or 'best on three of four datasets') and the table should include uncertainty estimates or per-seed results.
  2. [Section 4.2, Figure 5] The claim that 'PAG self-verify BoN consistently outperforms majority voting' is load-bearing for the verifier contribution, but the evidence is thin. The gain is about 1 point on MATH500 and 1.5 points on AIME2024, with no error bars, significance tests, or multiple seeds. The method selects the response with the highest probability of the 'correct' token, which requires the verifier's output probabilities to be calibrated across candidate solutions; no calibration analysis is provided. Please report per-N values with bootstrap confidence intervals or standard errors across seeds, and include a calibration measure (e.g., a reliability diagram or expected calibration error) for the 'correct' token probabilities. Without this, the headline comparison to self-consistency is not established.
  3. [Section 4.1, Baselines and Reproducibility] SCoRe, one of the principal baselines, is a re-implementation because the official implementation is not publicly available, and no code, data, or trained checkpoints are released with this paper. The headline comparison depends on the fidelity of this re-implementation. Please either release code and checkpoints, or provide a detailed description of the SCoRe implementation (including the warm-up stage, reward schedule, and any deviations from the original paper) so the comparison can be verified. At minimum, the paper should state what was done to validate the re-implementation against the numbers reported in the SCoRe paper.
minor comments (5)
  1. [Section 3.1, Eq. (1)] The verifier reward is defined as R_v(v_j, y_{j-1}, x) but appears in Eq. (1) as R_v(v_{j+1}, y_j, x). Please fix the indexing or the definition so the objective is unambiguous.
  2. [Tables 1 and 2] The term 'Avg@32' is used in the table captions but is never defined. Please specify whether it is pass@32, majority voting over 32 samples, or an average over 32 generated responses, and state how Acc.@t1 and Acc.@final are computed under this protocol.
  3. [Figure 5] The x-axis of Figure 5 ranges from 21 to 25, which is an unusual range for 'Number of Parallel Samples.' Please explain why this range was selected and consider showing the full range from 1 to 32, or add a caption note about the chosen values.
  4. [Appendix B] The description says evaluation checkpoints are selected by 'highest validation performance' but does not describe how the validation set is constructed or whether validation is disjoint from the test sets. Please specify the validation protocol to rule out selection bias in the reported numbers.
  5. [Throughout] There are several typos and inconsistencies in notation: 'verifer' in Section 1, 'MinevaMATH' vs. 'MinervaMath' in Section 4, 'Sequential' in the Figure 8 caption, 'repsponses' in the Table 3 caption, 'Self-Verifer' in Figure 5, and 'the the' in Section 2. Please proofread and standardize the spelling of 'MinervaMath' and 'verifier.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PAG's claims are empirical, trained on external ground-truth labels and evaluated on held-out benchmarks.

full rationale

PAG's contributions are empirical rather than derived predictions, and the load-bearing training signal is external. Equation 1 rewards attempts and verifications with binary labels R_y and R_v from an "external ground-truth verifier" (Section 3.1), and the Conclusion explicitly concedes "reliance on external ground-truth verifiers to provide reward signals during training"; thus the model is fit to externally labeled correctness, not to the benchmarks whose results are later reported. Evaluations are against held-out datasets (MATH500, AIME2024/2025, MinervaMath, RewardBench) and compared with re-implemented or published baselines. The self-verify best-of-N result is an empirical comparison with majority voting, not a quantity derived from the training objective, and the paper does not rename a fitted parameter as a prediction. Overlapping-author citations (DAPO17K data, HybridFlow infrastructure, VAPO/Seed-Thinking in related/future work) are used as resources or context, not as the justification for the claimed gains, so they do not constitute load-bearing self-citation. The discrepancy between Section 4.1's "across all evaluated datasets" wording and Table 2's MinervaMath value, like the absence of error bars on Figure 5, is a statistical-support or correctness concern, not a circularity concern. No step in the paper's argument reduces, by construction or by citation chain, to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The ledger is small: no fitted constants or invented entities. The method's load-bearing assumptions are the availability of external ground-truth rewards and the reliability of the model's self-verification as a gate.

free parameters (2)
  • Reward shaping scaling factor alpha = 1
    Hand-picked in Section 3.2 and Appendix B; tested at 5 and 10 with no gains, so no fitting occurred, but the reported results use alpha=1.
  • Maximum turns Tmax = 2
    Set due to limited compute in Section 3.1; all main experiments use 2 training turns and 2 evaluation turns.
assumptions (3)
  • domain assumption Ground-truth binary rewards for attempt correctness and verification accuracy are available during training
    Equation (1) defines rewards R_y and R_v using an external ground-truth verifier; the method cannot train without this supervision.
  • domain assumption Instruction-tuned LLMs can be fine-tuned directly with multi-turn PPO without a warm-up or SFT phase
    Claimed in Section 3.2 and the introduction; it is a premise of the method's simplicity and is not proven by theory.
  • domain assumption Self-generated verification verdicts can be parsed and treated as a binary gate for revision
    The selective revision mechanism in Section 3.1 depends on reliably extracting the verdict token.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier." pith.science (2026). https://pith.science/paper/QIJHMMIU

@misc{pith2026250610406,
  author       = {Pith},
  title        = {Pith review of: PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QIJHMMIU}},
  note         = {Machine review of arXiv:2506.10406}
}
read the original abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks, yet they still struggle to reliably verify the correctness of their own outputs. Existing solutions to this verification challenge often depend on separate verifier models or require multi-stage self-correction training pipelines, which limit scalability. In this paper, we propose Policy as Generative Verifier (PAG), a simple and effective framework that empowers LLMs to self-correct by alternating between policy and verifier roles within a unified multi-turn reinforcement learning (RL) paradigm. Distinct from prior approaches that always generate a second attempt regardless of model confidence, PAG introduces a selective revision mechanism: the model revises its answer only when its own generative verification step detects an error. This verify-then-revise workflow not only alleviates model collapse but also jointly enhances both reasoning and verification abilities. Extensive experiments across diverse reasoning benchmarks highlight PAG's dual advancements: as a policy, it enhances direct generation and self-correction accuracy; as a verifier, its self-verification outperforms self-consistency.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Verifier-guided multi-turn RL plus latent-state distillation raises DriveLMM-o1 MCQ to 76.54% for a tool-free teacher and yields a 28-token student at 416 ms (88% faster).

  2. CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.

  3. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Using Pass@k as an RLVR reward, with bootstrap sampling and an analytical advantage formula, improves exploration and later Pass@1 performance of reasoning LLMs.

Reference graph

Works this paper leans on

69 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023

    Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023

    work page 2023

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    What matters for on-policy deep actor-critic methods? a large-scale study

    Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. InInternational Conference on Learning Representations,

    work page

  4. [4]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023

    arXiv 2023

  5. [5]

    Rm-r1: Reward modeling as reasoning, 2025

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. Rm-r1: Reward modeling as reasoning, 2025

    work page 2025

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 69 references
  1. [9]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2021

  2. [10]

    Ash, and Akshay Krishnamurthy

    Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps:...

    2025

  3. [11]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    2023 arXiv

  4. [12]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  5. [13]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    2024 arXiv

  6. [14]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  7. [15]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35...

    2022

  8. [16]

    Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023. 11

    2023 arXiv

  9. [17]

    Competition-level code generation with alphacode.Science, 378 (6624):1092–1097, 2022

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode.Science, 378 (6624):1092–1097, 2022

    2022

  10. [18]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe TwelfthInternational Conference on Learning Representations, 2023

    2023

  11. [19]

    Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  12. [20]

    Inference-time scaling for generalist reward modeling, 2025

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, 2025

    2025

  13. [21]

    Llm4sr: A survey on large language models for scientific research.arXiv preprint arXiv:2501.04306, 2025

    Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. Llm4sr: A survey on large language models for scientific research.arXiv preprint arXiv:2501.04306, 2025

    2025 arXiv

  14. [22]

    S2r: Teaching llms to self-verify and self-correct via reinforcement learning.arXiv preprint arXiv:2502.12853, 2025

    Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S2r: Teaching llms to self-verify and self-correct via reinforcement learning.arXiv preprint arXiv:2502.12853, 2025

    2025 arXiv

  15. [23]

    Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651, 2023

    2023 arXiv

  16. [24]

    Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    2024 arXiv

  17. [25]

    O1 by openai.https://openai.com/o1/, 2024

    OpenAI. O1 by openai.https://openai.com/o1/, 2024

    2024

  18. [26]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  19. [27]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  20. [28]

    Recursive introspection: Teaching language model agents how to self-improve.arXiv preprint arXiv:2407.18219, 2024

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve.arXiv preprint arXiv:2407.18219, 2024

    2024 arXiv

  21. [29]

    Stable- baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable- baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

  22. [30]

    Self- critiquing models for assisting human evaluators.arXiv preprint arXiv:2206.05802, 2022

    William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self- critiquing models for assisting human evaluators.arXiv preprint arXiv:2206.05802, 2022

    2022 arXiv

  23. [31]

    URLhttp://jmlr.org/papers/v22/20-1364.html

  24. [32]

    Seed-thinking-v1

    ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-v1. 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    2025

  25. [33]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  26. [34]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  27. [35]

    Multi-turn reinforcement learning from preference human feedback.arXiv preprint arXiv:2405.14655, 2024

    Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. Multi-turn reinforcement learning from preference human feedback.arXiv preprint arXiv:2405.14655, 2024

    2024 arXiv

  28. [36]

    Heimdall: test-time scaling on the generative verification, 2025

    Wenlei Shi and Xing Jin. Heimdall: test-time scaling on the generative verification, 2025. 12

    2025

  29. [37]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  30. [38]

    A systematic review of large language model (llm) evaluations in clinical medicine.BMC Medical Informatics and Decision Making, 25(1):117, 2025

    Sina Shool, Sara Adimi, Reza Saboori Amleshi, Ehsan Bitaraf, Reza Golpira, and Mahmood Tara. A systematic review of large language model (llm) evaluations in clinical medicine.BMC Medical Informatics and Decision Making, 25(1):117, 2025

    2025

  31. [39]

    Reflexion: an autonomous agent with dynamic memory and self-reflection

    Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023

    2023 arXiv

  32. [40]

    Mind the gap: Examining the self-improvement capabilities of large language models.arXiv preprint arXiv:2412.02674, 2024

    Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models.arXiv preprint arXiv:2412.02674, 2024

    2024 arXiv

  33. [41]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=...

    2025

  34. [42]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  35. [43]

    Rethinking reward modeling in preference-based large language model alignment

    Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference-based large language model alignment. InThe Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=rfdblE10qm

    2025

  36. [44]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  37. [45]

    Llms cannot find reasoning errors, but can correct them given the error location

    Gladys Tyen, Hassan Mansoor, Victor Cărbune, Yuanzhu Peter Chen, and Tony Mak. Llms cannot find reasoning errors, but can correct them given the error location. InFindings of the Association for Computational Linguistics ACL 2024, pages 13894–13908, 2024

    2024

  38. [46]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge.arXiv preprint arXiv:2407.19594, 2024

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge.arXiv preprint arXiv:2407.19594, 2024

    2024 arXiv

  39. [47]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025

    2025 arXiv

  40. [48]

    Self-rewarding correction for mathematical reasoning

    Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.19613, 2025

    2025 arXiv

  41. [49]

    Building math agents with multi-turn iterative preference learning

    Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392, 2024

    2024 arXiv

  42. [50]

    A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization.arXiv preprint arXiv:2504.04950, 2025

    Wenyuan Xu, Xiaochen Zuo, Chao Xin, Yu Yue, Lin Yan, and Yonghui Wu. A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization.arXiv preprint arXiv:2504.04950, 2025

    2025 arXiv

  43. [51]

    Is DPO superior to PPO for LLM alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. InForty-firstInternational Conference on Machine Learning, 2024. URLhttps://openreview.net/forum?id=6XH8R7YrSk

    2024

  44. [52]

    Selfee: Iterative self-revising llm empowered by self-feedback generation.Blog post, 2023

    Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising llm empowered by self-feedback generation.Blog post, 2023

    2023

  45. [53]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  46. [54]

    Self-rewarding language models, 2025

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2025. URLhttps://arxiv.org/abs/2401.10020. 13

    2025 arXiv

  47. [55]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  48. [56]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, X...

    2025 arXiv

  49. [57]

    What’s behind ppo’s collapse in long-cot? value optimization holds the secret.arXiv preprint arXiv:2503.01491, 2025

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value optimization holds the secret.arXiv preprint arXiv:2503.01491, 2025

    2025 arXiv

  50. [58]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=Ccwp4tFEtE

    2025

  51. [59]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  52. [60]

    Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446, 2024

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446, 2024

    2024 arXiv

  53. [61]

    The lessons of developing process reward models in mathematical reasoning, 2025

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025. URL https://arxiv.org/abs/2501.07301

    2025 arXiv

  54. [63]

    PAG 15 eval w/o verifier,

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXiv preprint arXiv:2503.15478, 2025. 14 Appendix A Additional Experiments A.1 Comparison with Dire...

    2025 arXiv

  55. [65]

    Simplify129 mod 11: 129÷11=11remainder8⇒129≡8 mod 11 So,129 34 ≡8 34 mod 11

  56. [66]

    Now we need to find834 mod 11and8 38 mod 11

    Simplify96 mod 11: 96÷11=8remainder8⇒96≡8 mod 11 So,96 38 ≡8 38 mod 11. Now we need to find834 mod 11and8 38 mod 11. Using Fermat’s Little Theorem which states thatap−1 ≡ 1 modp for a primep and a not divisible byp, we know thatap−1 ≡ 1 modp. Herep=11, soa 10 ≡1 mod 11for anya...

  57. [67]

    Now, let’s denote the probability that Alice wins the game asP

    If Bob gets tails, the game essentially resets with Alice’s turn again. Now, let’s denote the probability that Alice wins the game asP. If Alice doesn’t win on her first turn (which happens with probability1 2), then it is Bob’s turn. The probability that Bob doesn’t win on hi...

  58. [68]

    If Bob doesn’t win, the game returns to the starting point, and the probability that Alice wins from this point onward is stillP. Thus, we can set up the following equation for the probability that Alice wins the game: P= 1 2 ⋅0+ 1 2 ⋅ ( 1 2 ⋅0+ 1 2 ⋅P ) Simplifying the equati...

  59. [69]

    The probability that Bob doesn’t win on his turn is1 2, and if Bob doesn’t win, the game returns to Alice’s turn

    If she doesn’t win on her first turn, it becomes Bob’s turn. The probability that Bob doesn’t win on his turn is1 2, and if Bob doesn’t win, the game returns to Alice’s turn. The probability that Alice then wins is the same as the probability that she wins the game starting fr...

  60. [71]

    Therefore, the number of two-digit primes whose digits sum to 8 is 3

    Counting these, we have 3 such numbers. Therefore, the number of two-digit primes whose digits sum to 8 is 3. The answer is correct. MATH EXAMPLE 3 System:Please reason step by step, and put your final answer within. User:What is the remainder when129 34+96 38 is divided by11?...

  61. [2021]

    URLhttps://openreview.net/forum?id=nIAxjsniDzg

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.