Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training language models on a five-turn loop where wrong answers get only "try again" can raise multi-turn reasoning accuracy by up to 14 percent while preserving single-turn performance.

desk verdict Useful recipe for multi-turn RL with unary feedback, but the headline 14% compares Pass@5 baseline to Succ@5 model and needs same-protocol controls before it should be quoted. read the letter →

arxiv 2507.14295 v2 pith:QWUJIRD2 submitted 2025-07-18 cs.LG cs.AI

Licheng Liu , Zihan Wang , Linjie Li , Chenwei Xu , Yiping Lu , Han Liu , Avirup Sil , Manling Li This is my paper
classification cs.LGcs.AI
keywords multi-turnreinforcementlearningunaryfeedbacklargereasoningmodelsself-correctionrewardshapingPPOMATHanswerrepetition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that minimal feedback is enough to make large reasoning models revise their answers across turns. The authors show that standard single-turn RL training makes models repeat the same wrong answer in nearly 70 percent of failed multi-turn interactions, because RL sharpens the output distribution. They introduce Unary Feedback as Observation (UFO), which turns static math datasets into multi-turn episodes where each wrong attempt receives only a generic "try again" signal, and trains with PPO. They report that UFO preserves single-turn accuracy and improves multi-turn success by up to 14 percent, with gains that transfer to out-of-domain benchmarks. If true, this means expensive turn-level human feedback is not needed to teach self-correction.

What carries the argument

The central object is the UFO trajectory, a finite-horizon MDP in which every incorrect answer is followed by a fixed unary feedback token ("Try Again") and correct answers end the episode, so the model never sees positive confirmation. This construction lets static single-turn datasets be replayed as multi-turn interaction histories, and PPO trains the policy to condition on that history. The supporting mechanisms are the collision-probability bound, which explains why peaked single-turn policies repeat answers, and the trajectory-level reward with exponential turn-wise decay plus a repetition penalty, which together reward early success and answer diversity.

What would settle it

Compare two multi-turn trainings that are identical except that one uses unary feedback after wrong answers and the other uses empty or neutral feedback, with the same five-turn budget and compute per problem; a replication where the gap disappears would show the unary signal is not what drives the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that a model trained with multi-turn RL using only unary feedback, a generic negative signal such as "Let's try again" after each incorrect answer, can learn to revise its reasoning and outperform a single-turn RL model on interactive problem solving. The authors frame this as an MDP where the state is the question plus the history of attempts and feedback, and they observe that single-turn RL collapses a model's ability to revise: after training, models give exactly the same answer across turns in about 70 percent of failed cases. Under the proposed UFO training recipe, Succ@5 on MATH-based problems rises by up to 14 percent over the single-turn PPO baseline, and the model generalizes to new domains and even to single-turn evaluation at k=1. They also show that an exponential reward decay and an answer-repetition penalty encourage models to solve problems in fewer turns while generating more diverse attempts.

Load-bearing premise

The result stands on the premise that the single-turn baseline and the UFO model are scored with the same success rule, so the 14 percent improvement is attributable to unary-feedback revision and not to giving the multi-turn model a different, more favorable evaluation.

Editorial extensions

If this is right

  • Multi-turn RL with UFO improves 5-turn success by up to 14 percentage points over single-turn RL with comparable inference cost on the training-domain benchmark.
  • Multi-turn-trained models keep or improve single-turn performance, including at k=1 evaluation, so unary-feedback training does not sacrifice basic reasoning.
  • The revision behavior transfers across math, STEM, QA, and general-knowledge benchmarks and across two model families ranging from 1B to 7B parameters.
  • Exponential reward decay cuts average actions by roughly 10 percent without lowering success, and the repetition penalty raises the share of effective answers from about 80 percent to over 90 percent.
  • UFO can be layered onto existing single-turn RL pipelines and static datasets, so no new data collection or execution environment is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same unary-feedback loop is applied to code generation or tool use, where the environment provides a binary pass-or-fail signal, revision behavior may emerge without hand-written hints.
  • The collision-probability argument suggests that any RL method that sharpens the answer distribution will hurt revision, so future reasoning pipelines may need explicit history conditioning or entropy control to stay adaptive.
  • The paper's stated limitation is the 1B-7B model range; a frontier-scale replication would show whether the benefit grows with base capability or is absorbed by stronger single-turn models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies whether large reasoning models can be trained to improve multi-turn problem solving using only minimal unary feedback (e.g., "try again") after incorrect answers. It proposes UFO, which converts static single-turn math datasets into multi-turn MDP rollouts by appending a generic failure signal to the interaction history, and trains with PPO plus a turn-dependent reward decay and a repetition penalty. The authors report that single-turn RL causes repetitive answering behavior, that UFO-trained models achieve up to 14% higher 5-turn success than single-turn PPO baselines, and that the gains transfer across several datasets and model families. A theoretical section argues that low-entropy policies have high collision probability and that sequential policies dominate parallel policies.

Significance. If the headline comparison is valid, the contribution is useful and timely: it shows that extremely weak feedback, available in ordinary user interactions, can serve as a training signal for multi-turn revision, and the method is simple enough to drop into existing RLVR pipelines. The paper is also commendably concrete: code and models are released, experiments span five model sizes and nine datasets, and the limitations section is candid about the focus on small models. However, the empirical significance currently hinges on an apples-to-oranges comparison; until the metric inconsistency is resolved, the main quantitative claim should not be taken at face value.

major comments (4)
  1. [§4.1, §4.2.1, Figure 5, Table 1, Eq. (3)] The baseline and the proposed method are evaluated under different metrics. Section 4.1 states that the single-turn baseline generates k independent responses and is evaluated with Pass@k, while the UFO model generates responses sequentially with unary feedback and is evaluated with Succ@k. Section 4.2.1 then reports that comparisons use Succ@5, and Figure 5 is captioned "Pass @ 5". Because Eq. (3) already establishes that a sequential policy can dominate a parallel policy by construction, the reported gain of up to 14% is exactly the direction one would expect even with no training if the baseline is denied feedback and the UFO model is allowed to revise. The paper must report the baseline under the same Succ@5 protocol (with "try again" feedback) and/or report Pass@5 for the UFO model, and Table 1 and Figure 5 must state which metric is used. Without this, the central improvement claim is not supported.
  2. [§3.3, §3.4, Eqs. (7), (9), (11)] The reward definitions are inconsistent. Eq. (7) defines the objective as a sum of per-turn binary rewards r_t, but Section 3.4 defines R_t in Eq. (9) as a decayed reward for a correct answer at turn t, and Eq. (11) defines the trajectory reward as R = R_t − Penalty(τ), which is a single scalar, not a sum over turns. If the intended reward is terminal-only, Eq. (7) is incorrect; if the intended reward is a sum of per-turn decayed rewards, Eq. (11) is incorrect. This matters for training, since the two formulations give different credit to early correct answers, and the paper should state unambiguously which objective was optimized.
  3. [Appendix C] The theoretical argument for sequential policies overclaims. The inclusion Π_par ⊆ Π_seq makes max_seq E[Succ@k] ≥ max_par E[Succ@k] trivially true, but the stronger derivation in Appendix C asserts p'_t ≥ p and p'_t = p/(1 − Σ_{i<t} p_i) without justification. For a policy conditioned on arbitrary interaction histories, the conditional success probability can be lower than the marginal p, and the formula treats sums of probabilities of eliminated answers as if they were mutually exclusive and known. The paper should either prove the stronger claim under explicit assumptions or remove it and state only the inclusion-based inequality.
  4. [§4.2.3, Eq. (10), Figure 11] Figure 11 is presented as evidence that the repetition penalty improves diversity, but the reported quantity is the proportion of effective answers, which is exactly the term 1 − E(τ)/T being penalized in Eq. (10). An upward trend in this metric during training is therefore partly a direct effect of the training objective, not an independent behavioral improvement. To support the diversity claim, the paper should report a diversity measure that is not part of the reward, such as the number of distinct reasoning paths or distinct intermediate steps.
minor comments (4)
  1. [Figure 5 and Table 1] The caption of Figure 5 says "Pass @ 5" while the text in §4.2.1 says Succ@5 is used; Table 1 also says "5-turn success rate" without specifying Pass@5 or Succ@5. These should be made consistent.
  2. [§4.2.1] The claim of "similar inference cost" is undefined: a 5-turn sequential rollout with accumulated history consumes more tokens than 5 parallel single-turn responses, so the comparison should state whether cost is measured in model calls, generated tokens, or wall-clock time.
  3. [Boxes 4–6] There are repeated typos in the rollout transcripts, e.g., "Please try agin" instead of "Please try again"; these should be corrected.
  4. [§2.3, Eq. (2)] The collision-probability bound Coll(q) ≥ exp(−H(q)) is stated without specifying that H is in nats; adding the entropy base would avoid ambiguity.

Circularity Check

2 steps flagged · score 4.0 of 10

Headline 14% gain mixes Succ@5 vs Pass@5 protocols (a structural advantage, not just training), and Figure 11 reports the very diversity quantity the reward penalty optimizes; controlled ablations give the central claim some independent support.

  1. other [Section 4.1 'Baseline and Metrics'; Section 4.2.1; Eq. (3) in Section 2.3]
    "For each problem, the baseline generates k independent responses in parallel and is evaluated using standard Pass@k metric. In contrast, our multi-turn model generates responses sequentially with unary feedback after each attempt, and is evaluated using both Succ@k and AvgTurns."

    The abstract's 'improves multi-turn reasoning accuracy by up to 14%' compares Succ@5 for UFO against Pass@5 for the single-turn baseline. These are not the same task: the baseline is denied feedback, while UFO receives 'Try again' and may revise. The paper's own Eq. (3) proves max_{π∈Π_seq} E[Succ@k] ≥ max_{π∈Π_par} E[Succ@k] because every parallel policy is a special case of a sequential policy. A positive gap is therefore guaranteed by construction, independent of RL training; the 14% cannot be cleanly attributed to UFO. Section 4.2.1 says 'using Succ@5' while Figure 5 is captioned 'Pass @ 5', so the paper never resolves which protocol was used.

  2. fitted input called prediction [Section 3.4 Eq. (10); Section 4.2.3 and Figure 11]
    "We define a normalized penalty term: Penalty(τ) = λ·(1 − E(τ)/T), where λ > 0 is a tunable penalty weight, and E(τ)/T measures answer diversity. ... Considering the normalized penalty term in our experiment (Equation 10), we count the number of non-repetitive answer for each validation round, as shown in Figure 11."

    The answer-repetition penalty in Eq. (10) is a decreasing function of E(τ)/T, the effective-answer ratio. Figure 11 then reports E(τ)/T rising from 80% to over 90% and reads this as evidence that the model 'learned to explore and self-reflect.' But E(τ)/T is exactly the quantity the reward was designed to maximize (higher E(τ)/T lowers the penalty), so the upward trend is the training objective itself, not an independent behavioral prediction. The diversity 'improvement' is built into the reward definition.

full rationale

The central UFO result is anchored to external ground-truth correctness labels and is not simply a renaming of the reward: the paper's T_max ablation (Fig. 6/7) and feedback-vs-no-feedback comparison (Fig. 8) compare multi-turn training under a shared multi-turn evaluation, giving the core claim some independent content. The loading-bearing self-citations are limited: RAGEN is cited for the multi-turn RL implementation, not as a uniqueness or existence theorem, so no self-citation chain forces the result. However, the headline 14% improvement is partly an artifact of scoring UFO with Succ@5 and the baseline with Pass@5, a difference the paper's own Eq. (3) shows is structurally favorable to any sequential policy even before training; the text is internally inconsistent about which metric was used. Separately, the diversity evidence in Figure 11 is the same E(τ)/T term that the repetition penalty optimizes, so it cannot serve as independent confirmation. The paper honestly notes its small-model scope as a limitation, which affects generality but not circularity. On balance, the empirical core has independent support, but two headline claims reduce partly by construction, yielding a moderate circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on standard RL assumptions plus a few hand-chosen hyperparameters. No free parameters are fitted to the evaluation data; gamma, lambda, and eta are training hyperparameters. The main domain assumption is that generic 'try again' feedback is a sufficient learning signal, which is exactly the hypothesis being tested.

free parameters (4)
  • reward decay factor gamma = 0.5
    Exponential decay gamma=0.5 used in Equation 9 for turn-wise reward decay; no sensitivity analysis is provided.
  • repetition penalty weight lambda = not reported
    Lambda in Equation 10 is described as tunable, but no value is given in the paper.
  • malformed-output penalty eta = not reported (negative)
    Small penalty for malformed outputs, value not specified.
  • training rollout counts = P=8, N=16
    Batch size and rollouts per prompt in the main training setup; affects variance and results.
assumptions (4)
  • standard math The collision probability lower bound Coll(q) >= exp(-H(q))
    Derived via Jensen's inequality in Appendix B; standard mathematics.
  • domain assumption Single-turn RL produces peaked, low-entropy output distributions
    Invoked in Section 2.3 and relies on cited prior work (Cui et al., 2025; Yue et al., 2025).
  • domain assumption Unary 'try again' feedback provides sufficient signal for models to learn revision
    Core hypothesis of UFO; it is the very claim being tested and is not independently established.
  • domain assumption Verifiable reward (final answer correctness) is available for all evaluation datasets
    The method relies on automatic binary correctness checks, which hold for the math and QA datasets used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning." pith.science (2026). https://pith.science/paper/QWUJIRD2

@misc{pith2026250714295,
  author       = {Pith},
  title        = {Pith review of: A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QWUJIRD2}},
  note         = {Machine review of arXiv:2507.14295}
}
read the original abstract

Multi-turn problem solving is critical yet challenging for Large Reasoning Models (LRMs) to reflect on their reasoning and revise from feedback. Existing Reinforcement Learning (RL) methods train large reasoning models on a single-turn paradigm with verifiable rewards. However, we observe that models trained with existing RL paradigms often lose their ability to solve problems across multiple turns and struggle to revise answers based on contextual feedback, leading to repetitive responses. We ask: can LRMs learn to reflect their answers in a multi-turn context? In this work, we find that training models with multi-turn RL using only unary feedback (e.g., "Let's try again") after wrong answers can improve both single-turn performance and multi-turn reasoning. We introduce Unary Feedback as Observation (UFO) for reinforcement learning, which uses minimal yet common unary user feedback during iterative problem solving. It can be easily applied to existing single-turn RL training setups. Experimental results show that RL training with UFO keeps single-turn performance and improves multi-turn reasoning accuracy by up to 14%, enabling language models to better react to feedback in multi-turn problem solving. To further minimize the number of turns needed for a correct answer while encouraging diverse reasoning when mistakes occur, we design reward structures that guide models to produce careful and deliberate answers in each turn. Code: https://github.com/lichengliu03/unary-feedback

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

    cs.AI 2026-02 conditional novelty 5.0 of 10

    BAO, a behavior-enhanced SFT plus regularized RL pipeline, improves proactive agents' task performance while lowering user-involvement rate, beating UserRL baselines on three UserRL gym tasks.

Reference graph

Works this paper leans on

57 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    arXiv 2025

  2. [2]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    arXiv 2024

  3. [3]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    arXiv 2024

  4. [4]

    Gemini: A family of highly capable multimodal models, 2025

    Gemini Team. Gemini: A family of highly capable multimodal models, 2025. URL https://arxiv.org/abs/2312.11805

    arXiv 2025

  5. [5]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    arXiv 2017

  6. [6]

    Archer: Training language model agents via hierarchical multi-turn rl, 2024

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024. URL https://arxiv.org/abs/2402.19446

    arXiv 2024

  7. [7]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025. URL https://arxiv.org/ab...

    arXiv 2025

  8. [8]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxiv.org/abs/2404.07972

    arXiv 2024

Show all 57 references
  1. [9]

    Training software engineering agents and verifiers with swe-gym, 2024

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024. URL https://arxiv.org/abs/2412.21139

    2024 arXiv

  2. [10]

    Webshop: Towards scalable real-world web interaction with grounded language agents, 2023 a

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023 a . URL https://arxiv.org/abs/2207.01206

    2023 arXiv

  3. [11]

    Alfworld: Aligning text and embodied environments for interactive learning, 2021

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021. URL https://arxiv.org/abs/2010.03768

    2021 arXiv

  4. [12]

    Mint: Evaluating llms in multi-turn interaction with tools and language feedback, 2024 a

    Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback, 2024 a . URL https://arxiv.org/abs/2309.10691

    2024 arXiv

  5. [13]

    Simworld: A world simulator for scaling photorealistic multi-agent interactions, 2025

    Yan Zhuang, Jiawei Ren, Xiaokang Ye, Xuhong He, Zijun Gao, Ryan Wu, Mrinaal Dogra, Cassie Zhang, Kai Kim, Bertt Wolfinger, Ziqiao Ma, Tianmin Shu, Zhiting Hu, and Lianhui Qin. Simworld: A world simulator for scaling photorealistic multi-agent interactions, 2025

    2025

  6. [14]

    Gonzalez, and Ion Stoica

    Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-v0: Train real-world long-horizon agents v...

    2025

  7. [15]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  8. [16]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  9. [17]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URL https://arxiv.org/abs/2503.09516

    2025 arXiv

  10. [18]

    Retool: Reinforcement learning for strategic tool use in llms, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https://arxiv.org/abs/2504.11536

    2025 arXiv

  11. [19]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  12. [20]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  13. [21]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URL https://arxiv.org/abs/2503.24290

    2025 arXiv

  14. [22]

    Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamathqa: A dataset for mathematical reasoning with large language models, 2024. URL https://arxiv.org/abs/2405.17633

    2024 arXiv

  15. [23]

    Theoremqa: A theorem-driven question answering dataset, 2023 a

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset, 2023 a . URL https://arxiv.org/abs/2305.12524

    2023 arXiv

  16. [24]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    2018 arXiv

  17. [25]

    Reasoning over public and private data in retrieval-based systems, 2022

    Simran Arora, Patrick Lewis, Angela Fan, Jacob Kahn, and Christopher Ré. Reasoning over public and private data in retrieval-based systems, 2022. URL https://arxiv.org/abs/2203.11027

    2022 arXiv

  18. [26]

    Measuring massive multitask language understanding, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2020. URL https://arxiv.org/abs/2009.03300

    2020 arXiv

  19. [27]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024 b

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024 arXiv

  20. [28]

    Graph of thoughts: Solving elaborate problems with large language models, 2023

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models, 2023. URL https://...

    2023 arXiv

  21. [29]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366

    2023 arXiv

  22. [30]

    Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree search self-refinement, 2024

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree search self-refinement, 2024. URL https://arxiv.org/abs/2406.07394

    2024 arXiv

  23. [31]

    Self-refine: Iterative refinement with self-feedback, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterativ...

    2023 arXiv

  24. [32]

    Critic: Large language models can self-correct with tool-augmented critiquing, 2023

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-augmented critiquing, 2023. URL https://arxiv.org/abs/2305.11738

    2023 arXiv

  25. [33]

    Large language models prompting with episodic memory, 2024

    Dai Do, Quan Tran, Svetha Venkatesh, and Hung Le. Large language models prompting with episodic memory, 2024. URL https://arxiv.org/abs/2408.07465

    2024 arXiv

  26. [34]

    Larimar: Large language models with episodic memory control, 2024

    Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jiří, Navrátila, Soham Dan, and Pin-Yu Chen. Larimar: Large language models with episodic memory control, 2024. URL https://arxiv...

    2024 arXiv

  27. [35]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017. URL https://arxiv.org/abs/1706.03741

    2017 arXiv

  28. [36]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback, 2023. URL https://arxiv....

    2023 arXiv

  29. [37]

    Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D

    Zachary Kenton, Noah Y. Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D. Goodman, and Rohin Shah. On scalable oversight with weak llms judging strong llms, 2024. URL https://arxiv.org/abs/2407.04622

    2024 arXiv

  30. [38]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023. URL https://arxiv.org/abs/2305.18290

    2023 arXiv

  31. [39]

    Parameter efficient reinforcement learning from human feedback, 2024

    Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, and Lucas Dixon....

    2024 arXiv

  32. [40]

    Self-play fine-tuning converts weak language models to strong language models, 2024

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL https://arxiv.org/abs/2401.01335

    2024 arXiv

  33. [41]

    Uno arena for evaluating sequential decision-making capability of large language models, 2024

    Zhanyue Qin, Haochuan Wang, Deyuan Liu, Ziyang Song, Cunhang Fan, Zhao Lv, Jinlin Wu, Zhen Lei, Zhiying Tu, Dianhui Chu, Xiaoyan Yu, and Dianbo Sui. Uno arena for evaluating sequential decision-making capability of large language models, 2024. URL https://arxiv.org/abs/2406.16382

    2024 arXiv

  34. [42]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  35. [43]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  36. [44]

    Chain-of-thought prompting elicits reasoning in large language models, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2022. URL https://arxiv.org/abs/2201.11903

    2022 arXiv

  37. [45]

    Self-consistency improves chain-of-thought reasoning in language models, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain-of-thought reasoning in language models, 2022. URL https://arxiv.org/abs/2203.11171

    2022 arXiv

  38. [46]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023 b . URL https://arxiv.org/abs/2305.10601

    2023 arXiv

  39. [47]

    React: Synergizing reasoning and acting in language models, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2022. URL https://arxiv.org/abs/2210.03629

    2022 arXiv

  40. [48]

    Pal: Program-aided language models, 2022

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2022. URL https://arxiv.org/abs/2211.10435

    2022 arXiv

  41. [49]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning, 2023 b . URL https://arxiv.org/abs/2211.12588

    2023 arXiv

  42. [50]

    Toolformer: Language models can teach themselves to use tools, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL https://arxiv.org/abs/2302.04761

    2023 arXiv

  43. [51]

    Let's verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, and Ilya Sutskeverand Karl Cobbe. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  44. [52]

    Autopsv: Automated process-supervised verifier, 2024

    Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, and Zhijiang Guo. Autopsv: Automated process-supervised verifier, 2024. URL https://arxiv.org/abs/2405.16802

    2024 arXiv

  45. [53]

    Xu, Damai Dai, Yifei Li, Deli Chen, Y.Wu, and Zhifang Sui

    Peiyi Wang, Lei Li, Zhihong Shao, R.X. Xu, Damai Dai, Yifei Li, Deli Chen, Y.Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step by step, 2023. URL https://arxiv.org/abs/2312.08935

    2023 arXiv

  46. [54]

    Rewarding progress: Scaling automated process verifiers for llm reasoning, 2024

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning, 2024. URL https://arxiv.org/abs/2410.08146

    2024 arXiv

  47. [55]

    Ursa: Understanding and verifying chain-of-thought reasoning in large language models, 2025

    Ruilin Luo, Zhuofan Zheng, Yifan Wang, Xinzhe Ni, Zicheng Lin, Songtao Jiang, Yiyao Yu, Chufan Shi, Ruihang Chu, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thought reasoning in large language models, 2025. URL https://arxiv.org/abs/2501.04686

    2025

  48. [56]

    Large language models cannot self-correct reasoning yet, 2023

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2023. URL https://arxiv.org/abs/2310.01798

    2023 arXiv

  49. [57]

    High-dimensional continuous control using generalized advantage estimation, 2015

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2015. URL https://arxiv.org/abs/1506.02438

    2015 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.