Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

Process Reward Models for LLM Agents: Practical Framework and Directions

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that small 3B-parameter LLM agents can be trained to surpass strong GPT-4o prompting baselines on the ALFWorld benchmark by learning a per-step process reward model from the agent's own Monte Carlo rollouts and iterating…

desk verdict AgentPRM is a practical, promising recipe, but the InversePRM section has a load-bearing sign error in its printed loss that contradicts the stated objective, so the paper as written does not support its headline claim. read the letter →

arxiv 2502.10325 v1 pith:COBT7Y5P submitted 2025-02-14 cs.LG cs.AI

Sanjiban Choudhury This is my paper
classification cs.LGcs.AI
keywords processrewardmodelLLMagentsreinforcementlearninginverseactor-criticALFWorldhackingtest-timescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that an LLM agent can be improved cheaply and steadily by training a process reward model — a per-step value function that scores each action — from Monte Carlo rollouts of the agent itself, and then optimizing the policy against that model using standard RLHF machinery. On the ALFWorld text-game benchmark, three iterations of this loop lift a 3B-parameter Llama agent to 91.0% success with Best-of-N inference, above the best single-attempt prompted baselines the authors measured, including claude-3.5-sonnet at 76.1% and GPT-4o at 65.7%. A second algorithm, InversePRM, learns the same kind of step-level critic from expert demonstrations alone, without any outcome reward, and reaches 86.6% success in two iterations using only 10k demonstrations — near the 91.0% expert and far above the 63.4% achieved by supervised fine-tuning on the same data. The paper also finds where the approach fails: with only 10k rollouts the process reward keeps climbing on validation while actual success peaks and then falls, a reward-hacking failure that motivates the larger-rollout, reward-shaping, and exploration strategies studied later in the paper. A sympathetic reader takes away that automatically generated step-level supervision can substitute for hand-designed rewards and for large-scale reinforcement learning in long-horizon agent tasks.

What carries the argument

The load-bearing object is the process reward model acting as an approximate Q-function over the turn-level MDP, trained on targets $$\widehat{Q}(s,a)=\frac{1}{|G(s,a)|}\sum_{(s_t,a_t)\in D(s,a)}\sum_{k=t}^{T-1}\$gamma^{{k-t}}$r_k$$ obtained by averaging Monte Carlo returns over all rollouts that pass through a state-action pair. The loop has three stages: roll out $\pi_{i-1}$ and tabulate $\widehat{Q}$; fit $Q_i$ by soft binary cross-entropy on those targets; then update $\pi_i$ via online DPO against $Q_i$ while keeping a KL penalty to $\pi_{i-1}$, a conservative-policy-iteration safeguard that protects the reward estimate from distribution shift. For InversePRM, the central identity is the telescope $r(s,a)=Q^\pi(s,a)-\gamma\mathbb{E}_{a'\sim\pi}Q^\pi(s',a')$, which converts the inverse-RL min-max game into a classification loss over positive expert transitions and negative learner transitions. At test time, Best-of-N sampling with the PRM as ranker is the mechanism that converts the learned critic into additional performance without further training.

What would settle it

Run the released code on ALFWorld and check whether Best-of-N with π3 and Q2 actually scores 91.0% on the 136 out-of-distribution tasks; a more mechanistic check is to measure the rank correlation between PRM scores and true Monte Carlo returns on rollouts of each updated policy, since a correlation that decays toward zero while training reward keeps climbing — the pattern shown at 10k rollouts in Fig. 3 — would refute the distribution-shift assumption that the whole loop rests on.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a process reward model for an LLM agent can be treated as a Q-function over a turn-level MDP, $Q^\pi(s_t,a_t)=\mathbb{E}_\pi\left[\sum_{k=t}^{T-1}\gamma^{k-t}r(s_k,a_k)\right]$, fitted to Monte Carlo targets computed from asynchronous rollouts of the current policy, and that iterating 'rollout, learn $Q$, optimize $\pi$ with a KL penalty to the previous policy' produces a self-improving loop that slots into an existing RLHF pipeline (AgentPRM). The paper reports that three iterations on ALFWorld with a 3B Llama and online DPO yield policies at 73.9%, 85.8%, and 88.1% success, with Best-of-N selection under the PRM reaching 91.0%, above every single-attempt prompting baseline tested, including much larger closed models. Where outcome rewards are absent, the paper claims the same benefit can be derived from demonstrations alone: rewriting the one-step reward through the telescoping identity $r(s,a)=Q^\pi(s,a)-\gamma\,\mathbb{E}_{a'\sim\pi}Q^\pi(s',a')$ turns inverse RL into a discrimination task between expert transitions and learner transitions, and the resulting InversePRM reaches 86.6% success in two iterations, surpassing SFT on the same data (63.4%) and rivaling the expert policy (91.0%) with only 10k demonstrations. The paper also stakes out a boundary of its own method: with 10k rollouts the process reward hacks — validation reward rises while success peaks near 82% and then declines after step 400 — and the authors show that more rollouts, reference-policy reward shaping, and structured exploration each help stabilize this low-rollout regime.

Load-bearing premise

The reward model trained on one policy's rollouts must keep giving trustworthy step scores for the newer, improved policy; this distribution-shift assumption fails in the paper's own 10,000-rollout runs, where the process reward keeps rising while actual success falls.

Editorial extensions

If this is right

  • A 3B open model trained this way can outperform prompted GPT-4o and claude-3.5-sonnet on ALFWorld, suggesting that process-level critics can substitute for model scale and prompt engineering in long-horizon agent tasks.
  • Because only the rollout-and-target stage is new, the framework rides on existing RLHF infrastructure, so the claimed gains come at modest engineering cost over current post-training pipelines.
  • Best-of-N with the PRM gives consistent test-time scaling, with the largest gains early in training (π0: 64.9% to 67.9%, π1: 73.9% to 84.3%) and diminishing returns once the policy saturates.
  • When outcome rewards are unavailable, InversePRM shows that 10k expert demonstrations can nearly match the expert policy (86.6% vs 91.0%), far exceeding SFT on the same data — demonstration density partially replaces reward engineering.
  • Reward hacking is observed and quantified: at 10k rollouts, validation process reward keeps increasing while success peaks at 82% and then falls after step 400, so rollout quantity and reward shaping are load-bearing for stability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's logic: because the PRM is a Q-function, its scores could serve as a reward-hacking early-stop detector — if validation process reward keeps rising while a held-out outcome proxy declines, training should halt — but the paper's own ensemble attempt found no such reliable signal, so a detector remains an open problem.
  • The InversePRM telescoping trick is not specific to ALFWorld; the same 'discriminate expert transitions from learner transitions in Q-space' idea could be dropped into web navigation or interactive coding settings, which the paper lists as future work.
  • The headline comparison is favorable to the paper: unlike Reflexion and AdaPlanner, which get multiple attempts per test task, AgentPRM uses a single attempt with Best-of-N only over token-level responses; whether BoN(π3,Q2) would also beat the 91.7% of multi-attempt AdaPlanner under equal attempts is not resolved by the reported numbers.
  • The reference-advantage shaping term A^µ(s,a) suggests a practical recipe for low-budget setups: keep a moderate reference policy, fit its value, and mix its advantage into the PRM target with α ≈ 0.5 to stabilize training when only 10k rollouts are available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AgentPRM, an iterative actor-critic framework for training LLM agents, in which a process reward model (PRM) is trained on Monte Carlo return targets obtained from rollouts of the current policy, and the policy is then updated against this PRM while regularized by KL to the previous policy. The paper also proposes InversePRM, which aims to learn a PRM from expert demonstrations plus learner rollouts by classifying the difference of Q-values, without explicit outcome rewards. Experiments on ALFWorld with a Llama3.2-3B model report that AgentPRM and InversePRM outperform strong GPT-4o and Claude baselines, and the paper analyzes test-time scaling, reward hacking, exploration, and reward shaping. Code is released.

Significance. If the results hold, the framework is a practical contribution: it provides a simple, scalable way to generate dense reward signals for LLM agents using asynchronous Monte Carlo rollouts, integrates with existing RLHF pipelines, and includes a demonstration-only variant. The paper is also upfront about reward hacking and studies mitigation strategies, which is useful for the community. However, the InversePRM algorithm as written contains a sign error in its loss, the underlying optimization in Eq. (9) is unconstrained, and the empirical evaluation reports no uncertainty quantification, so the central claims currently rest on a specification that appears internally inconsistent and on single-run results.

major comments (4)
  1. [Section 3.2, Eq. (11) and Algorithm 2] The InversePRM loss as printed has a sign error. The loss is L = -E_D+[log sigma(Delta)] + E_D-[log(1 - sigma(Delta))], with Delta = Q(s,a) - gamma Q(s',a'). For a negative example, minimizing +log(1 - sigma(Delta)) drives sigma(Delta) toward 1, i.e., it maximizes Delta, the same direction as the positive term. Thus the loss cannot distinguish expert transitions from learner transitions, contradicting the prose in Section 3.2 and the intended objective in Eq. (9). The correct surrogate should have a minus before the D- term. Because the abstract explicitly credits InversePRM with outperforming GPT-4o baselines, this sign error is load-bearing and must be corrected.
  2. [Section 3.1, Eq. (9)] The optimization in Eq. (9) is an unconstrained maximization over Q of a linear functional, E_D+[Q - gamma Q'] - E_D-[Q - gamma Q']. This objective diverges unless Q is constrained or regularized. The paper does not specify any constraint before introducing the sigmoid surrogate in Eq. (11). The authors should either formulate Eq. (9) as a constrained optimization or state explicitly that Eq. (11) is the bounded surrogate actually being optimized, and then show that the surrogate (with the sign corrected) is consistent with Eq. (9).
  3. [Tables 1 and 2] The empirical evaluation reports success rates from single runs with no error bars, confidence intervals, or multiple seeds. Given that the rollouts, online DPO training, and ALFWorld task generation are stochastic, the headline claim that 3B models 'outperform strong GPT-4o baselines' needs variance quantification. At minimum, report mean and standard deviation over at least three seeds, or provide bootstrap confidence intervals over the 136 test tasks, to establish that the reported margins are not noise.
  4. [Section 2.2, Stage 3] The policy is optimized against a PRM trained on rollouts of the previous policy, and the paper relies on a KL penalty to control distribution shift. Figure 3 shows that with 10k rollouts the process reward keeps increasing while outcome success falls after step 400, indicating that the assumption can fail. This is a known limitation and the paper analyzes it, but the framework would be stronger with a more concrete protocol for detecting or mitigating over-optimization without requiring access to the outcome reward, which the paper leaves as an open question.
minor comments (5)
  1. [Section 2.3 and Tables 1/2] The text states that the evaluation sets contain 139 in-distribution and 134 out-of-distribution tasks, but Tables 1 and 2 report results on 136 out-of-distribution games. Please clarify this discrepancy.
  2. [Section 3.3] The text refers to 'Fig. 2 (a)' and 'Fig. 2 (b)' when discussing InversePRM training and inference curves, but the actual figures are Fig. 5 (a) and Fig. 5 (b).
  3. [Section 2.2] Equations (2) and (4) are identical, as are Eqs. (3) and (5). Defining the loss and policy update once and referencing them later would reduce redundancy.
  4. [Table 1 caption] The caption notes that Reflexion and AdaPlanner receive multiple attempts at test time, but the table still lists them alongside single-attempt baselines. Please make this distinction more prominent in the table itself, for example with a footnote in the header.
  5. [Section 4.1, Listing 1] The Steered Exploration prompt snippet contains formatting and spacing artifacts; please ensure the final version typesets the prompt cleanly.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: AgentPRM is standard policy iteration and InversePRM is a reparameterized IRL objective, with only a non-load-bearing self-citation.

full rationale

The derivation chain is self-contained against external benchmarks. AgentPRM computes Q-targets by Monte Carlo rollouts (Eq. 1), trains the PRM with soft binary cross-entropy (Eq. 4), and updates the policy to maximize Q with KL regularization (Eq. 5); this is standard actor-critic/policy iteration and no equation defines the target quantity in terms of itself. InversePRM uses the exact Bellman identity r(s,a)=Q(s,a)-γE_{a'}Q(s',a') (Eq. 8) to convert an IRL objective (Eq. 7) into a Q-difference classification objective (Eq. 9); the identity is not an assumption equivalent to the result, and the policy is still evaluated by the external ALFWorld environment. The LEAP expert policy [34] and LEAP theory citations [38,39] are self-citations, but they are used as a demonstration source/baseline and in the speculative exploration discussion, not as load-bearing premises of the AgentPRM or InversePRM derivations. One serious flaw found is algorithmic, not circular: the printed InversePRM loss in Eq. (11)/Algorithm 2 has a plus sign before the D- term, so minimizing it would push Q-differences up for both expert and learner transitions and could not learn the intended discriminator; this is a correctness/reproducibility defect, not a reduction of the prediction to its inputs, and therefore does not raise the circularity score beyond the minor self-citation level.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework is a combination of standard RL components (MC rollouts, Q-learning, IRL) applied to LLM agents. The main free quantities are the usual RL hyperparameters (β, γ, α, δ, N) plus the choice of 10k or 70k rollouts. No new physical or mathematical entities are introduced.

free parameters (5)
  • KL regularization coefficient β
    Chosen by hand in Eq. 3, 5, 12; controls the distribution shift of the policy relative to π_{i-1} and is essential to prevent reward hacking.
  • Discount factor γ
    Used in the Q-target computation (Eq. 1) and in the InversePRM loss (Sec. 3.2); set to a standard RL value but not reported explicitly.
  • Best-of-N size N = 16
    Used at inference in Tables 1 and 2; the test-time scaling analysis (Fig. 2b) shows performance depends on N.
  • Preference margin δ
    Hyperparameter in the relative-loss variant of the PRM (Sec. 2.3); determines which action pairs become preferences.
  • Reward shaping weight α = 0.5
    In Eq. 13, α mixes the learned PRM target with the reference-policy advantage; the experiment uses α = 0.5.
assumptions (5)
  • domain assumption The environment is a turn-level MDP with stationary transition dynamics and bounded rewards in [0,1].
    State this at the start of Sec. 2.1; all subsequent Q-function and policy updates rely on it.
  • domain assumption The telescoping identity r(s,a) = Q(s,a) - γ E_{a'}Q(s',a') holds for the Q-function used in the IRL loss.
    Invoked in Eq. 8 in Sec. 3.1 to reparameterize rewards in terms of Q values, but the loss (Eq. 12) applies it to arbitrary Q without a constraint that Q be the true value function.
  • standard math Policy iteration with KL regularization to the previous policy converges to a good policy.
    The paper cites conservative policy iteration [23] in Sec. 2.2 to justify the KL penalty; this assumes standard RL convergence conditions.
  • domain assumption Online DPO can optimize a policy against a learned reward model served by the PRM.
    Stage 3 uses Online DPO as the RL algorithm; this is taken from RLHF practice rather than proven in the paper.
  • domain assumption The expert demonstrations used in InversePRM are representative enough to define a useful reward.
    Sec. 3.3 uses 10k demonstrations from LEAP; the success of InversePRM depends on the quality and coverage of these demos.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Process Reward Models for LLM Agents: Practical Framework and Directions." pith.science (2026). https://pith.science/paper/COBT7Y5P

@misc{pith2026250210325,
  author       = {Pith},
  title        = {Pith review of: Process Reward Models for LLM Agents: Practical Framework and Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/COBT7Y5P}},
  note         = {Machine review of arXiv:2502.10325}
}
read the original abstract

We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.

Figures

Figures reproduced from arXiv: 2502.10325 by the authors.

Figure 1
Figure 1. Overview (a) AgentPRM: Trains an LLM policy π using outcome rewards through three iterative stages. Stage 1: Roll out the current policy πi−1 and compute the PRM target dataset D. Stage 2: Train PRM Qi on D via supervised learning. Stage 3: Update policy πi using RL with PRM Qi. (b) InversePRM: Trains π using expert demonstrations in three stages. Stage 1: Roll out πi−1 to generate positive D + and negative D − tran… view at source ↗
Figure 2
Figure 2. Training and Inference. (a) Success rate vs training steps during online DPO with PRMs for 3 iterations of AgentPRM. π0 is initialized with SFT. PRM Q0 is trained on π0 rollouts. OnlineDPO(π0, Q0) is run for 400 training steps, during which the success rate goes up till it plateaus. The final checkpoint π1 is taken and the process repeated to get π2, π3 till success rate limit is reached. (b) Inference with Best-of-… view at source ↗
Figure 3
Figure 3. shows how both success rates (outcome rewards) and process rewards vary over training steps when the PRM is trained over 10k rollouts. After 400 steps, the success rate begins to fall from 82% to 70%. In contrast, the reward on the validation set keeps increasing. This shows clear signs of reward hacking. An open question remains how to reliably detect over-optimization without evaluating the success rate (which is … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Absolute vs Relative Loss for PRM. Success rate over training steps for PRM trained with 70k roll￾outs. Both losses lead to similar performance. To train PRMs in a relative manner, we use the following procedure: 1. (Stage 1) Collect rollouts and construct a dictionary…
Figure 5
Figure 5. Figure 5: Training and Inference of InversePRM. (a) Success rate (%) vs. training steps for 2 iterations of InversePRM using online DPO with PRMs. The initial policy π0 is initialized identically to AgentPRM. PRM Q0 is trained on π0 rollouts. OnlineDPO(π0, Q0) runs for 400 train…
Figure 6
Figure 6. Figure 6: Different exploration strategies. Success rate vs training steps with OnlineDPO(π0, Q0). Both Reset-50-50 and SteeredExploration learn faster and reach higher performance. Strategy 1: Reset Distribution. A simple yet effective exploration strategy is to reset the agent…
Figure 7
Figure 7. Figure 7: Process Reward Shaping. Success rate vs training steps of OnlineDPO(π0, Q0) when training with shaped rewards vs non-shaped rewards with 10k rollouts. Non-shaped rewards are noisy at low sample regimes with unstable performance. Shaped rewards lead to much more stable …

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    ARCO introduces a co-evolving rubric model with generation and scoring heads plus a trajectory decomposition constraint that improves exact-match scores on multi-hop QA tasks over outcome, rubric, and process reward b...

  2. MASPRM: Multi-Agent System Process Reward Model

    cs.MA 2025-10 conditional novelty 6.0 of 10

    MASPRM trains a per-agent, per-message value head from outcome-only MCTS rollouts and uses it to guide step-level beam search and MCTS, improving exact match on GSM8K by up to +30.7 points over a greedy multi-agent pass.

  3. Reinforcement Learning for Machine Learning Engineering Agents

    cs.LG 2025-09 conditional novelty 6.0 of 10

    RL-trained Qwen2.5-3B outperforms prompted Claude-3.5-Sonnet and GPT-4o on 12 MLEBench tasks by an average of 22% and 24%, using two targeted RL modifications.

  4. Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning

    cs.LG 2025-08 conditional novelty 6.0 of 10

    Decoupling LLM-agent planning from summarization and rewarding tool-call completeness rather than final-answer correctness improves planning by 8-12% and end-to-end answers by 5-6% over end-to-end RL baselines.

  5. Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B LLM trained with sparse completion rewards and a GRPO-style algorithm reaches state-of-the-art on ALFWorld and ScienceWorld.

  6. A Diagnostic Framework for AI Agent Behavior

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A two-layer diagnostic framework for AI agent behavior: distinguishing foundational computational substrate from behavioral modulation layer.

  7. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  8. SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

    cs.CL 2025-05 conditional novelty 4.0 of 10

    SPA-RL attributes a single final task reward to individual agent steps and uses the attributions plus an executability signal as dense rewards for PPO, yielding modest success-rate gains on WebShop, ALFWorld, and VirtualHome.

Reference graph

Works this paper leans on

55 extracted references · 16 canonical work pages · cited by 8 Pith papers

  1. [1]

    Step: Stacked llm policies for web actions

    Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked llm policies for web actions. In Conference on Language Modeling (COLM), 2024

    work page 2024

  2. [2]

    pi0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  3. [3]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    arXiv 2023

  4. [4]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    arXiv 2022

  5. [5]

    Reflexion: Language agents with verbal reinforcement learning.(2023)

    Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.(2023). arXiv preprint cs.AI/2303.11366, 2023

    arXiv 2023

  6. [6]

    Fireact: Toward language agent fine-tuning, 2023

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning, 2023

    2023

  7. [7]

    Decomposed prompting: A modular approach for solving complex tasks

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022

    arXiv 2022

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 55 references
  1. [9]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  2. [10]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  3. [11]

    Solving math word problems with process-and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022

    2022 arXiv

  4. [12]

    Rewarding progress: Scaling automated process verifiers for llm reasoning

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146, 2024

    arXiv

  5. [13]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2024

  6. [14]

    Trl: Transformer reinforce- ment learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforce- ment learning. https://github.com/huggingface/trl, 2020. 14

    2020

  7. [15]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proc. IEEE Int. Conf. Computer Vision and Pattern Recogni- tion, 2020

    2020

  8. [16]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024

  9. [17]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  10. [18]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2024

    2024 arXiv

  11. [19]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  12. [20]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  13. [21]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  14. [22]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  15. [23]

    Approximately optimal approximate reinforcement learning

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267– 274, 2002

    2002

  16. [24]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020

    2010 arXiv

  17. [25]

    Autogen: Enabling next-gen llm applications via multi-agent conversation framework

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  18. [26]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024

    2024

  19. [27]

    Adaplanner: Adaptive planning from feedback with language models

    Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  20. [28]

    Specification gaming: the flip side of ai ingenuity

    Victoria Krakovna. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 2020. Accessed: 2025-02-12

    2020

  21. [29]

    Reward hacking

    Lilian Weng. Reward hacking. Blog post, 2024. Accessed: 2025-02-12

    2024

  22. [30]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  23. [31]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. 15

    2022 arXiv

  24. [32]

    Iq-learn: Inverse soft-q learning for imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems , 34:4028–4039, 2021

    2021

  25. [33]

    Q* approximation schemes for batch reinforcement learning: A theoretical comparison

    Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pages 550–559. PMLR, 2020

    2020

  26. [34]

    Better than your teacher: Llm agents that learn from privileged ai feedback

    Sanjiban Choudhury and Paloma Sodhi. Better than your teacher: Llm agents that learn from privileged ai feedback. arXiv preprint arXiv:2410.05434, 2024

    2024 arXiv

  27. [35]

    Inverse reinforcement learning without reinforcement learning

    Gokul Swamy, David Wu, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning, pages 33299–33318. PMLR, 2023

    2023

  28. [36]

    Policy search by dynamic programming

    James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. Advances in neural information processing systems, 16, 2003

    2003

  29. [37]

    (more) efficient reinforcement learning via posterior sampling

    Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013

    2013

  30. [38]

    Sequence model imitation learning with unobserved contexts

    Gokul Swamy, Sanjiban Choudhury, J Bagnell, and Steven Z Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems , 35:17665–17676, 2022

    2022

  31. [39]

    Data-driven planning via imitation learning

    Sanjiban Choudhury, Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Se- bastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14):1632–1672, 2018

    2018

  32. [40]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and J Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS), 2011

    2011

  33. [41]

    Reinforcement and imitation learning via interactive no-regret learning

    Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014

    2014 arXiv

  34. [42]

    Deeply aggrevated: Differentiable imitation learning for sequential prediction

    Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International Confer- ence on Machine Learning (ICML), 2017

    2017

  35. [43]

    An application of reinforcement learning to aerobatic helicopter flight

    Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Ng. An application of reinforcement learning to aerobatic helicopter flight. Advances in neural information processing systems, 19, 2006

    2006

  36. [44]

    Learning dexterous in-hand manipulation

    OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020

    2020

  37. [45]

    Model-based reinforcement learning with a generative model is minimax optimal

    Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory, pages 67–83. PMLR, 2020

    2020

  38. [46]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  39. [47]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023

    2023 arXiv

  40. [48]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023. 16

    2023 arXiv

  41. [49]

    Archer: Training language model agents via hierarchical multi-turn rl

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024

    2024 arXiv

  42. [50]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  43. [51]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

    2024 arXiv

  44. [52]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  45. [53]

    Let’s reward step by step: Step-level reward model as the navigators for reasoning

    Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.10080, 2023

    2023 arXiv

  46. [54]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  47. [55]

    Teaching large language models to reason with reinforcement learning

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024. 17

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.