REVIEW 3 major objections 5 minor 1 cited by
Accelerating RL for LLM Reasoning with Optimal Advantage Regression
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A*-PO: regressing the optimal advantage, estimated offline from the reference policy, makes RL for LLM reasoning up to 2x faster and over 30% lighter on memory without losing accuracy.
desk verdict A genuinely useful, simple RLVR method with a clean theoretical reduction that is slightly over-sold in the abstract, and a theory whose main assumption (v_ref>0) is not formally verified on the unfiltered training sets. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the optimal advantage function $A^{\star}(x,y) = r(x,y) - V^{\star}(x)$, used as the regression target. $V^{\star}$ is recovered in closed form as a log-sum-exp expectation under the reference policy, so it can be precomputed offline. The squared loss on $\beta \ln(\pi/\pi_{\text{ref}})$ is convex in the log-ratio and is zero at the optimal policy for any sampling distribution, so a no-regret online oracle (online gradient descent or follow-the-perturbed-leader) can minimize it without exploration; the decoupling coefficient measures the sample complexity of that minimization.
What would settle it
Train A*-PO and an exploration-based method (PPO or GRPO) on a reasoning dataset containing many prompts whose Pass@50 under the reference policy is zero, such as competition problems the base model cannot solve, and compare held-out accuracy: if A*-PO stagnates at the base model's level while the baselines improve, the no-exploration claim is bounded by the positive-$v_{\text{ref}}$ assumption; if all methods stagnate, the premise that RL post-training cannot solve zero-pass problems is confirmed.
Extended reading notes
Core claim
The central claim is that the optimal policy of the KL-regularized RL objective can be learned by least-squares regression onto the optimal advantage, with no exploration mechanism. Because the optimal value has the closed form $V^{\star}(x) = \beta \ln \mathbb{E}_{y\sim\pi_{\text{ref}}(\cdot|x)}[\exp(r(x,y)/\beta)]$, it can be estimated offline from reference-policy samples. The on-policy update minimizes $\ell(\pi) = \mathbb{E}_{(x,y)\sim\pi}[\, (\beta \ln \frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)} - (r(x,y) - V^{\star}(x)))^2\,]$, whose global minimizer is $\pi^{\star}$. The paper proves that with realizability, bounded losses, and a positive reference-policy success probability, the performance gap to $\pi^{\star}$ decays at polynomial rates in both the number of offline samples $N$ and online iterations $T$, and that the final iterate of online gradient descent converges at rate $\widetilde{O}(1/\sqrt{T})$ for log-linear policies.
Load-bearing premise
The estimate of the optimal value is only accurate, and the theorem's rate is only valid, if every training prompt has a reference-policy success probability bounded above zero; prompts the base model never solves fall outside the guarantee.
Editorial extensions
If this is right
- RL post-training for reasoning can be run with one online rollout per prompt, eliminating the cost of multi-generation baselines and critic networks.
- Training time drops by up to 2x and peak memory by over 30% versus PPO, GRPO, and REBEL, without sacrificing accuracy across GSM8K, MATH, and competition-level benchmarks.
- The final policy stays closer to the reference model (smallest KL divergence), which reduces distribution shift during training.
- The theory implies that explicit exploration, such as optimism in the face of uncertainty, is unnecessary whenever the reference model has nonzero probability of solving the training prompts, making the algorithm simpler and more stable.
- For log-linear policies, A*-PO converges in the last iterate with a dimension-free rate, unlike prior policy-gradient and regression-based methods that require mixing over policies or stronger structural assumptions.
Reading between the lines
- Because Stage 1 already flags prompts where none of the N samples are correct, the same offline estimate could be reused as a training-data filter; the paper's ablation shows filtering removes roughly a quarter of the training time without hurting MATH500 accuracy, so a production pipeline could make the filter adaptive to per-prompt pass rates.
- The trajectory-level formulation appears to be what avoids exponential vocabulary dependence: by querying the ground-truth reward, the analysis sidesteps the token-level critic that prior work requires, which suggests similar offline value estimation could accelerate token-level RLHF pipelines that lack a verifiable reward.
- The regression objective is convex in the log-ratio, so any improvement in online convex optimization oracles would transfer directly to better sample complexity for LLM reasoning; pairing the loss with sharper no-regret algorithms is a concrete way to tighten the current bounds.
- A fixed $N=8$ may be wasteful for easy prompts and insufficient for hard ones; an adaptive allocation that spends more Stage-1 samples where the reference policy's pass rate is low should reduce the bias term in Theorem 1 exactly where it matters most.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces A*-PO, a two-stage reinforcement learning algorithm for fine-tuning LLMs on mathematical reasoning. Stage 1 estimates the KL-regularized optimal value function V* offline by sampling N responses per prompt from the reference policy and using the closed-form expression V*(x) = β log E_{πref}[exp(r/β)]. Stage 2 performs on-policy updates with one generation per prompt, regressing β log(π/πref) against the estimated optimal advantage r - \hat V*. The authors prove a no-regret reduction (Theorem 1) showing that, under realizability, boundedness, and a positive reference success probability assumption, the KL-regularized objective can be optimized without explicit exploration. Experiments on GSM8K, MATH, and AIME/HMMT with Qwen2.5 models report competitive or better accuracy than PPO, GRPO, and REBEL, with lower training time, memory usage, and KL divergence to the base policy.
Significance. If the results hold as stated, the paper makes a useful contribution: it replaces critics and multiple online rollouts with a single offline-estimated value and one generation per prompt, which is conceptually clean and practically attractive. The theoretical reduction is a standard and credible no-regret argument, and the gradient identity in Lemma 3 is correct as far as I checked. The empirical tables and ablations are reasonably thorough, and the provided implementation link and appendices are strengths. The significance is diminished, however, by a mismatch between the theoretical guarantee and the implemented algorithm (two different β's), and by an unverified structural assumption that the headline 'no exploration' claim relies on. The method appears genuinely competitive, but the abstract and introduction overstate the scope of the guarantees.
major comments (3)
- [Section 4 vs Section 5, Theorem 1] The theoretical guarantee is proved for Algorithm 1 with a single β used in both Stage 1 and Stage 2, but the experiments use β1=1/2 for estimating V* and β2=1e-3 for the regression loss. These are not interchangeable: \hat V_{β1}^* estimates V* for β1, whereas the Stage 2 loss corresponds to a KL-regularized objective with coefficient β2. For β2=1e-3 and any prompt with positive reference success probability, V*_{β2}(x) is approximately 1, while \hat V_{β1}^*(x) can be substantially smaller; the minimizer of (β2 log(π/πref) - (r - \hat V_{β1}^*))^2 is therefore not π*_{β2} in general. As written, the theory does not cover the evaluated algorithm. The authors should either prove a two-β variant or report experiments with β1=β2 and show that the results are preserved.
- [Section 5, Assumption 3 and Theorem 1; Section 4.4] The 'no exploration' theorem requires min_x p_ref(x) ≥ v_ref > 0, and its third error term degrades as (1/v_ref)^2 whenever v_ref < 1/(exp(1/β)-1). The paper does not verify this premise on the unfiltered MATH or OpenR1 training sets used for the main experiments. Section 4.4's filtering ablation (Figure 4) shows a material fraction of prompts with zero correct responses among N samples, and Section 6 concedes that problems with Pass@K=0 under πref are not expected to be solved by RL post-training. For such prompts \hat V^*(x)=0, producing a downward-biased regression target and no correct-response learning signal. The theorem formally applies only to the filtered subpopulation, while the abstract and Tables 1-2 describe unfiltered runs. The authors should report Pass@N distributions on the training sets and either restrict the theoretical claim to filtered training or provide evidence for Assumption 3 on the full sets.
- [Abstract and Table 1] The efficiency claim is not supported uniformly across all baselines. The 'up to 2x' training-time reduction in Table 1 is only against PPO and GRPO (e.g., 20.53 vs 11.01 h at 7B); against REBEL the largest speedup is 14.67 vs 11.01 h, roughly 1.33x. The 'over 30%' peak-memory reduction occurs only against REBEL at 1.5B and 3B (34.3% and 33.7%); against PPO and GRPO the memory savings are much smaller, and at 7B the reduction versus REBEL is about 22.5%. The abstract and Section 4.1 should disaggregate these claims by baseline rather than presenting them as uniform improvements over PPO, GRPO, and REBEL.
minor comments (5)
- [Section 4.2] There is a typo in the sentence 'the smallest KL divergence to the base mdoel πref'; 'mdoel' should be 'model'.
- [Figure 2 caption] The caption says 'across four baselines and three model sizes,' but the figure shows three baselines (PPO, GRPO, REBEL) plus A*-PO as the proposed method; 'four methods' would be more accurate.
- [Algorithm 1, line 6 and Section 4.4] The optional filtering line in Algorithm 1 is described as optional, but Section 4.4 shows that filtering materially changes training time and can improve MATH500 accuracy; the main text should state more prominently that the main results use unfiltered training and explain why the theoretical assumptions are expected to hold in that setting.
- [Appendix F, proof of Lemma 3] The displayed derivation of the gradient identity changes notation from 2β to 2β^2 without an explicit justification; adding one sentence explaining that the constant tracks the coefficient β in the loss would improve readability.
- [Section 5, Corollaries 1 and 2] Corollary 1 and Corollary 2 present nearly identical bounds and are both labeled informal; clarifying which statement is exact and which is informal, and why both are needed, would help the reader.
Circularity Check
No significant circularity: the regression target is a Monte Carlo estimate of the closed-form optimal value function derived from the KL-regularized objective, and the theoretical guarantee is conditional on explicit, un-fitted assumptions.
full rationale
A*-PO's derivation is self-contained rather than circular. The optimal policy and optimal value function are derived from first principles: V*(x) = beta ln E_{y~pi_ref}[exp(r(x,y)/beta)] is the closed-form solution of the stated KL-regularized objective, not an object defined in terms of the algorithm's outputs. Stage 1 estimates V* by offline sampling from pi_ref, which is a standard Monte Carlo estimate of that explicit expectation; the estimate is not a fitted parameter later renamed as a prediction. Stage 2 regresses beta ln(pi/pi_ref) onto r(x,y) - Vhat*(x), and the claim that pi* minimizes this least-squares loss is a pointwise algebraic identity when Vhat* = V*, so it does not assume the conclusion it is used to prove. The theoretical bound in Theorem 1 rests on explicit assumptions--realizability, boundedness, and v_ref > 0--that are not satisfied by construction; in particular, Assumption 3 is a genuine structural premise, and the paper honestly flags that RL post-training is not expected to solve prompts whose Pass@50 under pi_ref is zero. This is a scope limitation or correctness risk for the unfiltered AIME/HMMT experiments, but it is not circularity. Self-citations to REBEL and Q# appear only as baselines or related work and are not load-bearing for the derivation. Overall, the central claim is not equivalent to its inputs by definition, and there is no fitted input masquerading as a prediction.
Assumptions & free parameters
free parameters (3)
- beta1 (offline V* estimation temperature) =
1/2
- beta2 (stage-2 KL regularization coefficient) =
1e-3
- N (number of offline samples per prompt) =
8
assumptions (5)
- standard math The optimal policy of KL-regularized RL has closed form π*(y|x) ∝ π_ref(y|x) exp(r(x,y)/β), and V*(x) = β ln E_{π_ref}[exp(r/β)].
- domain assumption Realizability: the optimal policy π* belongs to the policy class Π (Assumption 1).
- domain assumption Bounded log-ratio: |β ln(π(y|x)/π_ref(y|x))| ≤ C_l for all π∈Π (Assumption 2).
- domain assumption Positive reference probability: E_{y∼π_ref}[r(x,y)] ≥ v_ref > 0 for all x (Assumption 3).
- domain assumption Fisher information lower bound: A(θ_t) ⪰ λI for all t (Assumption 4).
Cite this review
Pith. "Pith review of Accelerating RL for LLM Reasoning with Optimal Advantage Regression." pith.science (2026). https://pith.science/paper/CYGDKISX
@misc{pith2026250520686,
author = {Pith},
title = {Pith review of: Accelerating RL for LLM Reasoning with Optimal Advantage Regression},
year = {2026},
howpublished = {\url{https://pith.science/paper/CYGDKISX}},
note = {Machine review of arXiv:2505.20686}
}
abstract
Reinforcement learning (RL) has emerged as a powerful tool for fine-tuning large language models (LLMs) to improve complex reasoning abilities. However, state-of-the-art policy optimization methods often suffer from high computational overhead and memory consumption, primarily due to the need for multiple generations per prompt and the reliance on critic networks or advantage estimates of the current policy. In this paper, we propose $A$*-PO, a novel two-stage policy optimization framework that directly approximates the optimal advantage function and enables efficient training of LLMs for reasoning tasks. In the first stage, we leverage offline sampling from a reference policy to estimate the optimal value function $V$*, eliminating the need for costly online value estimation. In the second stage, we perform on-policy updates using a simple least-squares regression loss with only a single generation per prompt. Theoretically, we establish performance guarantees and prove that the KL-regularized RL objective can be optimized without requiring complex exploration strategies. Empirically, $A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks, while reducing training time by up to 2$\times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL. Implementation of $A$*-PO can be found at https://github.com/ZhaolinGao/A-PO.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
OBLR-PO combines an SNR-based learning rate and a gradient-weighted baseline for RLVR, but the main theory is undermined by a flawed smoothness proof.
Reference graph
Works this paper leans on
-
[1]
Peak memory usage is averaged over 100 batches. Following prior work [Zeng et al., 2025], we evaluate model performance on standard mathematical reasoning benchmarks, including MATH500 [Hendrycks et al., 2021], Minerva Math [Lewkowycz et al., 2022], and OlympiadBench [He et al., 2024], as well as competition-level benchmarks such as AMC 2023 and AIME/HMMT...
work page 2025
-
[2]
The minimum value ofa2 occurs whenais at its maximum value, which is -2. So,a2 = (−2)2 = 4. Therefore, the greatest possible value of the expression is: 1 9 −4 =1 9 − 36 9 = 1−36 9 = −35 9 So, the greatest possible value of the expression is−35 9 . Qwen2.5-7B afterA⋆-PO Generation:To find the greatest possible value of the expressiona+ 1 b 1 b −a , we sta...
-
[3]
Whena=−6andb= 5: −6 +1 5 1 5 + 6 = −29 5 31 5 =−899 25
-
[4]
The maximum value of1 b2 occurs whenbis at its minimum value, which is 3. So,1 b2 = 1 32 = 1 9
-
[6]
Since3≤b≤5 , the minimum value ofb is 3
The term1 b2 is maximized whenb is minimized. Since3≤b≤5 , the minimum value ofb is 3. Therefore, the maximum value of1 b2 is 1 32 = 1 9
-
[7]
Since−6≤a≤ −2, the minimum value ofa2 is (−2)2 = 4
The term−a2 is maximized whena2 is minimized. Since−6≤a≤ −2, the minimum value ofa2 is (−2)2 = 4. Therefore, the maximum value of−a2 is−4. Combining these, the greatest possible value of the expression is: 1 9 −4 =1 9 − 36 9 =−35 9 . However, we need to check if this is indeed the maximum value by considering the boundary conditions. Let’s verify by calcu...
-
[8]
Whena=−6andb= 3: −6 +1 3 1 3 + 6 = −17 3 19 3 =−323 9
-
[9]
Whena=−2andb= 3: −2 +1 3 1 3 + 2 = −5 3 7 3 =−35 9
Show all 13 references
-
[11]
From these calculations, we see that the greatest possible value is indeed achieved whena=−2andb= 3, giving us: −35 9
Whena=−2andb= 5: −2 +1 5 1 5 + 2 = −9 5 11 5 =−99 25. From these calculations, we see that the greatest possible value is indeed achieved whena=−2andb= 3, giving us: −35 9 . In this example, Qwen2.5-7B after A⋆-PO exhibits verification behavior where it performs systematic err...
-
[12]
a+c= 1−1 = 03
a+b= 1 +ei π 2 = 1 +i2. a+c= 1−1 = 03. a+d= 1−i4. b+c=i−1 5. b+d=i−i= 0 6. c+d=−1−i So, the expression becomes: |(a+b)(a+c)(a+d)(b+c)(b+d)(c+d)|=|(1 +i)·0·(1−i)·(i−1)·0·(−1−i)|= 0. This configuration does not yield the maximum value. Let’s consider another configuration, such ...
-
[13]
X y exp(⟨θT+1 , ϕ(x, y)⟩)P y′ exp(⟨θT+1 , ϕ(x, y′)⟩) − exp(⟨θ⋆, ϕ(x, y)⟩)P y′ exp(⟨θ⋆, ϕ(x, y′)⟩) # ≤Ex
a+b=e i π 4 +ei 3π 4 2. a+c=e i π 4 +ei 5π 4 3. a+d=e i π 4 +ei 7π 4 4. b+c=e i 3π 4 +ei 5π 4 5. b+d=e i 3π 4 +ei 7π 4 6. c+d=e i 5π 4 +ei 7π 4 Using the properties of complex numbers, we can simplify these expressions. For example,eiθ+ei(θ+2π 4 ) =eiθ(1 +ei π 2 ) = eiθ(1 +i)....
2020
-
[2023]
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach
doi: 10.1017/9781009093057. Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining.arXiv preprint arXiv:2504.07912, 2025. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha K...
-
[2025]
Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy
URLhttps://arxiv.org/abs/2501.12948. Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for llms.arXiv preprint arXiv:2402.00396, 2024. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognit...
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.