REVIEW 4 major objections 4 minor 2 cited by
Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Group Relative Policy Optimization can be run off-policy: using a slightly stale policy to estimate the advantage provably improves reward, stabilizes LLM post-training, and lowers serving communication cost.
desk verdict A clean GRPO-specific lower bound and a nice justification for zero-variance masking, but the off-policy experiments run an algorithm whose drift term the theory does not control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the improvement lower bound of Theorem 1, built on the GRPO advantage $A_\alpha(x,y) = (r(x,y)-\mu_{\alpha,r}(x))/\sigma_{\alpha,r,\varepsilon}(x)$, a whitened reward whose mean and variance are computed under the sampling policy α. The bound's negative terms are total variation distances $TV(\pi, \alpha)$ and $TV(\pi_k, \alpha)$, weighted by $(1-\sigma_{\alpha,r,\varepsilon})/\sigma_{\alpha,r,\varepsilon}$, which is policy- and data-dependent (unlike PPO's constants) and blows up as reward variance goes to zero. From the bound the paper derives a penalized/constrained objective, then a clipped surrogate $L^c_\alpha$ with a KL regularizer, and shows the on-policy GRPO clipped objective is the special case α = π_k.
What would settle it
Run off-policy GRPO with a fixed v while recording TV(π_k, π_{k-v}) and the empirical reward change per iteration; if reward improvement J(π_{k+1}) - J(π_k) is negative in a regime where the lower bound's right-hand side is positive, or if increasing v flips the improvement sign as the TV penalty grows, the closeness premise of Theorem 1 is violated.
Extended reading notes
Core claim
The central claim is that maximizing the GRPO objective—the expected whitened reward under importance sampling—is a surrogate for maximizing expected reward, provided the sampling policy stays near the current policy. Concretely, the paper proves for any policies π, π_k, α and reward bounded in [0,1] that $$J(\pi) - J(\pi_k) \geq L_\$\alpha$(\pi) - 2\frac{1-\sigma_{\$\alpha$,r,\varepsilon}}{\sigma_{\$\alpha$,r,\varepsilon}} TV(\pi, \$\alpha$) - 2 TV(\pi_k, \$\alpha$)$$ so the GRPO advantage objective lower-bounds actual policy improvement up to TV-distance penalties. As a corollary, when α = π_k the on-policy GRPO objective enjoys the same guarantee, and the explosion of (1-σ)/σ at zero reward variance explains the stabilizing effect of DAPO-style masking. The paper then derives the clipped surrogate used in practice from a KL-constrained version of this bound, and verifies on LLM reasoning tasks that off-policy GRPO (α = π_{k-v} with v=10 and one SGD pass per batch) performs on par or better than on-policy GRPO while updating the served model less frequently.
Load-bearing premise
The guarantee assumes the stale sampling policy α stays close to the current policy π_k and that reward variance stays bounded away from zero; the paper's practical algorithm takes α = π_{k-v} and approximates the ratio π_k/α by 1 without enforcing or measuring that closeness, so if the policy drifts over v steps the penalty terms in the bound can dominate and improvement is not guaranteed.
Editorial extensions
If this is right
- On-policy GRPO's clipped objective is now justified from first principles as maximizing a lower bound on reward improvement.
- Zero-variance masking (DAPO) is theoretically grounded: masking controls the exploding coefficient $(1-\sigma_{\alpha,r,\varepsilon})/\sigma_{\alpha,r,\varepsilon}$.
- Off-policy GRPO with $v>1$ and $i=1$ delivers reward improvement guarantees under closeness of $\pi_{k-v}$ to $\pi_k$, enabling fewer model-serving updates.
- In experiments, off-policy GRPO stabilizes GSM8K training (Pass@1 from an unstable 45% to a stable 50%) and matches or exceeds on-policy performance on AIME24 and MATH500.
- Communication and inference-serving overhead can be amortized without losing performance.
Reading between the lines
- The lower bound suggests an adaptive scheduler: $v$ could be increased while measured $TV(\pi_k, \pi_{k-v})$ stays small, and shrunk when it grows, rather than fixing $v=10$.
- The same $(1-\sigma)/\sigma$ weighting could justify soft reweighting of prompts by inverse predicted entropy instead of hard masking.
- The theory opens the door to combining off-policy GRPO with replay buffers, since the advantage is already estimated under a stale policy.
- A direct comparison against off-policy PPO variants would isolate whether the benefit comes from the whitened GRPO advantage or from the staleness itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper revisits Group Relative Policy Optimization (GRPO) and proposes an off-policy variant in which the whitened advantage is computed from responses sampled by a delayed policy α = π_{k−v}. The main theoretical claim is Theorem 1: for rewards in [0,1], the expected-reward improvement J(π(·|x)) − J(π_k(·|x)) is lower-bounded by the importance-sampled off-policy advantage L_α(π(·|x)) minus two total-variation penalty terms, one involving TV(π, α) and one involving TV(π_k, α). Corollary 1 gives the on-policy version. The authors then pass from this lower bound to a KL-constrained problem and to a clipped surrogate objective, and present experiments on GSM8K and DeepScaleR comparing on-policy GRPO (v=1, i=1) with off-policy GRPO (v=10, i=1), reporting that off-policy training is stable and performs on par or better. The paper also provides a theoretical interpretation of DAPO's zero-variance masking through the quantity (1 − σ_{α,r,ε})/σ_{α,r,ε} appearing in the lower bound.
Significance. Theorem 1 is a useful and non-obvious lower bound when its hypotheses are met: it identifies policy-dependent constants that are absent from the absolute constants in PPO-style bounds and gives a principled explanation for the stabilizing effect of zero-variance masking. The proof is elementary, and the on-policy corollary is correct modulo the ε-range issue noted below. The paper also clearly states its experimental limitations and builds on open reproducible components (TRL, vLLM, Open-R1). The main weakness is the gap between theory and the implemented algorithm: the practical off-policy objective approximates the importance ratio by 1 and never measures the total-variation drift term that Theorem 1 requires to be small, so the formal guarantee does not currently cover the experiments. The clipped-surrogate derivation is also a heuristic rather than a formal consequence of the lower bound. These issues make the paper a solid proof-of-concept requiring revision rather than a fully established theoretical claim.
major comments (4)
- [Section 3.1 / Appendix C.1, Theorem 1] The theorem is stated for 0 < ε < 1, but the proof requires σ_{α,r,ε}(x) ≤ 1 so that (1 − σ_{α,r,ε}(x))/σ_{α,r,ε}(x) ≥ 0. For r ∈ [0,1], σ^2_{α,r}(x) ≤ 1/4, hence σ_{α,r,ε}(x) ≤ sqrt(1/4 + ε), which is ≤ 1 only for ε ≤ 3/4. The bound is therefore not valid for ε ∈ (3/4, 1). Please restrict the statement to 0 < ε ≤ 3/4, or normalize the reward by sqrt(1/4 + ε), and correct the accompanying text that says 'we have σ_{α,r}(x) ≤ 1/4', which should be σ^2_{α,r}(x) ≤ 1/4.
- [Section 3.2, Eq. (6)] The derivation from Theorem 1 to the clipped surrogate is not a formal consequence of the bound. Theorem 1 bounds J(π(·|x)) − J(π_k(·|x)) by L_α(π(·|x)) minus penalty terms that depend on TV(π, α); maximizing the penalized objective in Section 3.2 does not automatically maximize the lower bound because L_α and the penalty are not independent, and the subsequent replacement of the TV constraint by a KL constraint and then by clipping is a heuristic standard in PPO-style algorithms. The sentence 'By virtue of Theorem 1, maximizing this objective above leads to policy reward improvement' should be reframed as intuition or motivation unless an exact oracle or additional analysis is supplied.
- [Section 3.2, Algorithm 1, Table 1] The experiments labeled off-policy GRPO (v=10, i=1) are not covered by Theorem 1. The theorem's lower bound contains the term −2 E_x TV(π_k(·|x), α(·|x)), which must be small for the bound to imply improvement, and Section 3.2 further approximates π_k(y|x)/π_{k−v}(y|x) by 1. Neither the total-variation distance nor the approximation error is measured in Section 5, and with v=10 and the reference swap in Algorithm 1 (line 16) the served policy π_{k−v} can drift from π_k. Please either measure and report the drift, reduce v to a range where the drift is empirically verified, or soften the claim that the theoretical guarantee applies to the implemented algorithm.
- [Section 5, Tables 2-3 and Figure 2] The empirical claim that off-policy GRPO 'either significantly outperforms or performs on par' is only weakly supported. On GSM8K both variants reach 50% Pass@1, and on DeepScaleR the ranges overlap: Aime24 is 0.2802–0.3229 for on-policy versus 0.2781–0.3250 for off-policy, and Math500 is 0.830–0.870 versus 0.822–0.872. No standard errors, repeated seeds, or significance tests are reported. The results are consistent with 'on par', but the 'significantly outperforms' part is not substantiated.
minor comments (4)
- [Section 3.1, proof of Theorem 1] The statement 'we have σ_{α,r}(x) ≤ 1/4' in the interpretation paragraph is dimensionally wrong: the variance is at most 1/4, so the standard deviation is at most 1/2. The same slip appears in Appendix C.1.
- [Throughout] The manuscript uses textual references to 'eq. (4)' and 'Equation (7)' without a visible equation-numbering scheme; numbered equations would make the paper much easier to referee and to read.
- [Throughout] There are several typos and infelicities, including 'standarized', 'librarires', 'layed conditions', and 'this will have the effect of controlling this term'; a careful proofreading pass is needed.
- [Section 3.2 and References] The citation 'vLLM [2025]' points only to a URL for a benchmark dashboard; this should be replaced with a proper reference or removed, since it is used to support statements about serving latency.
Circularity Check
No circularity: Theorem 1 is an algebraic lower bound derived from definitions; self-citations are motivational, not load-bearing.
full rationale
The paper's central claim is Theorem 1, an inequality lower-bounding J(pi)-J(pi_k) by the off-policy advantage objective L_alpha(pi) minus total-variation penalty terms. The proof (Appendix C.1) is a direct algebraic manipulation: L_alpha(pi) = (J(pi)-J(alpha))/sigma_alpha, combined with Kantorovich-Rubinstein duality to bound |J(pi)-J(alpha)| and |J(pi_k)-J(alpha)| by 2*TV terms. The improvement conclusion is not assumed; it is a consequence of the inequality, and the bound can be negative when the TV terms dominate. Nothing is fitted to make the bound come out positive. The clipped off-policy surrogate follows from standard Pinsker/KL-constrained relaxation of the penalized objective, and the on-policy GRPO clipped objective is recovered as the special case alpha=pi_k. The zero-variance masking explanation is post hoc: DAPO's empirical masking is interpreted through the term (1-sigma)/sigma in the bound, but the theorem was not constructed from masking, so this is not circular. The citations to Mroueh [2025] are used for motivation ('a recent analysis suggests...') and for empirical concordance ('this concurs with the theoretical findings in Mroueh [2025]'), but Theorem 1's proof does not cite or depend on that work; hence the self-citation is not load-bearing. The practical implementation does introduce an unverified approximation: alpha=pi_{k-v} with pi_k/pi_{k-v} approximated by 1, and the requirement that TV(pi_k, alpha) be small is asserted as 'reasonable' for small learning rate and small v rather than enforced or measured. This is a gap between theory and the implemented algorithm and a legitimate correctness risk, but it is not a circular reduction: the approximation is not used to define the theorem's conclusion, nor is the theorem used to justify the approximation by definition. No identified step reduces the paper's stated results to its own inputs or to a fitted parameter renamed as a prediction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- epsilon (advantage stabilization) =
not specified; typically small (e.g., 1e-4 in GRPO)
- v (off-policy delay) =
10 in the reported experiments
- beta (KL regularizer) =
0.1 (GSM8K) and 0.001 (DeepScaleR)
assumptions (6)
- domain assumption The reward function r is bounded in [0,1] (or is rescaled so that ||r||_infinity <= 1)
- standard math Kantorovich-Rubinstein duality for total variation distance
- standard math Pinsker inequality bounding TV by KL divergence
- domain assumption The off-policy alpha stays in the vicinity of pi_k (TV(pi_k, alpha) <= delta)
- ad hoc to paper The ratio pi_k / alpha is approximated by 1 in the practical off-policy clipped objective
- domain assumption The clipped surrogate objective is an adequate practical proxy for the KL-constrained objective
Cite this review
Pith. "Pith review of Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training." pith.science (2026). https://pith.science/paper/BAOAZKMS
@misc{pith2026250522257,
author = {Pith},
title = {Pith review of: Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/BAOAZKMS}},
note = {Machine review of arXiv:2505.22257}
}
read the original abstract
We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart.
Figures
Forward citations
Cited by 2 Pith papers
-
REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning
Using only the replay buffers from RL training of domain-expert LLMs, REGEN trains a multi-domain generalist via offline RL and matches online multi-teacher distillation accuracy at a fraction of the reported training cost.
-
Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs
SOUP mixes off-policy historical prefixes with on-policy continuations at token level and reports small but consistent math-reasoning gains over on-policy GRPO/DAPO baselines in selected configurations.
Reference graph
Works this paper leans on
-
[1]
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,
-
[7]
N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. Tülu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,
-
[9]
URLhttps://arxiv.org/abs/2503.06639. M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville. Faster, more effi- cient RLHF through off-policy asynchronous learning. InThe Thirteenth International Conference on Learning Representations,
-
[11]
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[12]
Accessed: 2025-05-14. M. Vojnovic and S.-Y. Yun. What is the alignment objective of grpo?,
work page 2025
-
[13]
URL https:// arxiv.org/abs/2502.18548. L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec. Trl: Transformer reinforcement learning.https://github.com/huggingface/trl, 2020a. L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec. Trl: Tr...
-
[14]
URLhttps://arxiv.org/abs/2503.14476. 14 REVISITING GROUP RELATIVE POLICY OPTIMIZATION AppendixA.Broader Impact and Limitations Our work analyzes the celebrated GRPO algorithm and develops an adaptation for the off- policy setting motivated by recent efforts for PPO that demonstrated higher stability and efficiency. Our primary contributions are theoretica...
arXiv 2025
-
[15]
(MIT license). The models that we used wereQwen/Qwen2.5-0.5B-Instruct (Apache REVISITING GROUP RELATIVE POLICY OPTIMIZATION 15 2.0 license) by Yang et al. [2024], andDeepSeek-R1-Distill-Qwen-1.5B (MIT license) by Guo et al. [2025]. AppendixC.Rew ard Improvement Lower Bound C.1.Proof of Theorem 1.We have : J(π(·|x)) =E y∼π(·|x)r(x, y) Letπ k be the current...
work page 2024
Show all 15 references
-
[2015]
Schulman, F
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
-
[2017]
URLhttps://proceedings.neurips.cc/paper_files/ paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math ...
2017 arXiv
-
[2020]
Y. Gan, R. Yan, X. Tan, Z. Wu, and J. Xing. Transductive off-policy proximal policy optimization. arXiv preprint arXiv:2406.03894,
-
[2021]
Degris, M
T. Degris, M. White, and R. S. Sutton. Off-policy actor-critic.arXiv preprint arXiv:1205.4839,
-
[2023]
HuggingFace
URLhttps://github.com/huggingface/lighteval. HuggingFace. Open r1: A fully open reproduction of deepseek-r1, January 2025a. URLhttps: //github.com/huggingface/open-r1. HuggingFace. Open r1: Update #3, Mar. 2025b. URLhttps://huggingface.co/blog/open-r1/ update-3. Accessed: 2025...
2025
-
[2024]
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
-
[2025]
URLhttps://arxiv.org/abs/2503.20783. M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, T. Zhang, L. E. Li, R. A. Popa, and I. Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://tinyurl.com/5e9rs33z,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.