{"id":"bbe7da1c-6d08-4bb0-9be5-806441cbe9f5","arxiv_id":"2505.23585","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A length-weighted reward baseline plus exact on-policy updates improves stability and diversity in LLM reinforcement learning for math, with small performance gains over GRPO.","lead":"OPO is a simplified reinforcement learning algorithm for large language models: it updates the model on fresh samples every step and subtracts a length-weighted reward baseline to reduce gradient variance. The paper reports better math reasoning scores and more stable, diverse training than GRPO without needing value models, reference models, or KL and entropy penalties.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The length-weighted baseline in Eq. 12 is a mean-field approximation of the true optimal baseline, not its exact form even under the stated orthogonality assumptions.","rationale":"The reader identified the orthogonality and identical per-token gradient norm assumptions as the weakest point. My stress-test agrees with that direction but sharpens it: even if those assumptions hold, Eq. 10 does not follow exactly. The optimal baseline weights each trajectory by its realized squared gradient norm; the paper's simplification replaces that realized weight by its expectation conditional on length. This is a mean-field approximation that requires additional unstated conditions (e.g., small variance of w given l and r, or independence of w from r). Thus the theoretical basis for the central claim is weaker than the paper asserts, independent of whether the orthogonality assumption is empirically true. The empirical results may still support OPO as a recipe, and the reader's CONDITIONAL verdict already accounts for this type of risk, so I do not change the verdict. The proposed gradient-norm measurement would directly test whether the length-weighted baseline is actually near-optimal in the operating regime, settling the concern rather than leaving it as a purely logical gap.","tokens_in":13470,"tokens_out":4065,"duration_ms":42252,"concrete_test":"On a sample of ~200 prompts from the training distribution, using both the initial SFT policy and a later on-policy checkpoint, sample K=8 responses per prompt and compute per-trajectory w(y) = ||∇_θ log π_θ(y|x)||² (e.g., by accumulating squared per-token gradient norms) and rewards r. Compute b_len = Σ l_i r_i / Σ l_i and b_opt = Σ w_i r_i / Σ w_i per prompt. Then estimate the variance of the policy-gradient estimator under each baseline via repeated resampling. If the variance under b_len exceeds that under b_opt by more than a small tolerance, or if b_len and b_opt differ by more than 10% of the reward range, Eq. 12 is not near-optimal and the theoretical claim fails. Also check whether w(y) is approximately independent of reward given length; if not, the length-weighted baseline is biased as well as noisy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical claim is that Eq. 12, b*(x) = Σ l_i r_i / Σ l_i, is the optimal baseline that minimizes policy-gradient variance. The derivation in §3.2 proceeds from the true optimum b* = E[||g(y)||² r] / E[||g(y)||²] (Eq. 9) to the length-weighted form by assuming token gradients are approximately orthogonal and per-token gradient norms have a common distribution. However, those assumptions only imply E[||g(y)||² | l_y] ∝ l_y. They do not imply that the realized weight w(y) = ||g(y)||² equals c·l_y, nor that w(y) is independent of the reward. Eq. 10 replaces the optimal weight w(y) with its conditional expectation c·l_y, which is an uncontrolled mean-field approximation. Unless Var(w | l, r) is negligible, the baseline in Eq. 12 is not the variance-minimizing baseline; it is just another heuristic baseline. Consequently, the claim that OPO's baseline is 'theoretically sound' collapses to an empirical recipe even if the orthogonality assumption is true. The paper provides no measurement of w(y) versus length or of gradient-estimator variance to support this approximation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes On-Policy RL with Optimal reward baseline (OPO), a policy-gradient method for fine-tuning LLMs on reasoning tasks. OPO combines exact on-policy updates with a length-weighted reward baseline b*(x) = Σ l_i r_i / Σ l_i, which the authors claim is a practical form of the variance-minimizing baseline for sequence generation. The method removes value models, reference models, and KL/entropy regularization. Experiments on DeepSeek-R1-Distill-Qwen-7B on MATH-500, AIME 2024, and AIME 2025 compare on-policy vs off-policy GRPO and OPO vs GRPO, and report diversity and repetition metrics (Self-BLEU, Rep-5). The central theoretical claim is that under approximate token-gradient orthogonality and a shared per-token gradient norm distribution, the optimal baseline reduces to the length-weighted reward average.","tokens_in":13676,"tokens_out":6821,"duration_ms":63894,"significance":"If the method worked as claimed, OPO would be a useful simplification of LLM RL training, removing auxiliary models and regularizers while improving stability. That would be a practically valuable result, especially given the public implementation in the verl library and the clear algorithmic description. The standard derivation of the variance-minimizing baseline in Appendix A is correct for a fixed input, and the experimental setup isolates some comparisons between methods. However, the central theoretical step from the exact optimal baseline to the length-weighted form is not justified by the stated assumptions, the same-batch baseline introduces a finite-sample bias that is not discussed, and the empirical results are reported without uncertainty estimates. The paper's headline claim of 'theoretically sound' optimality is therefore not established, and the empirical evidence is weaker than the text suggests.","major_comments":[{"comment":"The step from Eq. (9) to Eq. (10) is not justified by the stated orthogonality and identical-distribution assumptions. Those assumptions imply E[||g(y)||^2 | l_y] ∝ l_y, not that the realized weight ||g(y)||^2 equals c·l_y, nor that this weight is independent of the reward r(x,y). Since b* in Eq. (9) is a ratio of expectations involving the reward-weighted squared gradient norm, replacing the random weight by its conditional expectation is an uncontrolled mean-field approximation. Unless Var(||g||^2 | l, r) is negligible, Eq. (12) does not minimize gradient variance. The paper provides no measurement of realized gradient norms or of gradient-estimator variance to support this approximation, so the 'theoretical soundness' claim made in the Introduction and Section 3.2 is unsupported.","section":"Section 3.2, Eq. (10)"},{"comment":"The baseline b*(x) is computed from the same K responses whose log-probabilities appear in the policy-gradient sum, so the estimator is not the unbiased policy gradient. For the plain mean baseline, E[(1/K)Σ_i ∇logπ(y_i)(r_i − mean_j r_j)] = ((K−1)/K) E[∇logπ(y) r(y)], a K-dependent shrinkage. The length-weighted baseline has the same finite-sample issue, compounded by the dependence of the denominator on all sampled lengths. The paper neither uses a leave-one-out baseline nor corrects the resulting scale factor, yet K=8 or K=16 is small enough that this bias is non-negligible. This is a distinct issue from variance reduction and affects the interpretation of the optimized objective in Eq. (11).","section":"Section 3.3, Eqs. (11)-(12)"},{"comment":"All reported results are single-run values averaged over the last five checkpoints, with no multiple seeds, error bars, or checkpoint-level variance. Several OPO/GRPO differences are small or in the opposite direction of the claimed improvement; for example, AIME 2025 pass@1 is 50.00 for OPO vs 50.21 for GRPO, and pass@2 is 60.88 vs 61.45. The diversity and repetition claims in Section 4.3 similarly rest on differences of a few tenths of a point in Self-BLEU and Rep-5. Without uncertainty quantification, the statement that OPO 'outperforms GRPO in most cases' is not statistically supported.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The on-policy vs off-policy comparison is confounded by hyperparameter differences. On-policy training uses a mini-batch size of 256, off-policy uses 128, and off-policy includes a 0.001 entropy penalty while on-policy uses none. These differences, rather than exact on-policy training itself, could drive parts of the observed gap in Table 1. The comparison should hold these factors fixed or provide an ablation isolating the on-policy component.","section":"Section 4.1"}],"minor_comments":[{"comment":"Equations (7)-(9) and the appendix write (g(y))^2, but for a vector-valued gradient the variance objective should use the squared norm ||g(y)||^2; otherwise the scalar notation is ambiguous.","section":"Section 3.2 and Appendix A"},{"comment":"There is a typo: 'reasponses' should be 'responses'.","section":"Section 4.1"},{"comment":"The caption contains the typo 'comparision' instead of 'comparison'.","section":"Table 1 caption"},{"comment":"The text says 'Tables 1 and 2 summarize the results' when discussing Rep-5 and Self-BLEU, but those metrics appear in Tables 3 and 4.","section":"Section 4.3"},{"comment":"The figure captions in the manuscript contain raw '\\uni0000...' character sequences that appear to be rendering artifacts; these should be cleaned before publication.","section":"Figures 1-3"},{"comment":"The MATH-500 rows report pass@1 through pass@8 but not pass@16, making the table inconsistent with the AIME rows; consider adding the value or explaining its omission.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the empirical recipe may be useful, but the 'optimal' label is stronger than the derivation supports. I would encourage the editor to require either direct measurements of gradient weights and estimator variance, or a reframing of the length-weighted baseline as a heuristic approximation with the 'optimal' terminology softened. The finite-sample bias of a same-batch baseline and the lack of uncertainty estimates should also be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but the theoretical headline oversells the derivation. What is new here is the combination: exact on-policy updates with a length-weighted baseline b* = sum(l_i r_i)/sum(l_i), and no critic, reference model, or KL/entropy term. The baseline lineage goes back to Weaver and Tao and Greensmith et al., and the paper is honest about that. The specific recipe is genuinely not in GRPO, ReMax, RLOO, or Reinforce++, and the implementation is merged into verl, so the work is reproducible. The empirical results on math reasoning are plausible and the exact on-policy comparison in Table 1 is a nice clean ablation.\n\nThe soft spot is the step from the true variance-minimizing baseline to Eq. 12. The paper's orthogonality and same-distribution assumptions give you E[||g||^2 | l] proportional to l, but to get the length-weighted form you need w(y) = ||g(y)||^2 to be proportional to l pointwise and independent of reward. Replacing w with its conditional expectation is an uncontrolled mean-field approximation. The stress-test note is correct on this. So the phrase \"theoretically sound\" is too strong; the baseline is a heuristic with a plausible story. The empirical evidence doesn't close the gap: single runs, no error bars, and several OPO vs GRPO deltas are within noise (AIME 2025 pass@1 is 50.00 vs 50.21, actually worse for OPO). The paper's \"consistently\" more diverse claim is also undercut by its own Table 4, where OPO has slightly worse Self-BLEU on MATH-500. The paper also concedes in Appendix B that a more comprehensive evaluation is future work.\n\nThat said, the recipe may work. No circularity: the baseline isn't fitted to the results. The paper is clear, the code is public, and the idea is simple enough that a referee can check it quickly. Who is this for? People training reasoning models who want a cheap, simple alternative to GRPO. It deserves a serious referee, but the referee should ask for error bars, an ablation comparing the actual gradient-norm weights to the length proxy, and a softening of the \"optimal\" language. I'd send it to review with the expectation of revision.","headline":"OPO's length-weighted baseline is a useful practical recipe, but the 'optimal' label rests on an unverified mean-field step and the empirical evidence is single-run with small gaps.","tokens_in":14242,"tokens_out":2504,"would_cite":true,"duration_ms":23812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Length-weighted reward baseline stabilizes LLM reinforcement learning","keywords":["reinforcement learning","large language models","policy gradient","variance reduction","reward baseline","on-policy training","mathematical reasoning","GRPO"],"falsifier":"Measure, for an actual policy being trained, the per-token squared gradient norms across a sample of responses; if the norm is not approximately proportional to response length, the derivation of $b^*(x)$ fails. Alternatively, compare the empirical variance of policy-gradient estimates under the length-weighted baseline versus the group-mean baseline on a fixed set of prompts; if the length-weighted baseline does not reduce variance, the paper's central mechanism is not doing the claimed work.","tokens_in":13228,"feed_emoji":"🧮","tokens_out":7101,"duration_ms":53353,"temperature":0.7,"pith_summary":"This paper proposes OPO, a reinforcement learning algorithm for large language models that combines two ideas: updating the policy only on data freshly sampled from the current policy, and subtracting a length-weighted average reward as the baseline. The authors argue that this baseline is the practically feasible form of the theoretically optimal variance-reducing baseline, once one assumes token-level gradients are roughly orthogonal and identically distributed. They show empirically on mathematical reasoning benchmarks that OPO outperforms GRPO and on-policy variants, with lower policy shift, higher output entropy, and no value model, reference model, or KL/entropy regularization. If correct, OPO suggests that a single policy model with a simple length-weighted baseline suffices for stable and diverse RL training.","feed_headline":"Length-weighted reward baseline stabilizes LLM reinforcement learning","feed_subtitle":"OPO removes value and reference models plus KL and entropy terms, yet trains stably and diversely on math benchmarks.","key_machinery":"The central object is the length-weighted reward baseline $b^*(x) = \\sum_i l_i r_i / \\sum_i l_i$, used to define the advantage $A_i = r(x,y_i) - b^*(x)$ in a basic policy-gradient objective. It is derived from the classical optimal baseline $b^* = \\mathbb{E}[g(y)^2 r]/\\mathbb{E}[g(y)^2]$ by assuming token-level gradients are orthogonal with identically distributed norms, so that the squared gradient norm is proportional to response length. The other load-bearing mechanism is exact on-policy training: every gradient step uses rollouts sampled from the current policy, avoiding the off-policy drift that the authors link to entropy collapse and large policy shifts.","core_discovery":"The central claim is that the variance-minimizing baseline for policy gradient in sequence generation reduces, under a plausible assumption, to a length-weighted average of rewards: $b^*(x) = \\sum_i l_i r(x,y_i) / \\sum_i l_i$. The derivation follows the classical optimal baseline formula, replacing the squared gradient magnitude with response length on the grounds that token gradients are approximately orthogonal and share a common norm distribution, making $\\|\\nabla_\\theta \\log \\pi_\\theta(y|x)\\|^2$ proportional to $l_y$. Combined with exact on-policy updates—one gradient step per freshly sampled batch—this baseline yields stable training that needs no value model, reference model, or KL/entropy regularization, and it outperforms GRPO on MATH-500, AIME 2024, and AIME 2025 while producing more diverse and less repetitive outputs.","pith_inferences":["A length-weighted baseline implicitly rewards longer responses more when longer responses correlate with higher reward; on math tasks this may function as a soft 'thinking-time' prior, and the authors' assumption predicts that its benefit should shrink on tasks where length and reward are anti-correlated.","The orthogonality assumption could be tested directly: measuring per-token gradient norms across a real policy should show whether $\\|\\nabla_\\theta \\log \\pi_\\theta(y|x)\\|^2$ is actually proportional to $l_y$; if not, the theoretical grounding shifts to the empirical recipe.","The baseline's variance reduction should also apply to token-level or step-level rewards; a straightforward extension would give a step-wise length-weighted baseline for process supervision.","Because the baseline uses only within-prompt samples, its effectiveness may depend on group size $K$; small $K$ raises baseline estimation variance, and one could quantify the $K$-dependence."],"forward_implications":["LLM alignment can be done with a single policy model and no auxiliary value or reference models, removing a major source of memory and compute overhead.","The length-weighted baseline can replace group-mean or greedy baselines in other on-policy algorithms such as Reinforce++, and the paper shows preliminary gains in training reward and entropy.","Without KL or entropy regularization, training remains stable, which simplifies hyperparameter tuning and removes the need to balance those penalties.","On-policy training alone raises pass@1 on math benchmarks while keeping KL divergence to the initial policy low, indicating less alignment tax.","Output diversity improves, as measured by Self-BLEU and Rep-5, which matters for user-facing sampling quality."],"supporting_citations":[{"why":"Establishes the theoretical optimal reward baseline that minimizes gradient variance, the starting point for the paper's derivation.","marker":"[GBB04]"},{"why":"Also derives the optimal reward baseline for gradient-based reinforcement learning, cited alongside GBB04 as the source of Equation 9.","marker":"[WT01]"},{"why":"Defines GRPO, the group-relative baseline algorithm that OPO compares against and whose setup (K samples per prompt) OPO adopts.","marker":"[SWZ+24]"},{"why":"Provides the rule-based reward function and the RL training recipe for mathematical reasoning that OPO uses.","marker":"[GYZ+25]"},{"why":"Defines Reinforce++, the algorithm to which OPO's optimal baseline is applied in the appendix.","marker":"[Hu25]"},{"why":"Introduces the REINFORCE policy-gradient estimator whose variance the optimal baseline is designed to reduce.","marker":"[Wil92]"},{"why":"Introduces PPO and the clipped surrogate objective that contextualizes the on-policy vs off-policy discussion.","marker":"[SWD+17]"}],"fun_headline_variants":["On-policy RL ditches value model, uses length-weighted baseline","Variance-optimal baseline stabilizes on-policy LLM RL","OPO: stable on-policy RL without KL or entropy terms","Optimal baseline enables on-policy RL that beats GRPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the length-weighted baseline is the optimal variance reducer rests on the assumption that token-level gradients are approximately orthogonal and have identically distributed norms; if real language-model gradients violate this, Equation 12 is no longer variance-minimizing and OPO's theoretical advantage over simpler baselines is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["On-policy RL ditches value model, uses length-weighted baseline","Variance-optimal baseline stabilizes on-policy LLM RL","OPO: stable on-policy RL without KL or entropy terms","Optimal baseline enables on-policy RL that beats GRPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3558,"prompt_tokens":908,"completion_tokens":2650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":2576}},"tokens_in":524,"tokens_out":2650,"duration_ms":17939,"temperature":1.0,"reasoning_tokens":2576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:43:04.730950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, for an actual policy being trained, the per-token squared gradient norms across a sample of responses; if the norm is not approximately proportional to response length, the derivation of $b^*(x)$ fails. Alternatively, compare the empirical variance of policy-gradient estimates under the length-weighted baseline versus the group-mean baseline on a fixed set of prompts; if the length-weighted baseline does not reduce variance, the paper's central mechanism is not doing the claimed work.","supporting_citations":[],"review_version":1}