{"id":"608ebdae-90d7-41ef-8623-0dadc721da62","arxiv_id":"2506.02864","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"BNPO dynamically normalizes binary rewards using a Beta distribution with parameters adapted from the current batch, claiming reduced gradient variance and state-of-the-art math reasoning performance.","lead":"BNPO is a new way to train reasoning language models: it normalizes rewards with a Beta distribution whose parameters adapt as training progresses. The paper claims this lowers gradient variance and outperforms REINFORCE, ReMax, GRPO, and REINFORCE++ on math benchmarks, though the reported gains are small.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's variance minimization is invalid: the proof drops the squared score-function norm from Var(g), so the claimed optimal α=1+a/3, β=1+b/3 is not established.","rationale":"I independently re-derived the proof of Theorem 1. The stated assumption is used only to eliminate the squared-mean term in Var(g); it cannot justify replacing E[(∇logπ)^2Z^2] by E[Z^2]. The proof then integrates over p(q) using E_o[(R−p)^2|q] = p(1−p), which would be correct only if E[(∇logπ)^2|q,o] were constant in o. In the paper's own Bernoulli model, different outputs o have different log-probabilities and hence different score norms; the conditional second moment is p(1−p)[(∇logπ|R=1)^2(1−p) + (∇logπ|R=0)^2p]/f_N^2. The claimed optimum (1+a/3, 1+b/3) depends only on the Beta prior, not on the policy's score structure, so it cannot minimize the true variance in general. My proposed numerical experiment on a logistic two-action policy directly computes the true variance and checks the minimizer; I expect it to differ from (4/3,4/3), which would refute the central claim. The empirical results are secondary: the headline SOTA claim has small deltas and no significance testing, but even if the experiments were clean, the theoretical foundation is invalid. I credit the authors for a clear write-up, a concrete algorithm, and an explicit assumption; the issue is not a missing edge case but a central algebraic step that does not follow. The reader's REJECT is therefore appropriate and unchanged.","tokens_in":13133,"tokens_out":7659,"duration_ms":74714,"concrete_test":"Implement the exact variance for a single-question, two-action softmax policy. Let p(q) ~ Beta(a,b) with a=b=1, and let the score for the correct output (R=1) be 1−p and for the incorrect output (R=0) be −p, as in a logistic policy. Compute V(α,β) = E_p[ p(1−p)((1−p)^3 + p^3) / f_N(p;α,β)^2 ] and numerically minimize over α,β>0. If the minimizer differs from (4/3,4/3), the paper's claimed optimum is refuted under its own setup. To confirm the failure is general, repeat with a=2,b=1 and with asymmetric score magnitudes (e.g., scores c(1−p) and −d·p for c≠d); if the minimizer shifts, the theorem's conclusion does not follow.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the proof of Theorem 1 (Appendix A), the authors write Var(gα,β) = E[(∇θlogπ(o|q))^2 · ((R−p)/f_N)^2] − (E[∇θlogπ(o|q)(R−p)/f_N])^2, then use the uncorrelatedness assumption only to zero the second term. They then assert that the variance is proportional to E[(R−p)^2/f_N^2] and substitute E_o[(R−p)^2|q] = p(q)(1−p(q)). This step is invalid: uncorrelatedness of X and Y does not imply E[X^2Y^2] = E[X^2]E[Y^2], and no assumption makes the squared score-function norm constant in o. The exact conditional second moment is E_o[(∇θlogπ)^2(R−p)^2/f_N^2 | q], which depends on how the score magnitude varies with the reward outcome. In a simple logistic two-action policy with R=1 for one action and R=0 for the other, this conditional expectation equals p(1−p)[(1−p)^3 + p^3]/f_N^2 up to a positive factor, not a constant multiple of p(1−p)/f_N^2. Minimizing the true expression generally yields parameters different from (1+a/3, 1+b/3), which depend only on the Beta prior (a,b) and ignore the policy's gradient structure. Moreover, the stated uncorrelatedness assumption is not merely unjustified: for a useful policy-gradient estimator, E[∇θlogπ (R−p)/f_N] must be a nonzero multiple of the true gradient, not zero as assumed. Thus the central variance-reduction theorem rests on an algebraic step that does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BNPO (Beta Normalization Policy Optimization), a modification of policy-gradient RL for LLM reasoning with binary rewards. The method defines the advantage as A(q,o)=(R(q,o)-p(q))/f_N(p(q);alpha,beta), where p(q) is the expected reward for question q and f_N is a Beta density with parameters adapted during training. The central theoretical claim (Theorem 1) is that, under an uncorrelatedness assumption, the variance of the BNPO gradient estimator is minimized uniquely at alpha=1+a/3 and beta=1+b/3, where (a,b) are the parameters of a Beta distribution modeling p(q). The paper also claims BNPO generalizes REINFORCE with baseline and GRPO in the binary-reward setting, introduces an advantage-decomposition extension for multiple binary rewards, and reports experiments on MATH training with Qwen2.5-Math-1.5B/7B showing the highest average pass@1 among compared methods.","tokens_in":13532,"tokens_out":6694,"duration_ms":75491,"significance":"If Theorem 1 were correct, BNPO would be a simple, principled upgrade to GRPO with an adaptive normalization mechanism, and the public code would facilitate reproducibility. The paper also makes a falsifiable empirical claim of state-of-the-art performance on reasoning benchmarks. However, the theoretical derivation contains a load-bearing algebraic error, and the empirical evidence is too weak to support the state-of-the-art claim on its own. The contribution therefore does not currently meet the standard for publication in a serious venue.","major_comments":[{"comment":"The variance expansion is correct up to Var(g)=E[(grad log pi)^2 ((R-p)/f_N)^2] - E[grad log pi (R-p)/f_N]^2, but the next sentence, \"Under the assumption, the variance ... is proportional to E[(R-p)^2/f_N^2]\", is invalid. Uncorrelatedness of X and Y (E[XY]=0) does not imply E[X^2Y^2]=E[X^2]E[Y^2]; the latter requires independence or an explicit condition on the conditional second moment. The exact quantity is E_o[(grad log pi)^2 (R-p)^2/f_N^2 | q], which generally depends on how the score-function norm varies with the reward outcome. For example, in a two-action logistic policy with R=1 for the correct action and R=0 otherwise, this conditional expectation is p(1-p)[(1-p)^3+p^3]/f_N^2 times a positive constant, not a constant multiple of p(1-p)/f_N^2. Therefore the claimed minimum at alpha=1+a/3, beta=1+b/3 does not follow.","section":"Appendix A, Step 1 (Variance Expression)"},{"comment":"The standing assumption that grad log pi(o|q) is uncorrelated with (R-p(q))/f_N(p(q);alpha,beta) is self-defeating for a policy-gradient estimator. Under this assumption E[g_alpha,beta]=0, whereas a useful estimator should have expectation equal to the policy gradient (or a non-zero weighted gradient). Thus the proof uses an assumption that is generally false for exactly the setting the method is meant to address. This is not a minor technicality; it is what allows the squared-mean term to be dropped, and it cannot be repaired by adding a short derivation within the current framework.","section":"Theorem 1"},{"comment":"The abstract's claim that BNPO \"achieves state-of-the-art performance among policy optimization methods\" is not supported by the reported numbers. On Qwen2.5-Math-1.5B, BNPO's average is 39.4 versus 39.0 for REINFORCE and GRPO; on Qwen2.5-Math-7B, BNPO's average is 47.8 versus 47.6 for ReMax, while on MATH500 BNPO is below GRPO (77.0 versus 78.6). These differences are within the range of typical run-to-run variation, and no error bars, multiple seeds, or significance tests are reported. The empirical evidence is therefore consistent with BNPO being equivalent to existing methods rather than superior.","section":"Section 5.2, Table 1"},{"comment":"The theorem's conclusion depends on the modeling assumption that p(q) follows a Beta distribution f_D(p;a,b), but this assumption is not validated against the data. The method-of-moments estimators for (a,b) are computed from a finite batch of Monte Carlo estimates of p(q), and the optimality of alpha=1+a/3 and beta=1+b/3 is only derived under this unverified parametric assumption. If the distribution of p(q) deviates from Beta, the variance expression and the claimed minimizer both change; the paper provides no diagnostic or robustness check for this assumption.","section":"Section 3, parameter estimation and Eq. (13)-(14)"}],"minor_comments":[{"comment":"The reduction to GRPO is stated as an equality involving the population quantities p(q) and sqrt(p(q)(1-p(q))), but actual GRPO uses sample mean and sample standard deviation over a finite group. The claim that BNPO \"reduces to GRPO\" therefore holds only in the infinite-sample limit, and the finite-sample equivalence in Eq. (16) should be qualified.","section":"Eq. (16)"},{"comment":"The text contains a typo: \"Unper p ~ f_D(p(q);a,b)\" should read \"Under p ~ f_D(p(q);a,b)\".","section":"Appendix A, Step 1"},{"comment":"The phrase \"REINFROCE with baseline\" contains a spelling error; it should be \"REINFORCE with baseline\".","section":"Section 3, after Eq. (15)"},{"comment":"The sentence \"Extending the theory in this way is a area of ongoing exploration\" contains a grammatical error; it should be \"is an area of ongoing exploration\".","section":"Section 6, Limitations"},{"comment":"The sentence \"We also that BNPO can reduces to REINFORCE with baseline and GRPO\" is missing a verb and should be corrected to \"We also show that BNPO can reduce to ...\".","section":"Conclusion"},{"comment":"The caption says the figure presents the evolution of (E[p(q)], Var[p(q)], alpha, beta), but the single y-axis labeled \"Value\" does not distinguish these four quantities; a legend or separate panels would improve readability.","section":"Figure 3"}],"recommendation":"reject","confidential_remarks":"The paper's main theoretical contribution is Theorem 1, and the proof of that theorem contains an algebraic step that does not follow, together with a self-defeating uncorrelatedness assumption. The empirical results are close to parity with baselines and do not provide independent support for the method. I see no way to repair the central claim within the scope of a revision; a substantially different theoretical argument would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the BNPO paper. The core idea is genuinely new: normalize binary rewards by a Beta density whose shape parameters are recomputed from the estimated moments of per-question success probabilities, and show that with fixed parameters this reduces to REINFORCE and GRPO. The writing is clear, and the reduction arguments in Section 3 are correct and useful. Credit where due: the adaptive scheme is not in the cited RLOO/GRPO/REINFORCE++ line.\n\nBut the load-bearing theory does not hold up. Theorem 1 claims Var(g) is minimized at alpha=1+a/3, beta=1+b/3, and the proof in Appendix A collapses to E[((R-p)/f_N)^2] after using an uncorrelatedness assumption to drop the mean cross-term. That step is wrong: uncorrelatedness of X and Y does not imply E[X^2 Y^2] = E[X^2]E[Y^2], and nothing makes the squared score-function norm constant in the output. In a simple two-action logistic policy, the true conditional second moment is p(1-p)[(1-p)^3 + p^3]/f_N^2, not a constant times p(1-p)/f_N^2. Minimizing that expression gives different parameters that depend on the policy's gradient structure. Worse, the uncorrelatedness assumption itself is incompatible with a useful policy-gradient estimator: E[grad_log_pi (R-p)] equals the true gradient, not zero, so assuming zero correlation would make the estimator biased. The theorem's conclusion therefore does not follow.\n\nThe experiments are extensive but the reported gains are small (0.2-0.4 average points over several datasets) with no error bars or significance tests. The gradient-norm stability plot is suggestive but is a proxy, not a direct variance measurement. The method-of-moments estimation of a,b from the same batch used for advantages is no worse than GRPO's own batch normalization, so that is not a new burden. The Beta modeling assumption for p(q) is a prior, not validated, but that alone would be minor if the theorem were correct.\n\nWho is this for? Researchers working on variance reduction in binary-reward RL for LLMs. The idea is worth thinking about, and the reduction to existing methods is a nice framing. But as it stands, the central claim is unproven and the empirical evidence is too weak to carry the paper. I would tell the authors to fix the proof or remove the variance-minimization claim, and to add statistical support to the experiments. A serious referee could help them do that; I would rather see this in review than have it linger as an arXiv-only preprint with a false theorem.","headline":"The adaptive Beta-normalization idea is novel and clearly written, but the proof of the central variance-reduction theorem drops the score-function norm, making the claimed optimal parameters unsupported.","tokens_in":14018,"tokens_out":3198,"would_cite":false,"duration_ms":31282,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Beta-shaped adaptive normalizer provably cuts policy-gradient variance.","keywords":["reinforcement learning","policy optimization","Beta distribution","variance reduction","reward normalization","reasoning tasks","LLM alignment","group relative policy optimization"],"falsifier":"On a question with a small enough action space to enumerate all outputs, compute the exact gradient-estimator variance for a grid of $(\\alpha,\\beta)$ values, using the true score function rather than the uncorrelatedness assumption; if the measured minimum is not at $(1+a/3, 1+b/3)$, the theorem's practical conclusion is falsified.","tokens_in":12938,"feed_emoji":"🎯","tokens_out":7541,"duration_ms":62660,"temperature":0.7,"pith_summary":"BNPO is a policy-gradient method for training reasoning models on binary rule-based rewards. Its central claim is that using an advantage normalized by a Beta density whose parameters adapt to the current reward-expectation distribution lowers gradient-estimator variance, with the unique minimum at α = 1 + a/3 and β = 1 + b/3. The method reproduces REINFORCE-with-baseline at (α, β) = (1, 1) and GRPO at (3/2, 3/2), so the dynamic adaptation is the entire difference. On Qwen2.5-Math models fine-tuned on MATH, BNPO posts the best average pass@1 on four math benchmarks and smoother gradient norms, which the paper takes as evidence of its variance-reducing property.","feed_headline":"Beta rule picks the variance-minimizing reward normalization","feed_subtitle":"A closed-form rule adapts reward scaling to the evolving policy and recovers REINFORCE and GRPO.","key_machinery":"The load-bearing object is the BNPO gradient estimator $g_{\\alpha,\\beta} = \\nabla_\\theta \\log \\pi(o|q)\\, (R(q,o)-p(q))/f_N(p(q); \\alpha, \\beta)$, combined with the variance identity that follows once the score-function norm is set aside. That identity expresses the variance as proportional to $B(\\alpha,\\beta)^2 \\cdot B(a+3-2\\alpha, b+3-2\\beta) / B(a,b)$, where $a$ and $b$ come from the assumed Beta distribution of per-question success probabilities. Minimizing this Beta-function product gives the parameter rule $\\alpha = 1 + a/3$, $\\beta = 1 + b/3$, and the proof uses trigamma monotonicity to show the Hessian is positive definite, making the minimum unique.","core_discovery":"BNPO defines the advantage $A_{\\alpha,\\beta}(q,o) = (R(q,o) - p(q))/f_N(p(q); \\alpha, \\beta)$, where $R$ is the binary outcome reward, $p(q)$ is the success probability of the current policy on question $q$, and $f_N$ is a Beta density used as the normalization term. The paper proves that when $p(q)$ follows a Beta$(a,b)$ distribution and the policy score function is uncorrelated with the normalized reward residual, the estimator's variance is finite iff $\\alpha < (a+3)/2$ and $\\beta < (b+3)/2$, and it attains a unique minimum at $\\alpha = 1 + a/3$, $\\beta = 1 + b/3$. The variance expression reduces to a product of Beta functions, $B(\\alpha,\\beta)^2 \\, B(a+3-2\\alpha, b+3-2\\beta)/B(a,b)$, and the convexity proof certifies uniqueness via trigamma inequalities. BNPO generalizes REINFORCE-with-baseline ($\\alpha=\\beta=1$) and GRPO ($\\alpha=\\beta=3/2$), and the paper reports state-of-the-art average pass@1 across MATH500, AMC23, AIME2024, and AIME2025 with two Qwen2.5-Math base models.","pith_inferences":["A natural stress test is to replace the Beta prior over $p(q)$ with a non-conjugate prior and see whether a similar closed-form optimum exists; the paper's Beta-function argument would not carry over directly.","Because the parameter rule only needs estimates of the mean and variance of $p(q)$, the same normalization could be applied to continuous rewards by treating each question's reward residual as a stand-in for $p(q)$, a step the paper leaves to future work.","The theorem's uncorrelatedness assumption could be checked empirically on a small policy; if the score norm correlates with the reward residual, the variance-minimum search would need to include the full score function to locate the true optimum.","If the measured variance reductions reproduce across model families, BNPO would be a drop-in replacement for GRPO in R1-style training, since it adds only moment estimation to the rollout loop and removes the need for a critic."],"forward_implications":["When the theorem's assumptions hold, the adaptive parameters $(1+a/3, 1+b/3)$ strictly dominate any fixed normalization, including GRPO's $(3/2,3/2)$, in variance of the policy-gradient estimate.","Because BNPO reduces to REINFORCE-with-baseline and to GRPO at specific $(\\alpha,\\beta)$ values, any improvement over those baselines is attributable to the dynamic adaptation of the normalization rather than a different objective.","The advantage-decomposition rule permits multiple binary rewards, such as accuracy and format, to be normalized separately; the paper reports small average gains when a format reward is added.","On the studied math benchmarks, BNPO achieves the best average pass@1 among the compared methods for both Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, with smoother gradient norms during training."],"supporting_citations":[{"why":"Supplies the rule-based binary-reward reasoning setting and the DeepSeek-R1 baseline that BNPO targets.","marker":"[Guo et al., 2025]"},{"why":"Defines GRPO, the static-normalization method BNPO generalizes and compares against.","marker":"[Shao et al., 2024]"},{"why":"Introduces RLOO/REINFORCE-with-baseline, which BNPO reproduces at (alpha,beta) = (1,1).","marker":"[Kool et al., 2019]"},{"why":"Provides the PPO clipping objective that BNPO uses as its policy update rule.","marker":"[Schulman et al., 2017]"},{"why":"Defines the original REINFORCE gradient whose variance the normalization is designed to reduce.","marker":"[Williams, 1992]"},{"why":"Supplies the MATH training set and MATH500 evaluation used in the experiments.","marker":"[Hendrycks et al., 2021]"},{"why":"Defines REINFORCE++, a comparison method that uses all-sample rewards.","marker":"[Hu, 2025]"},{"why":"Motivates the REINFORCE-with-baseline family used in Kimi K1.5, another R1-style system.","marker":"[Team et al., 2025]"}],"fun_headline_variants":["Beta normalization lowers variance in policy gradients","Adaptive Beta reward scaling stabilizes LLM reasoning","BNPO: Beta rule minimizes gradient variance in RL","Closed-form Beta normalization for variance-reduced RL","Beta-distributed rewards improve policy optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes the score function $\\nabla_\\theta \\log \\pi(o|q)$ is uncorrelated with the normalized reward residual $(R(q,o)-p(q))/f_N(p(q);\\alpha,\\beta)$, so the variance comparison treats the squared score norm as constant; if this correlation is not zero, the claimed optimal $(\\alpha,\\beta)$ may not be the variance-minimizing choice.","fun_headline_variants_meta":{"raw":{"variants":["Beta normalization lowers variance in policy gradients","Adaptive Beta reward scaling stabilizes LLM reasoning","BNPO: Beta rule minimizes gradient variance in RL","Closed-form Beta normalization for variance-reduced RL","Beta-distributed rewards improve policy optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1575,"prompt_tokens":1074,"completion_tokens":501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":690,"tokens_out":501,"duration_ms":5124,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:14:12.688219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a question with a small enough action space to enumerate all outputs, compute the exact gradient-estimator variance for a grid of $(\\alpha,\\beta)$ values, using the true score function rather than the uncorrelatedness assumption; if the measured minimum is not at $(1+a/3, 1+b/3)$, the theorem's practical conclusion is falsified.","supporting_citations":[{"cited_title":"Buy 4 REINFORCE samples, get a baseline for free!, 2019","cited_arxiv_id":null,"evidence_quote":"Introduces RLOO/REINFORCE-with-baseline, which BNPO reproduces at (alpha,beta) = (1,1)."}],"review_version":1}