Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator

Haoyu Han; Heng Yang

REVIEW 3 major objections 4 minor 3 references

Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The noise-to-signal ratio of the REINFORCE gradient estimator is exactly computable for LQG and polynomial systems, and it blows up as policies approach optimality.

desk verdict The exact variance/NSR computation for LQG and polynomial systems is real and likely correct, but the abstract's exponential-in-horizon NSR claim is not proven by the supplied theorems. read the letter →

arxiv 2602.01460 v3 pith:KJW4776S submitted 2026-02-01 math.OC cs.LG

Haoyu Han , Heng Yang This is my paper · ORCID

classification math.OCcs.LG

keywords REINFORCEpolicygradientnoise-to-signalratiovariancelinear-quadraticregulatorGaussianIsserlistheoremexactmoments

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that the noise-to-signal ratio (NSR) of the REINFORCE policy-gradient estimator—the estimator's variance divided by the squared norm of the true gradient—is not a benign constant but varies sharply across policy parameters and typically worsens as training approaches an optimum. For finite-horizon linear-quadratic systems with Gaussian linear policies, and for polynomial systems with polynomial feedback, the paper proves the NSR can be computed exactly, either in closed form or through an explicit Gaussian-moment algorithm, with no approximation. For general nonlinear dynamics it gives upper bounds on the estimator variance. The exact formulas reveal scaling laws: the NSR grows as the policy becomes more deterministic and as the initial-state distribution broadens, and it can grow exponentially with horizon when the closed-loop dynamics are unstable. Because the optimal policy is deterministic, REINFORCE-style updates become increasingly noisy exactly where they need to converge, offering a concrete mechanism for training slowdowns, oscillations, and policy collapse.

What carries the argument

The load-bearing mechanism is exact Gaussian-moment evaluation: for centered Gaussian vectors, moments of products of quadratic forms are computed by Wick/Isserlis contractions, captured in the paper's IS_Omega shorthand (Lemma 2). For multi-step LQG, Theorem 4 lifts the T-step system into a single-step system with block lower-triangular matrices, so the return becomes one quadratic form in the initial state and one in the stacked exploration noise; parity bookkeeping (only even-degree terms survive in centered Gaussian expectations) reduces the variance expansion to sums of trace terms. For polynomial systems, Proposition 9 shows that the gradient estimator is itself a polynomial in Gaussia

What would settle it

Run the one-step double-integrator experiment with sigma_0 >> sigma and measure NSR by Monte Carlo for several ratios; Eq. (8) predicts linear growth in sigma_0^2/sigma^2, and any statistically significant deviation from the closed-form value would falsify the exact-moment machinery.

Watch

Extended reading notes

Core claim

The paper's central claim is that the noise-to-signal ratio (NSR) of the REINFORCE estimator is exactly characterizable for two structured classes of control problems, and that this exactness exposes a non-uniform, often diverging NSR landscape. For LQG systems with Gaussian linear policies, Theorems 3 and 5 express the mean gradient and the estimator's second moment as explicit Gaussian-moment functions (Wick/Isserlis); for polynomial dynamics with polynomial feedback, Proposition 9 shows the estimator is a polynomial in Gaussian variables, so all needed moments are exactly computable. In both settings the NSR is obtained without Monte Carlo approximation, scaling inversely with policy cova

Load-bearing premise

The exact formulas require the initial state and exploration noise to be centered Gaussian variables; if either is non-Gaussian or has a nonzero mean, or if the true gradient vanishes, the exact NSR identities do not apply.

Editorial extensions

If this is right

NSR diverges as the policy covariance goes to zero, so any REINFORCE-style update faces an exploration–exploitation tension near deterministic optimal policies.
For unstable closed-loop dynamics, estimator variance can grow exponentially with horizon, making long-horizon tasks with unstable controllers especially noisy.
The exact formulas allow computing NSR along an optimization path without Monte Carlo rollouts, turning a statistical diagnostic into a deterministic one.
The upper bounds for nonlinear systems show variance scales with the policy Jacobian, so policies with sharp state-to-action changes can have worse gradient estimates.
Gradient descent on the deterministic objective converges, while SGD and Adam with the same policy can oscillate or collapse once the NSR is large.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper leaves implicit: because the lifted dynamics reduce multi-step LQG to one-step form, the same exact-moment construction should apply to REINFORCE with a state-dependent baseline, subtracting a function of state and changing only the polynomial degree of the estimator.
The NSR formulas amount to a deterministic diagnostic: monitoring the exact expression along a run could decide when to lower the learning rate or raise exploration covariance, a use the paper does not explore.
For neural policies, Theorem 10 suggests that penalizing the Jacobian norm of the policy mean (or otherwise limiting sensitivity) would directly shrink the variance bound; this is not tested here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The new thing here is genuinely new: an exact finite-horizon computation of the REINFORCE estimator's variance and NSR for LQG and for polynomial systems with Gaussian policies, via lifting plus Gaussian moment evaluation. The derivations look careful and internally consistent; the parity bookkeeping in the multi-step decomposition checks out, and the mean-gradient formulas (15)-(16) are cleanly derived. This is a real advance over the prior O(σ^-2) bounds in Zhao et al. and the LQG upper bounds in Preiss et al. The polynomial extension and the nonlinear upper bound are sensible. Credit where due: the core moment machinery is serious and appears sound under the stated centered-Gaussian assumptions.

The soft spots are in the claims, not the core machinery. The stress-test note is right: the paper asserts in the abstract and contributions that NSR grows exponentially with horizon for unstable closed-loop systems, but Theorem 7 bounds variance, Theorem 8 bounds a state-map norm, and neither bounds the denominator ||∇J||^2. For unstable F, the exact gradient formulas themselves involve P_t and Λ_t that grow exponentially, so the ratio's behavior is not established. Figure 3 shows variance growth, not NSR growth. That is a gap in a headline claim, and the paper should either prove the NSR statement or reframe it as a variance-scaling result. Minor related issue: the Θ analysis in §2.2 is fine for the σ0/σ scaling but suppresses T and F dependence, so it cannot support the horizon claim either.

The other soft spots are proportionally mild. The empirical sections are illustrative: no code, no error bars, and the 'policy collapse' language outruns the evidence. The centered-Gaussian and nonzero-gradient assumptions are restrictive but stated, and the deterministic-optimum degeneracy is handled by defining NSR only where the gradient is nonzero—that's fair, though the 'blow-up near optimality' statements are about approach, not attainment.

Who should read this: anyone working on policy-gradient variance, exploration-exploitation tradeoffs, or LQG policy optimization. It deserves a serious referee, but the referee should push on the horizon-scaling claim and request code and error bars. I'd recommend peer review with major revision, and I'd cite it for the exact moment evaluation even while flagging the overclaim.

Referee Report

3 major / 4 minor

Summary. The paper studies the variance and noise-to-signal ratio (NSR) of the REINFORCE policy-gradient estimator in finite-horizon control problems. For linear-quadratic systems with linear-Gaussian policies, it derives exact expressions for the mean and second moment of the estimator via Gaussian Isserlis/Wick identities and a lifted-state representation (Theorems 3 and 5), with mean-gradient formulas in Eqs. (15)-(16). It extends exact Gaussian-moment evaluation to polynomial dynamics with polynomial feedback (Proposition 9) and gives variance upper bounds for general nonlinear systems (Theorem 10). Numerical experiments on LQG, polynomial, and MLP-policy systems show NSR increasing near optima and occasional policy collapse. The abstract and contributions also claim that NSR grows exponentially with horizon for unstable closed-loop dynamics.

Significance. If the exact-moment theorems hold, this is a useful contribution to policy-gradient variance analysis: it provides non-asymptotic, parameter-free characterizations (no fitted constants) for LQG and polynomial systems, and it identifies the initial-state covariance/policy-covariance scaling. The explicit gradient formulas and the lifted decomposition are valuable. The practical claim that NSR is highly non-uniform and worsens near optima is supported by the exact formulas in the examples. However, the exponential-horizon NSR claim is not established by the supplied bounds, and this weakens the paper's headline.

major comments (3)

[§2.2, Theorems 7–8] The contribution list and Section 5 state that 'we prove the NSR can grow exponentially with the horizon when the closed-loop dynamics are unstable.' This is not supported by the proof chain. Theorem 7 gives upper bounds on Var_Fro(bG_K) and Var_Fro(bG_l); Theorem 8 bounds ||F_S||. NSR is Var_Fro(bG)/||∇J||^2. For ρ(F)>1, the true gradient in Eq. (15) contains P_t and Λ_{t+1}, both of which grow exponentially with T, so the denominator also grows exponentially. The Θ analysis in §2.2 absorbs all T- and F-dependence, so it cannot justify the horizon-growth statement. Figure 3 plots variance, not NSR. To retain the claim, the paper needs a lower bound on the variance and an upper bound on ||∇J||^2 in the same regime, or the claim should be downgraded to variance growth.
[§2.2, Analysis of NSR after Theorem 7] The scaling statement '∥E[bGK]∥^2_F, ∥E[bGℓ]∥^2_2 = Θ((σ^2+σ0^2)^2) if they are nonzero' is not correct for the log-std gradient: Eq. (16) gives ∇ℓJ = -2 diag(Σ Σ γ^t(...)), whose squared norm is Θ(σ^4), independent of σ0. The subsequent NSR bound uses a denominator that omits this separate σ^4 contribution. The α→∞ conclusion may survive, but the Θ justification as written is wrong and needs to be rewritten. Moreover, non-degeneracy conditions for the gradients being nonzero are not stated, and NSR is undefined at the deterministic optimum (Corollary 6).
[§2.2, Figure 3] Figure 3 and the sentence 'This verifies the prediction of Theorem 7 and 8. Thus, in long-horizon tasks, closed-loop instability can dramatically worsen the NSR' conflate variance with NSR. The figure shows true variance growth with T; it does not show NSR. If the gradient also grows exponentially, the figure is consistent with NSR remaining bounded. The authors should either plot NSR for these systems using the exact Theorem 5 procedure, or change the interpretation to variance growth.

minor comments (4)

[Theorem 5] The theorem statement refers to equations in the proof for the exact decompositions. For a theorem claiming exact computation, these formulas should be restated as a standalone algorithm or proposition so the reader can verify them without reading the proof.
[Theorem 10(ii)] The phrase 'where C is a constant' is unnecessarily vague; the proof gives C = 2T^2 sqrt(m(m+14)). State this explicitly.
[Theorem 7] The proof bounds E[||bG||^2], not the variance itself. The inequality chain Var ≤ E[||bG||^2] is valid, but the theorem should make clear that it is a second-moment bound, which is why the derived NSR bound is only an upper bound.
[Throughout] Notation is occasionally overloaded, e.g., the block covariance is written as Σ := I_T ⊗ Σ while Σ also denotes the per-step policy covariance. A bar or subscript would improve clarity. There are also minor typos such as 'F+S ,∈R^{nT×n}' in Theorem 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NSR formulas are derived from first principles, not fitted; the self-citations are contextual.

full rationale

I walked the derivation chain. The central claims (Theorems 3, 5, Proposition 9) compute the score-function gradients and the second moments directly: the gradients come from differentiating the expected return under the linear-Gaussian policy (Eqs. (6)-(7) and (15)-(16)), and the second moments come from expanding the REINFORCE estimator, using the lifted quadratic return decomposition R = -(x+2y+z) (Eqs. (10)-(14)), and applying Isserlis/Wick moment evaluation (Lemma 2, proved in Appendix C). No constant or parameter is fitted to data, and no target quantity (variance or NSR) is assumed. NSR is then assembled by its definition Var/||∇J||^2 from two independently derived pieces. The σ,σ0 scaling laws, including the σ→0 blow-up in the one-step case, follow algebraically from the E1,K term Θ(σ0^6/σ^2) divided by the gradient norm Θ(σ0^4); Corollary 6 follows from Eq. (16) with Qa≻0. Proposition 9 is a direct consequence of the fact that polynomial maps of Gaussians remain polynomial and Isserlis evaluates the resulting monomial moments; citing Magnus for this classical fact is not load-bearing. The self-citations (Han & Yang 2024; Muppidi et al. 2024) appear only in contextual remarks and do not carry the argument. I also checked the paper's own stated limitations: exactness is explicitly conditioned on centered Gaussian s0 and εt, and NSR is only reported away from stationary points; these are scope conditions, not circular inputs. The only substantive concern is a non-circular proof gap: the headline that NSR grows exponentially in T for unstable closed-loop systems is not fully established by Theorem 7, which upper-bounds only the variance, while the gradient denominator in NSR may itself grow with T through Pt and Λt. That is an over-claim or missing argument, but it is not a reduction of a prediction to its inputs, so it does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No new entities or fitted constants; the paper's contribution is exact computation under standard structural assumptions. All parameters (Σ, Σ0, Qs, Qa, A, B) are problem data, not fitted to make the results work.

assumptions (6)

standard math Isserlis/Wick theorem for Gaussian moments
Used in Lemma 2 and throughout Theorem 5/Proposition 9 to evaluate products of quadratic forms under centered Gaussian distributions.
standard math Policy gradient lemma / score-function identity ∇θJ=E[∇θ log π R]
Basis for the REINFORCE estimator in Eq. (1); not proved in the paper.
domain assumption Centered Gaussian initial state and exploration noise
s0∼N(0,Σ0), εt∼N(0,Σ) independent; needed for exact moment evaluation and odd-moment cancellation. If violated, exactness fails.
domain assumption Finite-horizon, discounted MDP with rewards r(s_{t+1},a_t)
Problem setup used to derive the LQG lifted return decomposition in Theorem 4.
domain assumption Q_a≻0 for Corollary 6 deterministic optimum
Needed for ∇ℓJ=0 iff Σ=0; if Q_a is only PSD, stationary points may exist with nonzero covariance.
domain assumption Finite fourth moments for Theorem 10
Assumed to make the Cauchy-Schwarz bounds finite; stated in Theorem 10.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator." pith.science (2026). https://pith.science/paper/KJW4776S

@misc{pith2026260201460,
  author       = {Pith},
  title        = {Pith review of: Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJW4776S}},
  note         = {Machine review of arXiv:2602.01460}
}

read the original abstract

Policy-gradient methods are widely used in reinforcement learning, yet training often becomes unstable or slows down as learning progresses. We study this phenomenon through the noise-to-signal ratio (NSR) of a policy-gradient estimator, defined as the estimator variance (noise) normalized by the squared norm of the true gradient (signal). Our main result is that, for (i) finite-horizon linear systems with Gaussian policies and linear state-feedback, and (ii) finite-horizon polynomial systems with Gaussian policies and polynomial feedback, the NSR of the REINFORCE estimator can be characterized exactly-either in closed form or via numerical moment-evaluation algorithms-without approximation. For general nonlinear dynamics and expressive policies (including neural policies), we further derive a general upper bound on the variance. These characterizations enable a direct examination of how NSR varies across policy parameters and how it evolves along optimization trajectories (e.g. SGD and Adam). Across a range of examples, we find that the NSR landscape is highly non-uniform and typically increases as the policy approaches an optimum; in some regimes it blows up, which can trigger training instability and policy collapse.

Figures

Figures reproduced from arXiv: 2602.01460 by the authors.

**Figure 1.** NSR of the REINFORCE estimator in one-step LQG with isotropic Σ = σ 2 I and Σ0 = σ 2 0I for double integrator (9). Moreover, by the AM–GM inequality, we have 0 ≤ σ 2σ 2 0 ≤ (σ 4 + σ 4 0 )/2, hence σ 2σ 2 0 + σ 4 0 + σ 4 σ 4 0 + σ 4 = Θ(1), and therefore NSR=Θ (σ 6 0/σ2 ) σ 4 0 + σ 4 + 1 −1=Θ (σ0/σ) 6 1 + (σ0/σ) 4 + 1 −1. Let α := σ 2 0/σ2 . Then α 3/(1 + α 2 ) is large only when α is large, i.e., when σ0 ≫ σ. In… view at source ↗

**Figure 3.** illustrates how the true variance VarFro(Gb) (computed exactly from Theorem 5) scales with the horizon T across four systems: it remains bounded when ρ(F) < 1, grows roughly polynomially when ρ(F) = 1, and grows exponentially when ρ(F) > 1. This verifies the prediction of Theorem 7 and 8. Thus, in long-horizon tasks, closed-loop instability can dramatically worsen the NSR. 0 5 10 15 20 25 horizon 10 4 10 5 10 6 10… view at source ↗

**Figure 2.** NSR and objective along optimization trajectories on a double-integrator system with T = 30. Top: optimizer trajectories overlaid on the NSR landscape. Bottom: learning curves (objective vs. iteration). The three optimizers (GD, SGD, Adam) start from the same initial policy (square), move toward the optimal policy (star), and terminate at triangles. The NSR increases markedly as the policy approaches optimality, con… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: NSR (bottom) and objective (top) during training on the quadratic system. As the policy oscillates near the optimum, a single inaccurate gradient step triggers policy collapse. 3. Polynomial Systems, Polynomial Feedback We then consider polynomial dynamics with polynom…

**Figure 5.** Figure 5: NSR and objective along optimization trajectories on polynomial systems. (a): quadratic system; (b): cubic system. Top: optimizer trajectories and NSR landscape; Bottom: learning curves. The NSR increases when approaching the optimal policy same as linear systems. GD c…

**Figure 6.** Figure 6: Trajectory of NSR and objective for LQR with MLP policy (left) and Pendulum with MLP policy (right). Both the NSR and the objective are calculated through Monte-carlo evaluation. The NSR increases when approaching the optimal policy, and the red square shows the synchr…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 linked inside Pith

[2]

Plugging into (68) gives E[∥ bGℓ∥2 2]≤TE h (2∥Σ−1/2 ¯ε∥4 2+2mT)R(τ) 2 i

Therefore, T−1X t=0 ∥gt∥2 2≤2∥ Σ−1/2 ¯ε∥4 2+2mT. Plugging into (68) gives E[∥ bGℓ∥2 2]≤TE h (2∥Σ−1/2 ¯ε∥4 2+2mT)R(τ) 2 i . Assume the standard multi-step quadratic decomposition of the return R(τ) =−(x+ 2y+z), x:=s ⊤ 0 M sss0, y:=s ⊤ 0 M se ¯ε, z:= ¯ε⊤M ee ¯ε, Proof ofK-part expansion.Define S:=F ⊤ S FS, E:=F ⊤ E FE, Step 1: expand(⋆).Since∥F Ss0∥2 2=s ⊤ ...
[666]

policy collapse

PMLR, 2024. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that mat- ters. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. A closer look at deep policy gradients.arXiv prepr...

arXiv 2024
[1978]

Our setting involves a non-homogeneous polynomial, hence we state the following lemma

derives formulas for expectations of products of quadratic forms of the type E hQN i=1 (u⊤Aiu) i . Our setting involves a non-homogeneous polynomial, hence we state the following lemma. Lemma 11.Forξ∼ N(0, Im), define w(ξ) := mX i=1 (ξ2 i −1) 2. Then for anyu∈R m and anyM∈S m, E[w(ξ)] = 2m,(26) E[w(ξ) (ξ⊤u)2] = (2m+ 8)∥u∥ 2,(27) E[w(ξ) (ξ⊤M ξ)] = (2m+ 8) ...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Plugging into (68) gives E[∥ bGℓ∥2 2]≤TE h (2∥Σ−1/2 ¯ε∥4 2+2mT)R(τ) 2 i

Therefore, T−1X t=0 ∥gt∥2 2≤2∥ Σ−1/2 ¯ε∥4 2+2mT. Plugging into (68) gives E[∥ bGℓ∥2 2]≤TE h (2∥Σ−1/2 ¯ε∥4 2+2mT)R(τ) 2 i . Assume the standard multi-step quadratic decomposition of the return R(τ) =−(x+ 2y+z), x:=s ⊤ 0 M sss0, y:=s ⊤ 0 M se ¯ε, z:= ¯ε⊤M ee ¯ε, Proof ofK-part expansion.Define S:=F ⊤ S FS, E:=F ⊤ E FE, Step 1: expand(⋆).Since∥F Ss0∥2 2=s ⊤ ...

[2] [666]

policy collapse

PMLR, 2024. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that mat- ters. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. A closer look at deep policy gradients.arXiv prepr...

arXiv 2024

[3] [1978]

Our setting involves a non-homogeneous polynomial, hence we state the following lemma

derives formulas for expectations of products of quadratic forms of the type E hQN i=1 (u⊤Aiu) i . Our setting involves a non-homogeneous polynomial, hence we state the following lemma. Lemma 11.Forξ∼ N(0, Im), define w(ξ) := mX i=1 (ξ2 i −1) 2. Then for anyu∈R m and anyM∈S m, E[w(ξ)] = 2m,(26) E[w(ξ) (ξ⊤u)2] = (2m+ 8)∥u∥ 2,(27) E[w(ξ) (ξ⊤M ξ)] = (2m+ 8) ...