{"id":"c3701769-9f66-4d81-b092-8554d6605b95","arxiv_id":"2506.08737","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding annealed Gaussian noise to rewards can help RL exploration, but this paper's proof of that claim is invalid and its SAC algorithm actually uses biased, non-zero-mean noise.","lead":"A simple RL exploration trick: add random noise to rewards during training and anneal it away. The paper reports empirical gains for PPO and SAC, but its theoretical proof is invalid and its SAC variant has a noise-annealing bug, so the central claims are not established.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3.4 is false as stated: output variance dominance does not imply trajectory variance dominance, so Theorems 3.5 and 3.6 do not establish RRP's exploration claim.","rationale":"The reader's weakest_assumption correctly identifies the transfer from one-step output variance to trajectory-level state variance as the unsupported step. My stress-test sharpens this into a falsification: Lemma 3.4's conclusion is not just underived, it is false. The counterexample is small and machine-checkable: g2 has larger output variance on the initial states than g1, but after one application it collapses all trajectories to a single state, so the summed trajectory variance is smaller over a sufficiently long horizon. This directly invalidates the lemma that both Theorem 3.5 (DQN) and Theorem 3.6 (A2C) rely on. Consequently the abstract's claim that adding zero-mean noise 'effectively enhances policy diversity... thereby expanding the range of exploration' is not supported by the paper's formal apparatus. I am not disputing that the empirical curves may reflect a real effect; rather, the advertised theoretical mechanism is broken at its core. Algorithm 1's non-zero-mean annealing is an additional implementation-level contradiction, but the false lemma is the more load-bearing issue because it undermines the stated theoretical contribution even for a perfectly zero-mean perturbation. No formal verification, code release, or parameter-free derivation offsets this: the proofs in Appendices A.2-A.4 reproduce the same circular step, and Eq. (14) is not a coherent state recursion. The verdict should therefore remain REJECT, consistent with the reader's assessment.","tokens_in":41505,"tokens_out":5953,"duration_ms":72428,"concrete_test":"Run a finite-state computation for Lemma 3.4 with S={0,1,2}, uniform initial distribution over {0,1,2}, g1(s)=s for all s, g2(0)=0, g2(1)=-10, g2(2)=10, and g2(10)=g2(-10)=0, horizon H=101 indexing h=0..101. Then V(g2)=200/3 > V(g1)=2/3, but V(T2)=2/3+200/3=202/3 while V(T1)=102*(2/3)=68, so V(T2)<V(T1), directly violating Lemma 3.4's conclusion. If this computation is verified, the theoretical claim that zero-mean reward perturbation expands exploration via trajectory variance is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that zero-mean reward noise expands exploration rests on the chain Lemma 3.2 -> Lemma 3.4 -> Theorems 3.5/3.6. The weak link is Lemma 3.4. It defines trajectories by deterministic recursion s_{h+1}=g(s_h), then Eq. (14) writes s_h = p(s0) * product of g evaluations, which is not a well-defined state; and Eq. (15) simply asserts the per-step variance inequality 1/N sum |s_h^(2)-mean|^2 > 1/N sum |s_h^(1)-mean|^2 that the lemma is supposed to prove. The condition V(g(2)) > V(g(1)) is insufficient for this: a map with high one-step output variance can contract its image to a point on the next iteration, eliminating trajectory variance, while a lower-variance map can preserve it. A concrete finite-state counterexample exists, so the lemma is not merely unproven but false. Since Theorems 3.5 and 3.6 invoke Lemma 3.4 to conclude V(T_RRP) > V(T_ori), the advertised theoretical support for expanded exploration collapses. A secondary inconsistency strengthens the concern: Algorithm 1's annealing step max{0, epsilon - epsilon*t/(lambda*T)} clips every negative noise sample to zero, so the implemented perturbation is positively biased rather than zero-mean, contradicting Eq. (1).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Random Reward Perturbation (RRP), a method that adds zero-mean Gaussian noise to environmental rewards and anneals the noise magnitude during training, as a lightweight and generally applicable exploration booster for model-free RL. The authors provide a theoretical analysis aiming to show that reward perturbation increases the variance of learned policies and of sampled state trajectories (Lemma 3.2, Lemma 3.4, Theorems 3.5 and 3.6), and support the claim with experiments integrating RRP into SAC and PPO across nine continuous-control tasks under dense and sparse rewards, including ablation studies of the noise scale and decay period.","tokens_in":41788,"tokens_out":4718,"duration_ms":54619,"significance":"If the theoretical claims were valid, RRP would be an attractive method: it is conceptually simple, requires almost no additional computation, and the experiments show consistent improvements over vanilla SAC and PPO across several challenging domains. The empirical section is broad and includes a useful MountainCar case study of visited-state density. However, the advertised theoretical contribution — the guarantee that trajectory variance, and hence exploration, strictly increases under RRP — is not established. The load-bearing proof chain is invalid at its central step (Lemma 3.4), and the transfer from value/advantage variance to policy and trajectory variance is asserted rather than derived. The method also relies on two tuned hyperparameters (σ0 and λ), so it is not parameter-free. No code release is indicated. The empirical results may still support RRP as a heuristic, but the paper's central theoretical claim, as stated in the abstract, is unsupported.","major_comments":[{"comment":"The proof of Lemma 3.4 is not valid. Equation (14) defines s_h^{(k)} = p(s_0) ∏_{h=1}^H g^{(k)}(s_{h-1}^{(k)}), which mixes a probability distribution with iterated function evaluations and is not a well-defined state. Equation (15) then simply asserts the per-step variance inequality 1/N ∑ |s_{h,i}^{(2)} − mean|² > 1/N ∑ |s_{h,i}^{(1)} − mean|² that the lemma is supposed to prove; no derivation from V(g^{(2)}) > V(g^{(1)}) is given. The statement itself is false: a function with high one-step output variance can be followed by a contracting iteration that collapses the spread of trajectories, while a lower-variance function can preserve it. Since Theorems 3.5 and 3.6 both invoke Lemma 3.4 to conclude V(T_RRP) > V(T_ori), the trajectory-level exploration claim collapses.","section":"§3.2, Lemma 3.4, Eqs. (14)–(16)"},{"comment":"Lemma 3.2 is a one-step SGD calculation on a fixed initial parameter vector θ0, but the paper immediately states that the variance increase 'accumulates over successive training steps' and uses this to justify the RL theorems. No induction is provided: after the first step, θ1 differs between the original and perturbed scenarios, and the TD targets y_ori and y_RRP in Eqs. (17) and (20) depend on each scenario's own target network or value network. Therefore Lemma 3.2 cannot be iterated to bound the variance of a multi-step RL training process.","section":"§3.2, Lemma 3.2 and text after Eq. (11)"},{"comment":"The step 'Since V(Q_RRP) > V(Q_ori), it follows that V(π_RRP) > V(π_ori)' is asserted without proof. Variance of Q-function outputs does not monotonically determine variance of the softmax policy or of actions sampled from it. In addition, the mapping g_k(s) = π_k(a|s) T(s'|s,a) defined in Eq. (19) is a transition kernel and not a deterministic function S → S as required by Lemma 3.4. Thus Theorem 3.5 does not establish V(T_RRP) > V(T_ori).","section":"§3.2, Theorem 3.5, Eqs. (18)–(19)"},{"comment":"Equation (22) decomposes Var[A_RRP] − Var[A_ori] as the sum of value-function variance differences plus σ², which assumes that R_env, V_ϕ(s), V_ϕ(s′), and ε are uncorrelated. No such independence is established, and V_ϕ is itself trained on the perturbed rewards, so this covariance omission is not benign. Moreover, even if Var[A_RRP] > Var[A_ori] held, the inference to V(π_RRP) > V(π_ori) and then to V(T_RRP) > V(T_ori) is again asserted rather than proved.","section":"§3.2, Theorem 3.6, Eqs. (21)–(22)"},{"comment":"The annealing update ε_{t,i} ← max{0, ε_t − ε_t t/(λT)} clips every negative noise sample to zero, so the perturbation actually applied in RRP-SAC has a positive mean whenever the noise scale is nonzero. This contradicts the zero-mean noise assumption in Eq. (1) and in the theoretical analysis. The implementation should scale the noise multiplicatively, e.g., ε_{t,i} ← ε_t · max{0, 1 − t/(λT)}, if a zero-mean annealed perturbation is intended.","section":"§4, Algorithm 1, line 10"}],"minor_comments":[{"comment":"The Taylor expansion writes the higher-order term as o(∥θ0 − θ0∥), which is identically zero; it should be o(∥θ1 − θ0∥).","section":"§3.2, Eq. (7)"},{"comment":"Equation (12) in the main text sums M(x_n, x_i) over i but not over n, making the displayed expression depend on an unspecified n; the corrected version appears in Appendix A.1, Eq. (40).","section":"§3.2, Eq. (12)"},{"comment":"The variance V(g) from Definition 3.1 is defined over a fixed dataset, but in the sequential setting the input distribution to g changes at every step; the lemma should specify over which input distribution the output variances are evaluated and how that distribution is updated.","section":"§3.2, Definition 3.3 and Lemma 3.4"},{"comment":"DQN's behavior policy is ε-greedy over Q-values, not the softmax policy written in Eq. (18); the softmax policy is not used by DQN in the paper's own setup.","section":"§3.2, Eq. (18)"},{"comment":"Several figure captions and axis labels contain font-encoding artifacts (strings such as '/uni00000013/uni00000015...'), making them unreadable in the provided manuscript; these need to be regenerated.","section":"Figures 2, 3, 5, and 6"},{"comment":"Many reported standard deviations are exactly 0.00 despite averaging over 5 seeds, which is implausible for continuous-control tasks; the authors should clarify the number of seeds and the presentation of confidence intervals.","section":"§5.1, Table 1"}],"recommendation":"reject","confidential_remarks":"The empirical study is reasonably broad and the proposed method is simple and cheap, but the paper's advertised theoretical guarantee is the main contribution and it is not supported. The central lemma is false as stated, the transfer to trajectory variance is unproved, and the RRP-SAC implementation uses a biased annealing step. These are load-bearing issues that would require a fundamentally revised theoretical section or a substantial reframing as an empirical-only study, so I do not see a path to acceptance within the current manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the recipe is genuinely cheap and the experiments look honest, but the theory that is supposed to explain RRP does not hold together, and the method is closer to prior work than the paper admits.\n\nWhat the paper does well: it runs a clean empirical comparison across nine continuous-control tasks with sparse and dense rewards, against RND, NovelD, ExploRS, NoisyNet, and vanilla SAC/PPO. The RRP variants consistently beat their backbones and roughly match the more expensive exploration methods. The ablations on noise scale and decay period are useful. I believe the empirical curves. I also credit the authors for stating in the Discussion that RRP struggles in extremely sparse or long-horizon tasks.\n\nThe soft spots are substantial. Lemma 3.2 is a correct one-step SGD variance computation under first-order Taylor, but it says nothing about policy variance or trajectory variance in RL. The bridge is Lemma 3.4, and it is not just unproven; it is false as stated. Equation (14) writes s_h as a product of function evaluations, which is not a well-defined state, and Equation (15) simply asserts the per-step variance inequality the lemma is supposed to establish. Output variance dominance at one step does not imply trajectory variance dominance: a map with larger one-step variance can contract its image on the next step and kill trajectory variance. So Theorems 3.5 and 3.6 do not establish the exploration claim. There is also a concrete implementation bug: in Algorithm 1, the SAC version anneals stored noise with max{0, epsilon - epsilon*t/(lambdaT)}, which clips every negative sample to zero. The implemented perturbation is positively biased, contradicting the zero-mean assumption in Eq. (1). PPO's version samples after annealing, so it is zero-mean, but SAC's is not.\n\nThe other issue is positioning. Sun et al. (2021) and Wang et al. (2020), both cited in Related Work, add reward noise for exploration, and neither is used as a baseline. That makes the 'novel' claim weaker than it looks. There is also no code released, which matters for a method whose only real asset is cheapness.\n\nProportion: the empirical finding may still be useful even if the theory is wrong. But the theory is not a minor decoration; it is the headline claim of the abstract. As it stands, the paper overclaims. The right fix is to remove or seriously rewrite the theoretical section, add reward-perturbation baselines, correct the SAC noise clipping, and release code. If that happens, this could be a decent short empirical paper. I would not accept it in its current form, but I would send it to review with a request for major revision rather than desk-reject; the experimental setup is solid enough to be salvageable.","headline":"A cheap exploration trick with believable empirical curves, but the theory is broken and the closest prior work is never compared.","tokens_in":42340,"tokens_out":3973,"would_cite":false,"duration_ms":42343,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding zero-mean noise to rewards provably expands the variance of RL trajectories, and the paper shows the trick boosts PPO and SAC on nine continuous-control tasks.","keywords":["reinforcement learning","exploration","reward perturbation","Gaussian noise","policy diversity","sparse rewards","PPO","SAC"],"falsifier":"Run two tabular Q-learning agents on the grid-maze of Figure 1b with identical seeds, one with original rewards and one with zero-mean noise added to rewards, and record the variance of visited states at each episode. If the noise-augmented agent does not exhibit strictly greater state-visitation variance than the baseline at the same training step, the trajectory-variance claim would be falsified.","tokens_in":41302,"feed_emoji":"🎲","tokens_out":6262,"duration_ms":67502,"temperature":0.7,"pith_summary":"Random Reward Perturbation (RRP) is the claim that adding zero-mean Gaussian noise to the reward signal during training widens the range of states and actions an agent explores, and that this can be added to any model-free reinforcement learning algorithm at negligible cost. The paper proves, for a one-step SGD update, that noise-perturbed labels increase the variance of a model's outputs, and then extends this to the variance of trajectories in DQN and A2C by treating transitions as a deterministic map. On nine continuous-control tasks, RRP-PPO and RRP-SAC improve sample efficiency and final returns over their vanilla counterparts, with the largest gains under sparse rewards. The contribution, taken sympathetically, is that random reward perturbation itself is a valid exploration strategy, complementary to action-level randomness and cheaper than novelty-based intrinsic motivation.","feed_headline":"Zero-mean reward noise provably broadens RL exploration","feed_subtitle":"A lightweight trick lifts PPO and SAC past vanilla and matches costlier novelty methods on sparse rewards.","key_machinery":"The load-bearing object is the variance of model outputs, defined as the trace of the covariance matrix over a dataset, and the variance of state trajectories over a horizon. Lemma 3.2 is the engine: for a single SGD step with Gaussian label noise, the output covariance grows by $\\frac{\\alpha^2 B \\sigma^2}{N} \\sum_j \\mathrm{Tr}(A_j A_j^T)$, where $A_j$ measures how the Jacobian products vary across the data, while the expected output mean is unchanged. Theorems 3.5 and 3.6 feed this into DQN and A2C TD targets, and Lemma 3.4 transfers the per-step variance dominance to trajectory-level variance by composing state maps. The mechanism's practical role is to make the optimization target itself wander, so the policy is pushed in different directions each update instead of locking onto a single reward surface.","core_discovery":"RRP defines a perturbed reward $R_{\\mathrm{RRP}}(s) = R_{\\mathrm{env}}(s) + \\varepsilon$ with $\\varepsilon \\sim \\mathcal{N}(0, \\sigma^2)$, and anneals $\\sigma$ linearly during training so the agent eventually optimizes the true environment reward. The central theoretical claim is that this makes the variance of sampled trajectories strictly larger than without perturbation, which the paper identifies with broader exploration. The argument goes through a sequence of steps: Lemma 3.2 shows one SGD step on noise-perturbed labels increases output variance while preserving the expected output mean; Theorem 3.5 applies this to DQN's TD targets, and Theorem 3.6 to A2C's value function and advantage estimates; Lemma 3.4 then converts higher output variance into higher trajectory variance by composing the policy with the transition function. Experiments with PPO and SAC support the claim: the perturbed variants consistently beat their backbones, especially in sparse-reward settings, and reach final performance comparable to RND, NovelD, and ExploRS with far less computation.","pith_inferences":["If the mechanism is right, reward noise is a generic 'optimization wobble' applicable to any gradient-trained predictor, though whether that wobble helps depends on the loss landscape; this is a testable conjecture outside RL.","The paper equates exploration with variance of visits, but the proof transfers variance from function outputs to trajectories only by assuming the per-step inequality it needs; a direct measurement of state-visitation variance during training would settle the link.","RRP's noise is agnostic to the state, so it could be layered on top of novelty or shaped rewards, turning intrinsic motivation into a slowly-wandering target rather than a fixed bonus."],"forward_implications":["RRP can be plugged into any model-free algorithm by editing only the reward line, requiring no extra networks, buffers, or density models.","It composes additively with action-level exploration: stochastic policies, $\\epsilon$-greedy, and entropy regularization continue to work and RRP adds further gains.","In sparse-reward environments, where most rewards are zero, the noise prevents the TD target from collapsing onto a constant, delaying premature convergence.","Because the noise is annealed to zero, the final policy is still optimized against the true environment reward, so exploration does not permanently distort exploitation.","On the paper's test suite, RRP improves both convergence speed and final returns of PPO and SAC, and beats NoisyNet while matching or approaching the final performance of RND, NovelD, and ExploRS at lower computational cost."],"supporting_citations":[{"why":"Defines DQN and its TD target, which Theorem 3.5 perturbs.","marker":"Mnih et al., 2015"},{"why":"Defines A2C, the actor-critic algorithm analyzed in Theorem 3.6.","marker":"Mnih et al., 2016"},{"why":"PPO is one of the two backbones that RRP is added to in the experiments.","marker":"Schulman et al., 2017"},{"why":"SAC is the other backbone, used with a replay buffer and annealed noise.","marker":"Haarnoja et al., 2018"},{"why":"Supplies the premise that exploration corresponds to variance of behaviors and visited states.","marker":"Ladosz et al., 2022"},{"why":"RND is the baseline intrinsic-motivation method that RRP compares against in cost and performance.","marker":"Burda et al., 2018"},{"why":"NovelD is another novelty-based baseline used in the comparison.","marker":"Zhang et al., 2021"}],"fun_headline_variants":["Noisy rewards broaden RL exploration beneficially","Adding random noise to rewards boosts RL exploration","Provable exploration boost via reward perturbation","Reward perturbation: a lightweight way to explore RL","Zero-mean noise on rewards enhances RL exploration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the variance increase proven for a single SGD step on a fixed dataset carries over to the state-visitation variance of a deep RL agent across the whole training run; the paper asserts this transfer rather than proving it.","fun_headline_variants_meta":{"raw":{"variants":["Noisy rewards broaden RL exploration beneficially","Adding random noise to rewards boosts RL exploration","Provable exploration boost via reward perturbation","Reward perturbation: a lightweight way to explore RL","Zero-mean noise on rewards enhances RL exploration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1603,"prompt_tokens":903,"completion_tokens":700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":519,"tokens_out":700,"duration_ms":8926,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:02:50.060949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run two tabular Q-learning agents on the grid-maze of Figure 1b with identical seeds, one with original rewards and one with zero-mean noise added to rewards, and record the variance of visited states at each episode. If the noise-augmented agent does not exhibit strictly greater state-visitation variance than the baseline at the same training step, the trajectory-variance claim would be falsified.","supporting_citations":[{"cited_title":"P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K","cited_arxiv_id":null,"evidence_quote":"Defines A2C, the actor-critic algorithm analyzed in Theorem 3.6."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SAC is the other backbone, used with a replay buffer and annealed noise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the premise that exploration corresponds to variance of behaviors and visited states."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RND is the baseline intrinsic-motivation method that RRP compares against in cost and performance."},{"cited_title":"E., and Tian, Y","cited_arxiv_id":null,"evidence_quote":"NovelD is another novelty-based baseline used in the comparison."}],"review_version":1}