{"id":"9ea5b403-96b1-4d83-847b-d900537dbee6","arxiv_id":"2505.11044","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RDD distills a Gaussian-random target and rewards the squared deviation of the predictor from the target mean, giving an unbiased estimator of 1/n visits that unifies RND and pseudo-count exploration.","lead":"This paper introduces RDD, an exploration bonus for reinforcement learning: the agent's predictor learns from noisy random target samples and the bonus is how far the predictor's output is from the target's mean. In an idealized model, that bonus decays like the reciprocal of the visit count, connecting curiosity-based exploration with count-based exploration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pseudo-count decay claim rests on identifying the SGD-trained predictor with the per-state empirical-risk minimizer, but the implemented target is a constant mean shared across states, so the 1/n bonus decay is not established.","rationale":"I agree with the reader's weakest_assumption: the weakest link is the identification of the SGD-trained shared predictor with the exact per-state empirical-risk minimizer f^n*, compounded by the implemented constant target mean. The reader's other technical objections (the invalid Eq. 13 factor, Theorem 4.3's rate claim, Corollary 4.2's variance comparison, garbled Table 1, missing CFN and code) are all real and support a CONDITIONAL verdict, but the deepest issue is that the theory's object of study is not the object being trained. A controlled experiment can settle whether the 1/n decay occurs in the actual implementation. Since the reader already flagged this as the weakest assumption and conditioned acceptance on repairs, my read does not move the verdict.","tokens_in":21067,"tokens_out":5874,"duration_ms":61920,"concrete_test":"Run the actual RDD algorithm with the paper's settings (mu=1, sigma=1.0, d=64, same optimizer/network) on a stream where one probe state s* is visited n times while other states are visited once, and record b(s*) and b(s_other) as functions of n. If b(s*) tracks sigma^2/n (or, more directly, the per-state empirical-mean value ||(1/n)sum f_i_tar(s*)-1||^2/d) while b(s_other) stays high, the pseudo-count mechanism is realized. If b(s*) decays only as fast as b(s_other), or plateaus, the central claim fails and the method reduces to an unnormalized prediction-error bonus without visit-count decay.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the RDD bonus b(s)=(1/d)||f_theta(s)-mu_bar_theta(s)||^2 is bounded by sigma^2/n plus a predictor-convergence discrepancy (Eq. 13), and therefore decays like a pseudo-count. Three linked gaps make this claim unsupported. First, Eq. 13 is mathematically invalid as printed: the first equality treats a deterministic left-hand side as an expectation, and the triangle inequality for squared norms requires a factor of 2, so the claimed bound sigma^2/n + discrepancy does not follow even after correcting the notation. Second, and more load-bearing, the bound is only about the optimal per-state predictor f^n*_theta(s) of Eq. 4, which minimizes MSE after n visits to that state. The actual predictor is a single shared network trained by SGD over all states; nothing in the paper shows it converges to f^n*(s), and if it does not, the discrepancy term in Eq. 13 does not vanish with n, so the bonus need not decay at all. Third, the implementation does not match the model: Table 4 and Section 5.4 fix the target mean to the scalar 1 (and sigma=1.0) for every state, so the target distribution is identical across states. Then the MSE-optimal predictor for the actual training objective is simply f_theta(s)=1 everywhere; the squared distance to 1 decays as the network learns this constant function, independently of how often each state is visited. The pseudo-count interpretation thus rests on a per-state independence and per-state memorization that the implemented algorithm does not have. Section 7 itself concedes that target distributions across states are not independent in practice. Absent a demonstration that the shared SGD predictor tracks per-state empirical means, RDD collapses to an unnormalized RND-like bonus and the unification claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Random Distribution Distillation (RDD), an exploration bonus for online RL. RDD samples the target network's output for a state from a normal distribution, trains a predictor with MSE loss, and sets the intrinsic reward to b(s) = (1/d)||f_theta(s) - mu_theta_bar(s)||^2, the squared distance between the predictor and the target distribution mean. The central theoretical claim is that the idealized statistic z_n = ||f^n_theta*(s) - mu_theta_bar(s)||^2 / sigma^2 is an unbiased and consistent estimator of 1/n with variance 2/n^2, so the bonus is bounded by a pseudo-count term sigma^2/n plus a discrepancy term that accounts for predictor convergence (Eq. 13). The paper further claims that RND is the sigma -> 0 special case and that DRND converges to RDD as the number of target networks grows. Experimental results are reported on Atari, Adroit, and Fetch tasks, along with ablations of the target mean, variance, and output dimension.","tokens_in":21259,"tokens_out":7748,"duration_ms":77776,"significance":"If the theoretical claims were correct and the implementation matched the model, RDD would be a useful unification of prediction-error and count-based exploration: it would provide a pseudo-count-like decay with a simple one-network bonus and with variance smaller than DRND's. The core computation in Appendix A, Var(z_n) = 2/n^2, is correct, and Figure 1 qualitatively supports the statistical variance-reduction story. However, the central bridge from the idealized statistic to the actual bonus, Eq. (13), is mathematically invalid as printed, the proof of consistency in Theorem 4.3 contains an error, and the implemented algorithm uses a constant target mean across states, whereas the theory requires a state-dependent target mean. These are load-bearing issues: they break the paper's main claim that the RDD bonus decays like a pseudo-count. The idea is salvageable, but the manuscript needs substantial revision before the claim can be accepted.","major_comments":[{"comment":"The decomposition in Eq. (13) does not follow mathematically. The left-hand side ||f_theta(s) - mu_theta_bar(s)||^2 is deterministic for fixed theta, so the first equality, which rewrites it as an expectation, is not meaningful in the form given. For the inequality, the squared-norm triangle inequality is ||a + b||^2 <= 2||a||^2 + 2||b||^2, so the correct bound would carry a factor of 2 on both the pseudo-count term and the discrepancy term. As printed, the claimed upper bound sigma^2/n + E[||f^n_theta*(s) - f_theta(s)||^2] is not established; the cross term needs to be handled explicitly and the coefficients corrected.","section":"§4.4, Eq. (13)"},{"comment":"The pseudo-count decay claim rests on identifying the trained predictor with f^n_theta*(s), the exact per-state empirical-risk minimizer of Eq. (4). In the implemented algorithm the predictor is a single network trained by SGD on all states jointly, and the paper gives no argument that it converges to the per-state empirical-risk minimizer for each state. If the discrepancy term E[||f^n_theta*(s) - f_theta(s)||^2] does not vanish with n, Eq. (13) does not imply that the bonus decays with the visit count n. This is the load-bearing step of the pseudo-count claim and needs either a convergence proof under the actual training procedure or a controlled experiment showing state-specific bonus decay that is independent of global training progress.","section":"§4.2–§4.3, Eqs. (4) and (13)"},{"comment":"The implemented algorithm does not instantiate the model analyzed in Lemma 4.1. Table 4 and the ablation in Section 5.4 set the target mean mu to the scalar 1 (and sigma = 1.0) for every state, whereas Lemma 4.1 requires a state-dependent target mean mu_theta_bar(s). With a constant target distribution N(1, 1) shared across all states, the MSE-optimal predictor of Eq. (3) is the constant function f_theta(s) = 1 everywhere, so the bonus b(s) = ||f_theta(s) - 1||^2 decays as the network learns this constant function, with no dependence on how often each state is visited. The empirical results therefore do not test the pseudo-count interpretation. The paper must either modify the algorithm to use a state-dependent random target mean network, as the theory assumes, or explicitly state that the pseudo-count claim applies only to an idealized version of the algorithm and not to the evaluated method.","section":"§5.4, Table 4"},{"comment":"Theorem 4.3 contains an error in its statement and its proof does not establish consistency. The text says 'lim_{n->infinity} P(|z_n - 1/n| < epsilon) = 0', which is the opposite of consistency; the limit should be 1. In addition, the epsilon in the theorem statement is a function of n and delta whose second term tends to sqrt(ln(2/delta)/2), a positive constant, so the bound does not show that for a fixed epsilon the probability tends to zero. This is repairable: applying Chebyshev's inequality to Var(z_n) = 2/n^2 from Eq. (21) gives P(|z_n - 1/n| > epsilon) <= 2/(n^2 epsilon^2) -> 0, which is a clean consistency proof. As printed, both the theorem statement and the Appendix C derivation need correction.","section":"§4.2, Theorem 4.3"}],"minor_comments":[{"comment":"The notation mu_theta_bar(s) = mu_theta_bar(s) x 1_d is self-referential and confusing; the scalar mean should be named differently, for example mu, and the vectorized form written as mu * 1_d.","section":"§4.3, Eq. (8)"},{"comment":"The pseudo-code says to compute 'the mean mu_theta(s_t) and second moment sigma_phi(s_t)', which suggests state-dependent target parameters, but Table 4 and Section 5.4 fix mu = 1 as a scalar constant. Please clarify whether mu is a learned state-dependent network or a constant, and make the pseudo-code consistent with the implementation.","section":"Algorithm 1, line 10"},{"comment":"The quantities yapprox and zapprox are used in the caption and the plots but are not defined in the caption; please define them in the text or the caption so the figure is self-contained.","section":"§5.1, Figure 1"},{"comment":"Several rows in Table 1 appear to contain identical values across the RDD and DRND columns (for example, Montezuma and PrivateEye), which makes it difficult to assess the claimed improvements; please reformat the table so that the per-algorithm columns are clearly separated and the reported values are unambiguous.","section":"Table 1"},{"comment":"The independence assumption stated in the Limitations section is precisely the assumption that the pseudo-count proof requires; the paragraph should state which theoretical conclusions are invalidated when the assumption fails, rather than only noting that similar states may have similar distributions.","section":"§7, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The central statistical lemma is correct and the RDD idea is potentially salvageable, but the gap between the implemented constant-mean target and the state-dependent target distribution assumed by the theory is substantial. The authors should be encouraged to address the implementation mismatch head-on, correct Eq. (13) and Theorem 4.3, and either derive the DRND variance formula in the appendix or cite a non-self-referential source for it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth knowing about, but the central claim is not established as written. The statistical observation at the heart of the paper is real: if you sample target outputs for a state from a normal distribution, the statistic z_n = ||f^n_theta*(s) - mu||^2 / sigma^2, where f^n_theta* is the average of n target samples, has expectation 1/n and variance 2/n^2. Lemma 4.1 and the variance computation check out. The observation that RDD reduces to RND as sigma -> 0 is also correct, and the DRND-as-infinite-ensemble story is plausible. That is genuine credit.\n\nThe soft spots are major, though. Equation 13 is simply invalid: the first equality treats a deterministic left-hand side as an expectation, and the triangle inequality for squared norms requires a factor of 2. The claimed bound b(s) <= sigma^2/n + discrepancy does not follow. Theorem 4.3 also does not do what it claims: the epsilon in the statement does not tend to zero as n grows, so it cannot establish consistency or any O(n^{-1}) rate. More fundamentally, the pseudo-count interpretation requires the trained predictor to behave like the per-state empirical-risk minimizer of Eq. 4. Nothing shows that the shared SGD-trained network does that, and the implementation fixed the target mean to the scalar 1 (Section 5.4, Table 4), so the target distribution is identical across states. In that case the optimal predictor is the constant 1, and the bonus decays as the network learns that constant, not with per-state visit count. The paper's own Limitations section concedes that the independence assumption fails in practice. Together, these break the load-bearing claim that the actual RDD bonus behaves like a pseudo-count.\n\nSmaller issues: Table 1 has repeated values across three methods and several implausible zero standard errors, no code is released, and CFN—which the paper explicitly references—is missing from the baselines. The DRND variance formula is quoted from the authors' own ICML paper, which is worth noting but not decisive.\n\nWho is this for: people working on intrinsic rewards and exploration. If the empirical results survive a clean re-run, RDD could be a simple and useful baseline. But the theoretical contribution does not currently support the unification story the paper advertises. I would send it to a serious referee, with clear instructions that Eq. 13 and Theorem 4.3 need repair, the implementation needs to be reconciled with the theory, and per-seed results plus code are required.","headline":"RDD has a correct core estimator but the bridge from that estimator to the actual exploration bonus is broken, and the implementation does not match the theory.","tokens_in":22035,"tokens_out":7060,"would_cite":false,"duration_ms":71959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Random Distribution Distillation claims that a prediction-error bonus can act as an unbiased estimator of the reciprocal visit count $1/n$, unifying curiosity-based and count-based exploration.","keywords":["exploration","intrinsic reward","random network distillation","pseudo-count","reinforcement learning","prediction error","sparse reward"],"falsifier":"Repeatedly visit a single state while training the RDD predictor only on those visits, and record the bonus $b(s)=\\frac{1}{d}\\|f_\\theta(s)-\\mu_{\\bar\\theta}(s)\\|^2$ as $n$ grows; if the bonus does not decrease like $\\sigma^2/n$ plus a shrinking discrepancy term, and instead plateaus at a constant, the pseudo-count interpretation is refuted.","tokens_in":20704,"feed_emoji":"🧭","tokens_out":13445,"duration_ms":111029,"temperature":0.7,"pith_summary":"Random Distribution Distillation (RDD) is an online reinforcement-learning exploration method that tries to combine the scalability of prediction-error curiosity with the principled decay of count-based bonuses. The intrinsic reward is the squared distance between a predictor network and the fixed random mean of a target network whose outputs are sampled from a normal distribution. The paper argues that in an idealized setting this reward estimates the reciprocal of the state's visit count, $1/n$, up to a convergence-discrepancy term, so the bonus shrinks as a state becomes familiar. If the argument holds, RND is the zero-variance special case of RDD, DRND is the finite-ensemble approximation, and RDD gives curiosity-driven exploration a pseudo-count mechanism. Experiments on sparse-reward Atari, Adroit, and Fetch tasks report faster convergence and higher returns than RND, DRND, and PPO.","feed_headline":"Random distribution distillation turns novelty into visit counts","feed_subtitle":"Prediction-error exploration gains a provable count-like decay, unifying RND and pseudo-count methods.","key_machinery":"The load-bearing object is the statistic $z_n(s)=\\|f^n_{\\theta^*}(s)-\\mu_{\\bar\\theta}(s)\\|^2/\\sigma_{\\bar\\phi}^2(s)$: the squared distance, normalized by target variance, between the empirical-risk-minimizing predictor after $n$ visits and the fixed random target mean. Because each visit draws an independent target output from $\\mathcal{N}(\\mu_{\\bar\\theta}(s),\\sigma_{\\bar\\phi}^2(s))$, the average of $n$ samples has expected squared deviation $\\sigma_{\\bar\\phi}^2(s)/n$ from the mean, making $z_n$ an unbiased estimator of $1/n$ with variance $2/n^2$. The same statistic grounds the bonus bound in Eq. 13, which decomposes the intrinsic reward into a pseudo-count term that decays with visits and a discrepancy term that shrinks as the predictor converges.","core_discovery":"The RDD bonus is $b(s)=\\frac{1}{d}\\|f_\\theta(s)-\\mu_{\\bar\\theta}(s)\\|^2$, where $f_\\theta$ is the predictor network and $\\mu_{\\bar\\theta}(s)$ is the fixed mean of a random target distribution $\\mathcal{N}(\\mu_{\\bar\\theta}(s),\\sigma_{\\bar\\phi}^2(s))$ from which fresh target outputs are sampled on each visit. The central theoretical claim is that the normalized statistic $z_n(s)=\\|f^n_{\\theta^*}(s)-\\mu_{\\bar\\theta}(s)\\|^2/\\sigma_{\\bar\\phi}^2(s)$, evaluated at the optimal mean-squared-error predictor after $n$ visits, is an unbiased and consistent estimator of $1/n$ with variance $2/n^2$. The implemented bonus satisfies the bound $\\|f_\\theta(s)-\\mu_{\\bar\\theta}(s)\\|^2 \\le \\sigma_{\\bar\\phi}^2(s)/n + \\mathbb{E}[\\|f^n_{\\theta^*}(s)-f_\\theta(s)\\|^2]$, splitting the reward into a pseudo-count term and a predictor-discrepancy term. From these results the paper concludes that RDD automatically couples exploration reward to both visit frequency and predictor convergence, that RND is the $\\sigma_{\\bar\\phi}\\to 0$ limit of RDD, and that DRND converges to RDD as the number of target networks grows.","pith_inferences":["A direct test would be to measure the bonus decay curve on repeated identical states in a real environment; if $1/n$ tracking requires the predictor to be trained to convergence on those states, the practical gain may come from the discrepancy term rather than from pseudo-counting.","The ablation that fixes the target mean to the scalar $1$ suggests the implemented RDD is closer to a shaped novelty bonus than to the state-specific pseudo-count of the theory, so a state-dependent-mean variant could distinguish which design choice drives the reported gains.","Because output dimension $d$ reduces the variance of $z_n$, $d$ and $\\sigma$ together act as a bias-variance knob for exploration intensity, suggesting a principled way to calibrate intrinsic-reward scale across environments.","If the predictor never fully fits repeated states, RDD may reduce to an RND-like unnormalized bonus; comparing RDD against RND with matched network capacity on identical visit sequences would reveal how much of the benefit is actually count-like."],"forward_implications":["RDD gives a curiosity bonus a principled decay: rewards for frequently visited states shrink, so exploration shifts to unfamiliar states without needing a count table or density model.","RND is recovered as the limit where the target variance goes to zero, making RND's unnormalized prediction error a special case of a count-based statistic.","DRND's ensemble of target networks converges to RDD as the ensemble size grows, positioning RDD as the asymptotic limit of DRND.","Because the bonus bound includes a predictor-discrepancy term, the intrinsic reward is self-limiting: as the predictor converges, the bonus shrinks even for states that remain unfamiliar.","In the reported experiments, RDD converges faster and reaches higher final returns than RND, DRND, and PPO on sparse-reward Atari, Adroit, and Fetch tasks."],"supporting_citations":[{"why":"Supplies the count-based exploration theory, with $1/\\sqrt{n}$-style bonuses, whose decay RDD aims to replicate.","marker":"[2]"},{"why":"Provides the classic unification of count-based exploration and intrinsic motivation that RDD's pseudo-count story builds on.","marker":"[5]"},{"why":"Defines RND, the prediction-error method RDD extends and the zero-variance limit it recovers.","marker":"[10]"},{"why":"Defines DRND, the finite-ensemble baseline whose statistic $y_n$ is compared in Corollary 4.2 and whose asymptotic limit RDD claims to be.","marker":"[12]"}],"fun_headline_variants":["RDD: predictor error with provable pseudo-count decay","Random distribution distillation unifies RND and counts","Exploration bonus that provably shrinks with visits","A single intrinsic reward: pseudo-count plus predictor gap","RDD: count-based decay for prediction-error exploration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The count-decay claim holds only if the trained predictor at a state equals the exact running average of independent, state-specific target samples; when that fails, as the paper's own experiments partly concede by fixing the target mean to a scalar, the bonus need not decay like $1/n$.","fun_headline_variants_meta":{"raw":{"variants":["RDD: predictor error with provable pseudo-count decay","Random distribution distillation unifies RND and counts","Exploration bonus that provably shrinks with visits","A single intrinsic reward: pseudo-count plus predictor gap","RDD: count-based decay for prediction-error exploration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3392,"prompt_tokens":1052,"completion_tokens":2340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":2264}},"tokens_in":668,"tokens_out":2340,"duration_ms":19834,"temperature":1.0,"reasoning_tokens":2264,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:02:39.312826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeatedly visit a single state while training the RDD predictor only on those visits, and record the bonus $b(s)=\\frac{1}{d}\\|f_\\theta(s)-\\mu_{\\bar\\theta}(s)\\|^2$ as $n$ grows; if the bonus does not decrease like $\\sigma^2/n$ plus a shrinking discrepancy term, and instead plateaus at a constant, the pseudo-count interpretation is refuted.","supporting_citations":[{"cited_title":"Unifying count-based exploration and intrinsic motivation,","cited_arxiv_id":null,"evidence_quote":"Provides the classic unification of count-based exploration and intrinsic motivation that RDD's pseudo-count story builds on."},{"cited_title":"Exploration and anti-exploration with distributional random network distillation,","cited_arxiv_id":null,"evidence_quote":"Defines DRND, the finite-ensemble baseline whose statistic $y_n$ is compared in Corollary 4.2 and whose asymptotic limit RDD claims to be."}],"review_version":1}