{"id":"cbf1ab84-99d0-4f1a-a5f5-6eb86c71ed91","arxiv_id":"2506.22566","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Untrained neural policies can be engineered to produce either ballistic or diffusive exploration trajectories, depending on whether the policy is fixed or re-sampled at each step.","lead":"This paper studies how a randomly initialized neural network policy explores an environment before any training happens. It shows that a fixed untrained policy produces straight-line paths while re-sampling the policy at every step produces diffusive, heavy-tailed paths, and it proposes using this as a design tool for exploration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed heavy-tailed steady-state distribution (Eq. 3) is neither a solution of the stationary Fokker-Planck equation nor normalizable; for d≠2 with σ_b>0, substitution has nonzero residual and the radial integral diverges for all d.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing flaw: the steady-state density in Eq. (3) does not satisfy the paper's stationary Fokker-Planck equation in general and is not normalizable. Direct substitution shows a nonzero residual for d≠2 and σ_b>0, and the radial integral diverges logarithmically for every dimension, contradicting the claim of normalizability. This is not a matter of external consensus; it is an internal mathematical inconsistency in the derivation of Contribution (2). The ballistic result (Lemma 1) is independent and appears sound, so the overall paper is not wholly without value, but the heavy-tailed steady-state claim, a central advertised contribution, is unsupported as stated. The reader's CONDITIONAL verdict—requiring a corrected stationarity analysis and quantitative evidence—remains appropriate, so no change to the verdict is recommended.","tokens_in":6824,"tokens_out":8477,"duration_ms":93506,"concrete_test":"Compute the left side of ∇²[(σ_b²+σ_w² r²/π) p∞]=0 at r=1 for d=3, σ_b=σ_w=1, with p∞ as in Eq. (3); a nonzero residual directly falsifies the claimed solution. Alternatively, simulate the per-step resampled random walk s_{t+1}=s_t+π_t(s_t) in d=3 using an infinite-width ReLU GP and track the empirical second moment of the state distribution over 10^5 steps; if it grows without bound, no normalizable steady state exists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (3) is derived by solving ∇²(Σ p∞)=0, with Σ(s)=σ_b² + σ_w²∥s∥²/π. Substituting the claimed f(r)=(σ_b² + σ_w² r²/π)^(-d/2), the product Σf = (a+c r²)^(1-d/2) has Laplacian 2 c d a (1-d/2)(a+c r²)^(-(d+2)/2), which is identically zero only for d=2 or σ_b=0. For the d=2 case, the integral ∫ r^(d-1)(a+c r²)^(-d/2) dr still diverges logarithmically; for general d the tail is 1/r against the volume element, so ∫ r^(d-1) f(r) dr = ∞. The statement that the density 'is normalizable if σ_b>0' is therefore false. Equivalently, the underlying diffusion with Σ~r² cannot have a finite stationary measure for d≥2; the radial process behaves like a geometric Brownian motion with positive drift, so the second moment grows without bound. Consequently the central claim of a heavy-tailed steady-state distribution is unsupported by the paper's own Fokker-Planck analysis. The ballistic Lemma 1 is not affected, but Contribution (2) and the abstract's 'non-trivial state-visitation distributions' rest on Eq. (3).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies exploration behavior of randomly initialized neural network policies. It argues that fixed Lipschitz policies induce short-time ballistic trajectories (Lemma 1), and that re-sampling the policy from its initialization distribution at every timestep yields, through the infinite-width Gaussian process limit and a Fokker-Planck continuous-time limit, diffusive trajectories with a heavy-tailed stationary state distribution f(r) ∝ (σ_b² + σ_w² r²/π)^{-d/2} (Eq. (3)). The paper then sketches an experiment with a barrier and narrow hallway to illustrate ballistic, diffusive, and hybrid exploration strategies, and frames these findings as evidence that architecture and initialization can be used as design tools for exploration before training.","tokens_in":7127,"tokens_out":11738,"duration_ms":128451,"significance":"The conceptual direction is appealing: linking the NNGP kernel of an untrained policy to the statistics of the trajectories it induces offers a principled way to reason about exploration without reward shaping. The paper is transparent about using standard tools (Neal's infinite-width limit and the Fokker-Planck equation) and does not introduce fitted parameters, which is a strength. However, both central theoretical pillars are mathematically invalid in their current form: Lemma 1 is not true for the stated general dynamics, and Eq. (3) is neither a solution of the stationary Fokker-Planck equation nor a normalizable density. The heavy-tailed steady-state claim, which is the main novelty, is therefore unsupported. The qualitative experiments do not compensate because they illustrate but do not validate the theory. If the mathematical claims could be repaired, the framework would be a useful conceptual contribution for the RL exploration community; in the present form the paper does not meet the bar for publication.","major_comments":[{"comment":"Equation (3) is not a solution of the stationary Fokker-Planck equation in the stated generality, and the density it defines is not normalizable for any d ≥ 1. Substituting f(r) = (a + c r²)^{-d/2} with a = σ_b² and c = σ_w²/π into ∇²(Σ p∞) = 0 gives Δ(Σ f) = 2 c d a (1 - d/2)(a + c r²)^{-(d+2)/2}, which vanishes only for d = 2 or a = 0. Furthermore, ∫ r^{d-1}(a + c r²)^{-d/2} dr diverges logarithmically at infinity for every d ≥ 1, including when σ_b > 0, contradicting the statement that the density \"is normalizable if σ_b > 0\". Correspondingly, the diffusion with Σ(s) ~ ‖s‖² is transient rather than stationary: the radial process has a positive drift of order (d-1)cR/2, so no finite stationary measure exists. This invalidates Contribution (2) and the abstract's claim of non-trivial or heavy-tailed steady-state state-visitation distributions.","section":"§2.4, Eq. (3)"},{"comment":"Lemma 1 as stated is false for general dynamics satisfying Assumption 1 and the stated Lipschitz conditions. The proof compares s_{t+1} = f(s_t, πθ(s_t)) to the candidate linear trajectory s₀ + ct, but nothing in the assumptions ensures f(s, a) = s + a. Already at t = 0 the error |f(s₀, c) - (s₀ + c)| need not vanish; for example, dynamics of the form f(s, a) = s + a + ε with a small constant ε satisfy the Lipschitz and locality assumptions, yet the trajectory is s₀ + (c + ε)t, not s₀ + ct. The lemma becomes correct only under the additional assumption f(s, a) = s + a, or with c defined as the actual per-step displacement, which is the 'simplest linear dynamics' used later in §2.4. As written, the lemma is advertised for general transition dynamics and thus overclaims.","section":"§2.2, Lemma 1"}],"minor_comments":[{"comment":"There is a factor-of-π inconsistency in the kernel at coincident points: evaluating Eq. (1) at s' = s gives K(s, s) = σ_b² + σ_w²‖s‖², whereas Eq. (3) uses K(s, s) = σ_b² + (σ_w²/π)‖s‖². The kernel normalization should be fixed so the two equations agree.","section":"§2.4, Eq. (1) vs Eq. (3)"},{"comment":"The continuous-time limit leading to the Fokker-Planck equation is not made precise: no scaling of the action magnitude or the time step is specified. A reader cannot verify the diffusion coefficient in Eq. (2) without knowing how the per-step action variance is related to the time unit.","section":"§2.4, Eq. (2)"},{"comment":"The experiments are purely qualitative. The figures lack axis labels, environment details, hyperparameters, number of seeds, and quantitative exploration metrics, which makes the claimed empirical support difficult to assess or reproduce.","section":"§2.5, Figures 1-3"},{"comment":"The convergence notation πθ → GP(0, K) should be made precise as weak convergence of finite-dimensional distributions as the hidden widths tend to infinity. Also, K(s, s') is written as taking values in R^{d×d}, while the subsequent analysis treats it as a scalar; the relation between the scalar kernel and the output dimension should be clarified.","section":"§2.3, Theorem 4"},{"comment":"The term 'quasi-Cauchy' is used for the distribution in Eq. (3), but since that density is not normalizable, calling it a distribution is misleading; the terminology should be revised along with the mathematical claim.","section":"§2.4, 'quasi-Cauchy'"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early-stage workshop paper. The central mathematical errors are fundamental: Eq. (3) does not define a stationary probability density for the stated process, and Lemma 1 relies on an unstated assumption about the transition dynamics. Even substantial revision would require changing the main claims rather than patching local technicalities, so I recommend rejection rather than major revision. The conceptual link between NNGP kernels and trajectory statistics is worth exploring in future work, but the current manuscript does not provide a sound foundation for that direction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the one-line view: this paper has a fresh idea worth taking seriously, and a central derivation that is simply wrong. The fresh idea is to think of untrained RL policies as drawing trajectory priors—fixed Lipschitz policies give ballistic motion, per-step resampling from an architecture's initialization distribution gives diffusive motion, and you can hybridize the two. That framing, with infinite-width GP theory as the tool, seems like a legitimate new angle on exploration, and it's presented without fitted parameters or self-citation. The paper would be a decent conceptual contribution if the analysis held up.\n\nWhat's good: Lemma 1 is an elementary Lipschitz bound and it's correct; the proof in the appendix is fine. The observation that the ReLU kernel gives a position-dependent diffusion coefficient that grows as ||s||² is also correct, and the hybrid fixed-then-resampled strategy is a small but reasonable proposal. The experiments are only qualitative, so I wouldn't lean on them.\n\nWhere it falls: the steady-state claim in Eq. (3) is not supported. The paper writes the stationary Fokker-Planck condition as ∇²(Σ p∞)=0 and offers f(r) ∝ (σ_b² + σ_w² r²/π)^{-d/2}. Substituting, Σ f = (a + c r²)^{1-d/2}, whose Laplacian is 2 c m [d + 2 c (m-1) r²/(a+c r²)] with m=1-d/2; this vanishes for general r only in d=2 or σ_b=0. Even in d=2, the radial integral ∫ r dr / (a + c r²) diverges logarithmically; in general d the tail is r^{d-1}/r^d = 1/r, so the density isn't normalizable for any d. The statement \"normalizable if σ_b>0\" is false. This isn't a nitpick: Contribution (2) and the abstract's 'non-trivial state-visitation distributions' rest on this equation. The diffusion with Σ ~ r² is effectively a geometric Brownian motion in radius, which doesn't have a stationary measure. So the main diffusive result is unsupported.\n\nNothing else looks broken. The references are standard and the paper doesn't misuse them. The flaw is contained, and the ballistic part survives, but the paper's most interesting claim needs either a corrected stationarity analysis (maybe with an absorbing boundary or a different policy class) or a honest retreat to a finite-time statement.\n\nWho it's for: people working on exploration in deep RL or on architecture-induced priors. It deserves peer review in the sense that a reviewer can give specific, constructive mathematical feedback—a desk rejection would be too harsh. But as it stands I would not cite the steady-state result, and I'd want the stationarity issue fixed before treating the framework as established.\n\nMy recommendation: send it to referees, but expect heavy revision.","headline":"A genuinely new framing of exploration as an architecture/initialization problem, but the central heavy-tailed steady-state claim rests on a Fokker-Planck solution that does not solve the equation and is not normalizable.","tokens_in":7626,"tokens_out":3383,"would_cite":false,"duration_ms":31200,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","60J60","60G15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Untrained neural policies already determine how an agent explores.","keywords":["reinforcement learning","policy initialization","exploration","infinite-width limit","Gaussian process","Fokker-Planck equation","heavy-tailed distribution","ReLU networks"],"falsifier":"Run the per-step resampled ReLU policy under linear dynamics $s_{t+1}=s_t+\\pi(s_t)$ in two and three dimensions with $\\sigma_b>0$, record a long trajectory, and compare the empirical radial density to $f(r)\\propto(\\sigma_b^2+\\sigma_w^2 r^2/\\pi)^{-d/2}$; if the distribution is not heavy-tailed with that exponent, or if the total mass inside a large ball keeps growing with time, the claimed steady state fails.","tokens_in":6599,"feed_emoji":"🎲","tokens_out":10963,"duration_ms":99117,"temperature":0.7,"pith_summary":"The paper argues that exploration in reinforcement learning is shaped before any training by the architecture and initialization of the policy network. A fixed, smooth (Lipschitz) policy produces nearly straight-line, ballistic trajectories on short timescales, while re-sampling the policy from its initialization distribution at every step produces diffusive trajectories with heavy-tailed state visiting. For infinitely wide ReLU networks, the paper derives a closed-form diffusion coefficient that grows with distance from the origin and predicts a quasi-Cauchy power-law state distribution. This matters because it suggests exploration can be designed through policy initialization rather than through reward bonuses or training, giving a new zero-shot handle on sparse-reward problems.","feed_headline":"Untrained policies pick the exploration style before training","feed_subtitle":"A fixed smooth network walks in straight lines; re-sampling it each step spreads a heavy-tailed search.","key_machinery":"The argument runs on two limits. First, the infinite-width Gaussian process limit converts a randomly initialized feedforward network into a Gaussian process with an architecture-dependent kernel, so action correlations across states are exactly $K(s,s')$. Second, the continuous-time Fokker-Planck limit for the per-step resampled random walk in state space turns the kernel into a diffusion tensor, yielding the PDE $\\partial p/\\partial t = -\\nabla\\cdot(\\mu p) + \\tfrac12\\nabla^2(\\Sigma p)$. For ReLU networks the kernel is non-stationary and gives $\\Sigma(s)=\\sigma_b^2+\\sigma_w^2\\|s\\|^2/\\pi$, which is what produces position-dependent diffusion and the claimed heavy-tailed distribution. A separate Lipschitz argument bounds the deviation of fixed-policy trajectories from straight lines, supplying the ballistic half of the picture.","core_discovery":"The central claim is that the exploration behavior of an RL agent at the very start of training is controlled by the untrained policy's architecture and weight initialization. Concretely, a fixed deterministic policy that is Lipschitz continuous maps nearby states to nearby actions, and under local dynamics this yields ballistic motion: the trajectory stays within a bound of order $\\delta L_a L_\\pi t^2$ of the straight line $s_0 + c t$. Sampling a fresh policy from the same initialization each timestep instead decorrelates actions across steps, producing diffusive motion; in the infinite-width limit the policy is a Gaussian process with kernel $K(s,s')$, and for a one-hidden-layer ReLU network the diffusion coefficient is $\\Sigma(s)=\\sigma_b^2+\\sigma_w^2\\|s\\|^2/\\pi$, leading the paper to a heavy-tailed steady-state solution $f(r)\\propto(\\sigma_b^2+\\sigma_w^2 r^2/\\pi)^{-d/2}$. The paper proposes a hybrid strategy—fixed network first, then per-step resampling—to combine ballistic escape with diffusive coverage, and demonstrates it in a hallway barrier experiment.","pith_inferences":["The closed-form stationary density as written, $f(r)\\propto(\\sigma_b^2+\\sigma_w^2 r^2/\\pi)^{-d/2}$, does not integrate to finite total probability in $d$ dimensions (the volume element leaves a logarithmically divergent tail), so a strict reading is that the heavy-tailed result is a transient or cutoff-dependent statement rather than a true stationary distribution; the $d=2$ and $\\sigma_b=0$ speci","Because the diffusion coefficient grows as $\\|s\\|^2$ for ReLU networks, the process is increasingly erratic far from the origin; in bounded environments this means the heavy tail will hit boundaries, and the 'exploration' may overshoot—a testable consequence is the first-passage time through a narrow doorway as a function of $\\sigma_w$.","The per-step resampling scheme is equivalent to a particular stochastic process on weights; connecting it to parameter-reset training procedures could turn 'primacy bias' mitigation into a quantitative exploration-design tool.","The argument suggests that an architecture search over activations could be performed analytically at initialization by comparing kernels, before any environment interaction."],"forward_implications":["If the claim holds, a practitioner can choose exploration behavior by choosing activation, width, and initialization variance, with no training required.","Per-step policy resampling becomes a simple, reward-free exploration strategy that produces fat-tailed state visitation in free space.","Hybrid schedules (fixed policy for $n$ steps, then resampling) give a tunable trade-off between directional escape and diffusive coverage, with $n$ bounded by the inverse Lipschitz constant and environment scales.","The framework bridges network kernel properties—stationary versus non-stationary—and trajectory geometry: stationary kernels produce uniform diffusion, non-stationary ones produce distance-dependent exploration.","The same Gaussian-process analysis can be applied to other activations (tanh, periodic, and others) to predict their exploration priors before running any RL loop."],"supporting_citations":[{"why":"Supplies the theorem that infinite-width neural networks converge to a Gaussian process with an architecture-dependent kernel, the foundation for the action covariance.","marker":"[11]"},{"why":"Provides the closed-form ReLU kernel and the discussion of stationary versus non-stationary kernels used to compute the diffusion coefficient.","marker":"[9]"},{"why":"Provides a method for computing the Lipschitz constant of ReLU networks, making the ballistic bound practically implementable.","marker":"[1]"},{"why":"Gives another Lipschitz-constant estimation method cited for the same purpose of bounding the fixed-policy trajectory.","marker":"[7]"}],"fun_headline_variants":["Ballistic or diffusive: untrained RL policies pick a path","Untrained nets walk straight or roam: architecture decides","RL exploration style is set before training begins","Fixed net walks straight; resampled net scatters","Policy initialization dictates early exploration motion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The heavy-tailed steady-state conclusion rests on the assumption that the per-step resampled policy process has a well-defined, normalizable stationary distribution over states, but the paper's own closed-form density does not satisfy that condition except in special cases.","fun_headline_variants_meta":{"raw":{"variants":["Ballistic or diffusive: untrained RL policies pick a path","Untrained nets walk straight or roam: architecture decides","RL exploration style is set before training begins","Fixed net walks straight; resampled net scatters","Policy initialization dictates early exploration motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1240,"prompt_tokens":884,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":500,"tokens_out":356,"duration_ms":6600,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:04:03.779807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the per-step resampled ReLU policy under linear dynamics $s_{t+1}=s_t+\\pi(s_t)$ in two and three dimensions with $\\sigma_b>0$, record a long trajectory, and compare the empirical radial density to $f(r)\\propto(\\sigma_b^2+\\sigma_w^2 r^2/\\pi)^{-d/2}$; if the distribution is not heavy-tailed with that exponent, or if the total mass inside a large ball keeps growing with time, the claimed steady state fails.","supporting_citations":[{"cited_title":"Periodic activation functions induce stationar- ity","cited_arxiv_id":null,"evidence_quote":"Provides the closed-form ReLU kernel and the discussion of stationary versus non-stationary kernels used to compute the diffusion coefficient."},{"cited_title":"Lipbab: Computing exact lipschitz constant of relu networks","cited_arxiv_id":null,"evidence_quote":"Provides a method for computing the Lipschitz constant of ReLU networks, making the ballistic bound practically implementable."},{"cited_title":"Lipschitz constant estimation of Neural Networks via sparse polynomial optimization","cited_arxiv_id":"2004.08688","evidence_quote":"Gives another Lipschitz-constant estimation method cited for the same purpose of bounding the fixed-policy trajectory."}],"review_version":1}