{"id":"79c3a706-3f8f-44f5-b1e3-14117384ebc8","arxiv_id":"2412.02291","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RAD introduces a growing symplectic factor and a per-parameter relativistic speed limit into an ADAM-like update, with a sublinear convergence guarantee for nonconvex stochastic problems.","lead":"RAD is a new optimizer for reinforcement learning that combines a per-parameter adaptive learning rate, like ADAM, with a physical speed limit inspired by special relativity. It is reported to stabilize training and improve scores on several RL benchmarks, especially Atari games, while reducing to ADAM at a specific setting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's convergence conditions are incompatible with the paper's recommended hyperparameters, leaving the empirical regime outside the proof.","rationale":"The reader identified Assumption 2 (bounded coordinate-wise gradients) as the weakest assumption. I agree that this assumption is questionable for deep RL, but the more precise and more damaging issue is internal to the theorem: the proof's own parameter conditions are incompatible with the default hyperparameters used in the experiments. Even under Assumption 2, with ζ_0 = e^{−12π}, δ = 1, and β2 = 0.999, the condition ζ_0 ≥ 16M^2δ^2(1−β2) requires M ≤ ~5e−8 and the learning-rate condition requires L ≤ ~3e−6 for α = 1e−3. These are not plausible for the reported networks and losses. Therefore the theorem, as stated and proved, does not certify convergence for the configuration that produced the headline empirical results. This strengthens the reader's conditional verdict: the convergence claim needs either a proof that covers the recommended hyperparameters (including a corrected Corollary 5 argument) or an explicit restriction to a parameter regime where the hypotheses can hold. The empirical comparisons may still be valuable, but the theoretical guarantee and the practical configuration are currently disconnected.","tokens_in":39264,"tokens_out":9122,"duration_ms":97647,"concrete_test":"Verify the hypotheses numerically for one released network, e.g., the DQN CNN on Seaquest-v4: estimate M = max_i |[∇L(x,θ)]_i| and L from training logs, then test whether α ≤ √ζ_0/(2L) and β2 ≥ 1 − ζ_0/(16M^2δ^2) hold with ζ_0 = e^{−12π}, δ = 1, β2 = 0.999, and α = 1e−4. If either inequality fails, Theorem 1 does not apply to the reported experiments. An analytic check already shows the first condition requires M ≤ ~5e−8 and α/L ≤ ~3e−9, so with α = 1e−3 one needs L ≤ ~3e−6, which is inconsistent with typical neural-network smoothness constants.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"RAD's central theoretical claim is Theorem 1 (Section V), but the proof's hypotheses are not satisfied by the configuration used in all experiments. Algorithm 3 defines ζ_k = min{ε_k, 1 − β2^{k+1}}, and Section VI.A recommends δ = 1 with κ = 12π in (15), so ζ_0 = e^{−12π} ≈ 4e−17. Theorem 1 requires α ≤ √ζ_0/(2L) and β2 ≥ 1 − ζ_0/(16M^2δ^2). The second inequality is equivalent to ζ_0 ≥ 16M^2δ^2(1−β2). With δ = 1, β2 = 0.999 and ζ_0 ≈ 4e−17, this forces M ≤ ~5e−8. No neural-network loss used in the paper has a coordinate-wise gradient bound anywhere near that small; the reported learning rates α = 1e−3 to 5e−4 likewise violate α ≤ √ζ_0/(2L) unless L is astronomically small. Thus the theorem does not cover the empirical regime. The issue is not merely that bounded gradients (Assumption 2) may fail; even granting Assumption 2, the parameter constraints make the result vacuous for the default setting. Additionally, Corollary 5's proof in Appendix E drops the 1/(k+1) factor in the first part of the sum and then claims a bound of o(N), but the resulting expression is O(N); the conclusion may be salvageable with a sharper estimate, but the written proof is invalid at that step.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes relativistic adaptive gradient descent (RAD), an optimizer derived by discretizing a conformal Hamiltonian system of independent one-dimensional relativistic particles. RAD applies per-coordinate speed limits through a relativistic kinetic energy term, combines them with exponential moving averages and bias correction, and reduces to ADAM when the speed coefficient is one and the symplectic factor is fixed. The paper proves a sublinear convergence bound for RAD under nonconvex stochastic optimization, presents a universal framework connecting conformal symplectic integrators to neural-network updates, and reports empirical gains over nine baseline optimizers across twelve RL environments, including a 155.1% improvement over ADAM on Seaquest-v4.","tokens_in":39574,"tokens_out":8852,"duration_ms":92882,"significance":"The framework is attractive and the derivation is self-contained: the paper gives a concrete recipe for translating conformal Hamiltonian dynamics into iterative optimizers, and the exact algebraic correspondence between RAD and ADAM is a genuine insight. The per-coordinate relativistic formulation is a novel and potentially useful design for stabilizing RL training. If the convergence proof can be repaired so that it actually covers the algorithm as run, and if the recommended hyperparameters are brought inside the theorem's hypotheses, the paper would be a solid contribution. The empirical section, despite relying on a limited number of seeds, supports the practical message that the increasing symplectic factor and speed limit improve training stability on several Atari and MuJoCo benchmarks.","major_comments":[{"comment":"The hypotheses of Theorem 1 are not satisfied by the configuration used in all experiments. Algorithm 3 sets ζ_k = min{ε_k, 1−β2^{k+1}}, and Section VI.A recommends δ=1 with κ=12π in (15), so ζ_0 = exp(−12π) ≈ 4×10^{−17}. With β2=0.999, the condition β2 ≥ 1 − ζ_0/(16M^2δ^2) forces M ≤ sqrt(ζ_0/(16(1−β2))) ≈ 5×10^{−8}, and α ≤ sqrt(ζ_0)/(2L) ≈ 6×10^{−9}/L. The reported learning rates are 10^{−3} to 5×10^{−4}, and no evidence is given that any network loss in the paper has a coordinate-wise gradient bound near 10^{−8}. Thus the O(1/N) bound in Theorem 1 does not apply to the algorithm as run in the experiments, even if Assumption 2 is granted.","section":"Section V (Theorem 1) and Section VI.A"},{"comment":"The theorem statement claims convergence for θ_k generated by Algorithm 3, but the proof explicitly omits bias correction and sets β1=0, whereas Algorithm 3 and all experiments use β1=0.9 and bias correction. The paper asserts without proof that the analysis extends to the general case. In addition, the proof conditions on θ_k rather than the full history F_k; since y_k and θ_k depend on the same past batches, the claim that y_k is independent of B_k is not justified under conditioning on θ_k alone. These gaps are fixable, but the stated theorem currently exceeds what is proved.","section":"Section V (Theorem 1) and Appendix H"},{"comment":"The proof of Corollary 5 is invalid as written. In bounding Σ_{k=0}^{N−1} 1/(B_k√ζ_k), the proof replaces 1/(k+1) by 1 in the first sum, then treats Σ_{k=0}^{N̂−1} e^{κ/2(1−k/N̂)} as part of an expression claimed to be o(N). With the factor dropped, that sum is Θ(N), so the final expression is O(N), not o(N). The conclusion of Corollary 5 may be salvageable by retaining the 1/(k+1) factor and obtaining O(log N), but the written proof does not establish it. Since this corollary provides the specific schedule used in the experiments, the error is load-bearing.","section":"Appendix E (proof of Corollary 5)"},{"comment":"Assumption 2, which bounds every coordinate of every stochastic gradient by a uniform constant M, is very strong for deep RL, where gradient norms can spike and coordinate magnitudes are not uniformly bounded. The proof uses this assumption to control the cross term and to assert 0 ≤ y_{k,i} ≤ M^2. The paper provides no empirical verification of such a bound for the benchmark tasks. Even setting aside the quantitative mismatch in the first comment, the convergence result is conditional on a property that is unlikely to hold for the evaluated losses.","section":"Section V (Assumption 2)"}],"minor_comments":[{"comment":"The remark that RAD degrades exactly to ADAM when δ=1 and ζ is a small positive constant is only exact if that fixed ζ is used from the first iteration; in Algorithm 3, ζ_k = min{ε_k, 1−β2^{k+1}}, so one should state that the equivalence holds when ε_k is constant and 1−β2^{k+1} ≥ ε for all k.","section":"Section IV.C"},{"comment":"On Swimmer-v3, RAD and ADAM both report 32±2 and the improvement column shows 0.0%, so the text's claim that RAD 'consistently demonstrates faster convergence and achieves the highest TAR across all MuJoCo tasks' is not supported by this table.","section":"Table I"},{"comment":"The proof of Corollary 5 uses both N̂ and N-hat notations inconsistently, and Figure 7's caption refers to 'Algorithm IV' when the RAD pseudocode is Algorithm 3.","section":"Appendix E and Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The central issue is that the paper's own recommended hyperparameters place the experimental regime outside the hypotheses of Theorem 1, and the proof of Corollary 5 has a concrete error. These are not cosmetic: the theoretical contribution currently does not cover the algorithm as deployed. However, the underlying idea is promising and the convergence proof is structurally standard, so the issues seem addressable by revising the theorem or the recommended schedule. I would also encourage the authors to state clearly that the proof covers β1=0 and no bias correction, and to either prove or explicitly defer the general case."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes RAD, a per-parameter relativistic adaptive gradient method derived from a conformal Hamiltonian multi-particle system, and tests it on twelve RL environments with five RL algorithms against nine baselines. That empirical sweep is the strongest part of the work: the Atari gains are large, the ablations on the speed coefficient and symplectic factor are informative, and the code is provided. The connection to ADAM as a special case is a nice observation, and the derivation itself is self-contained.\n\nThe soft spot is the theory. The stress-test note is right: with the paper's recommended defaults (δ=1, β2=0.999, and ζ_0 = e^{-12π} ≈ 4e-17 from the Corollary 5 schedule with κ=12π), Theorem 1 requires α ≤ sqrt(ζ_0)/(2L) and β2 ≥ 1 − ζ_0/(16M^2δ^2). The first forces α around 1e-8 or smaller for any reasonable L, and the second forces the coordinate-wise gradient bound M ≤ 5e-8. No neural-network loss used in the paper satisfies that. So the sublinear convergence guarantee is vacuous for the default configuration, and the paper does not flag this mismatch. The proof in Appendix E for Corollary 5 also has a real flaw: it drops the 1/(k+1) factor in the first half of the sum and then claims the total is o(N), but that first sum is O(\\hat{N}), which need not be o(N). The result might be salvageable with a sharper argument or an extra assumption on \\hat{N}, but the written proof is invalid at that step.\n\nThat said, the empirical contribution stands on its own. Many optimizer papers have proofs under assumptions that are not checked in practice, and here the experiments are extensive and the improvements on Atari are striking. The MuJoCo gains are modest (0.8%–5.4% in SAC), which is a fair caveat but not a fatal one. The paper should not be desk-rejected; it deserves a serious referee. The referee should ask the authors to either adjust the theorem's conditions to cover the practical regime, or explicitly reposition the convergence result as holding under idealized assumptions that the recommended settings do not meet. The Corollary 5 proof needs to be fixed or the claim weakened. With those revisions, this could be a useful contribution to the RL optimization literature.","headline":"A promising RL optimizer with a broad empirical study, but the main convergence theorem does not actually cover the recommended hyperparameters used in the experiments.","tokens_in":40110,"tokens_out":4889,"would_cite":false,"duration_ms":48405,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAD derives a stable RL optimizer from relativistic multi-particle dynamics, proving sublinear convergence, and contains ADAM as a limiting case.","keywords":["conformal Hamiltonian system","symplectic integrator","relativistic gradient descent","adaptive optimization","reinforcement learning","nonconvex stochastic optimization","training stability","ADAM"],"falsifier":"Track the maximum per-coordinate gradient norm across a long RL run (e.g., Seaquest-v4) and check whether it stays below the paper's M while the average squared gradient norm decays as O(1/N). If gradients exceed any finite M but the observed decay still matches the bound, the proof's assumption is unnecessary; if the decay stops precisely when gradients spike, the assumption is load-bearing. Alternatively, run RAD on a synthetic nonconvex problem with heavy-tailed noise and compare its empirical convergence against the O(1/N) prediction.","tokens_in":39089,"feed_emoji":"⚛️","tokens_out":6927,"duration_ms":65991,"temperature":0.7,"pith_summary":"This paper proposes RAD, an optimizer for deep reinforcement learning that treats each trainable weight as a tiny relativistic particle moving under a damped Hamiltonian force. By discretizing this conformal Hamiltonian system with a symplectic integrator, RAD caps how fast any parameter can move per step, which the authors argue suppresses the wild gradient spikes that destabilize RL training. The paper proves that RAD converges sublinearly to a stationary point for general nonconvex stochastic objectives, provided each gradient coordinate stays bounded, and that smaller gradient variance and larger batches tighten the bound. It also shows that with speed coefficient one and a fixed small symplectic factor, RAD reduces exactly to ADAM, making ADAM a special case of the relativistic update. Across five RL algorithms and twelve environments, RAD reports consistently higher policy returns than nine baselines, including up to 155.1% improvement over ADAM on Seaquest-v4.","feed_headline":"Beats ADAM by 155% in Atari RL","feed_subtitle":"Caps each weight's update speed, stabilizing RL training and proving sublinear convergence.","key_machinery":"The central object is the conformal symplectic integrator applied to a multi-particle relativistic Hamiltonian. A conformal Hamiltonian system adds damping $-rp$ to the canonical equations, so phase-space area contracts as $e^{-rt}$; a conformal symplectic integrator preserves this contraction after discretization. The paper's update rule (Eq. 13) is the integrator's discrete map, with the symplectic factor $\\zeta_k = \\min\\{\\epsilon_k, 1-\\beta_2^{k+1}\\}$ controlling how much of the relativistic normalization survives: it starts tiny, mimicking ADAM, then anneals upward to restore the dissipative symplectic flow. The speed coefficient $\\delta$ sets the per-step cap $|\\theta_{k+1,i}-\\theta_{k,i}| \\leq \\alpha/\\delta$, which is the relativistic speed limit that suppresses abnormal gradients.","core_discovery":"RAD's central claim is that long-term training stability in NN optimization can be engineered by constructing the optimizer as a conformal symplectic integrator of a relativistic multi-particle Hamiltonian. Each parameter $\\theta_i$ is an independent one-dimensional particle with momentum $p_i$, and the Hamiltonian is $H = \\sum_i c\\sqrt{p_i^2 + m^2 c^2} + J(\\theta)$. Discretizing the conformal equations with a first-order symplectic Euler map and adding exponential-moving-average momenta with bias correction yields the update $\\theta_{k+1,i} = \\theta_{k,i} - \\alpha \\frac{\\sqrt{1-\\beta_2^{k+1}}}{\\sqrt{\\delta^2 y_{k+1,i} + \\zeta_k}} \\frac{v_{k+1,i}}{1-\\beta_1^{k+1}}$, where the symplectic factor $\\zeta_k$ grows from near zero to $1-\\beta_2^{k+1}$. This factor is the paper's key addition: early in training it keeps the algorithm close to ADAM for fast progress, while later it grows to restore symplecticity and long-term stability. The paper proves that under smoothness, per-coordinate bounded gradients, and bounded variance, the expected squared gradient norm averages to $O(1/N)$, and that choosing $\\delta=1$ with $\\zeta=\\epsilon$ recovers ADAM exactly, so the convergence theorem also gives ADAM a convergence guarantee.","pith_inferences":["The bounded-gradient assumption (Assumption 2) is likely violated in real deep RL, where gradients are heavy-tailed; if so, the $O(1/N)$ proof does not transfer directly, although the normalization step may still limit the damage in practice. This is an editorial extension, not a claim of the paper.","The interpretation of $\\zeta$ as a mass-energy term suggests a systematic hyperparameter law: choosing $\\zeta$ roughly proportional to a fraction of the current second-moment scale should stabilize training without sacrificing adaptivity; we would test this by sweeping $\\zeta$ schedules on a single Atari game.","The same conformal-symplectic recipe might also tame optimization in non-RL settings with heavy-tailed gradients, such as large language model fine-tuning, where abnormal gradient spikes are known to occur.","RAD's connection to ADAM hints that the small rational factor $\\epsilon$ in ADAM is not merely a numerical safeguard but an implicit symplectic factor; if true, simply raising $\\epsilon$ in ADAM should improve its stability, which the paper's Hopper ablation supports."],"forward_implications":["If RAD's claims hold, RL practitioners can replace ADAM with RAD at essentially no memory cost (only the same two momenta) and get flatter, more stable learning curves plus higher final returns.","Because ADAM is the $\\delta=1$, $\\zeta=\\epsilon$ limit of RAD, any improvement RAD shows on a task can be read as evidence that the growing symplectic factor, not the momentum scheme, is restoring stability.","The theorem guarantees $O(1/N)$ average squared gradient norm under bounded-gradient and bounded-variance assumptions; increasing batch size tightens the constant, giving a practical lever for harder RL problems.","The two-stage behavior (fast ADAM-like early search, symplectic late refinement) offers a principled alternative to heuristic switches from ADAM to SGD.","RAD's derivation framework extends to other kinetic energies and higher-order integrators, so the same conformal-symplectic recipe can generate new optimizers with built-in stability."],"supporting_citations":[{"why":"Supplies the conformal symplectic integrators and dissipative-Hamiltonian framework that RAD's update rule is built from.","marker":"[22]"},{"why":"Provides RGD and the relativistic kinetic energy constraint that RAD extends to multiple particles.","marker":"[26]"},{"why":"Defines ADAM, the algorithm RAD reduces to and the primary baseline it outperforms.","marker":"[18]"},{"why":"Supplies bias-correction and convergence theory for adaptive methods in nonconvex settings, which RAD's analysis extends.","marker":"[31]"},{"why":"Provides the symplectic Euler and leapfrog integrators used in the discretization.","marker":"[30]"},{"why":"Supplies the dynamical and symplectic perspective linking gradient methods to dissipation.","marker":"[21]"}],"fun_headline_variants":["Relativity-inspired optimizer beats ADAM by 155% in Atari","RAD optimizer: caps weight speed, stabilizes RL training","New optimizer with relativistic updates trounces ADAM","Physics-based optimizer proves sublinear convergence, wins Atari","RAD: conformal symplectic optimizer outpaces ADAM by 155%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"RAD's convergence proof assumes that every coordinate of every stochastic gradient that appears during training stays within a fixed bound M, an assumption that can fail when RL gradients spike to arbitrarily large values.","fun_headline_variants_meta":{"raw":{"variants":["Relativity-inspired optimizer beats ADAM by 155% in Atari","RAD optimizer: caps weight speed, stabilizes RL training","New optimizer with relativistic updates trounces ADAM","Physics-based optimizer proves sublinear convergence, wins Atari","RAD: conformal symplectic optimizer outpaces ADAM by 155%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1439,"prompt_tokens":1105,"completion_tokens":334,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":721,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":721,"tokens_out":334,"duration_ms":4225,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:38:10.918519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the maximum per-coordinate gradient norm across a long RL run (e.g., Seaquest-v4) and check whether it stays below the paper's M while the average squared gradient norm decays as O(1/N). If gradients exceed any finite M but the observed decay still matches the bound, the proof's assumption is unnecessary; if the decay stops precisely when gradients spike, the assumption is load-bearing. Alternatively, run RAD on a synthetic nonconvex problem with heavy-tailed noise and compare its empirical convergence against the O(1/N) prediction.","supporting_citations":[{"cited_title":"On dissipative symplectic integration with applications to gradient-based optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the conformal symplectic integrators and dissipative-Hamiltonian framework that RAD's update rule is built from."},{"cited_title":"Conformal symplectic and relativistic optimization,","cited_arxiv_id":null,"evidence_quote":"Provides RGD and the relativistic kinetic energy constraint that RAD extends to multiple particles."},{"cited_title":"Adam: A method for stochastic optimization,","cited_arxiv_id":null,"evidence_quote":"Defines ADAM, the algorithm RAD reduces to and the primary baseline it outperforms."},{"cited_title":"Adaptive methods for nonconvex optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies bias-correction and convergence theory for adaptive methods in nonconvex settings, which RAD's analysis extends."},{"cited_title":"Hairer, G","cited_arxiv_id":null,"evidence_quote":"Provides the symplectic Euler and leapfrog integrators used in the discretization."}],"review_version":1}