{"id":"c32348a3-856d-454c-9e7e-ef025d75ab3f","arxiv_id":"2505.08376","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An Adam-style adaptive optimizer with Katyusha momentum is applied to six diffusion-policy RL methods and reported to beat or match baselines on six robotic control tasks.","lead":"This paper proposes ADPO, a framework that swaps the optimizer inside six diffusion-policy reinforcement learning methods for a customized Adam-style update with acceleration momentum, and reports better or comparable performance on GYM and ROBOMIMIC robot tasks. A generalist reader might care because it explores whether a simple optimizer change can make diffusion-based robot policy fine-tuning faster and more stable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADPO's reported gains may be an artifact of an unspecified inner loop of T gradient steps per update, since T is also the environment horizon; the algorithm is under-specified (rho, lambda, T), so the advantage cannot be attributed to ADAPG.","rationale":"The paper's central claim is empirical: replacing the policy optimizer with ADAPG accelerates diffusion-based RL and improves stability or final performance across six baselines and six tasks. For this claim to hold, the only difference between each ADPO variant and its baseline should be the update rule; all other compute, gradient steps, and hyperparameters should be matched. The manuscript fails this condition in at least two concrete ways. First, Algorithm 1 wraps every actor update in an inner loop 'for i = 1 to T do', and T is the same symbol as the environment horizon in Table III. If the inner loop actually runs T gradient steps per batch, ADPO methods consume many more optimizer steps than the baselines, so any empirical gain is confounded. Second, the gradient fed into ADAPG is written as g_i = grad_theta V^{pi_theta}(rho) with rho undefined, and no derivation shows how this follows from the actor losses in Eqs. (10), (13), (16), (17), (21), and (22). The ADAPG update in Eq. (24) also contains lambda and beta1/beta2 that are never specified, so the algorithm cannot be reproduced as written. Table II further tunes omega per task and method, which weakens the comparison further. The reader's rejection is therefore justified: the central claim is not adequately supported. A controlled run with matched inner-loop counts is the decisive check, and until that is done the paper should not be treated as evidence for the claimed acceleration.","tokens_in":12041,"tokens_out":4996,"duration_ms":54129,"concrete_test":"Obtain the authors' code or reimplement the method, then run ADPPO versus DPPO on Hopper-v2 under two conditions: (i) exactly as Algorithm 1 is written with the inner-loop count set to T (1000), and (ii) with the inner-loop count fixed to 1 for both methods, keeping lambda=0.9, beta1=0.9, beta2=0.999, and all other hyperparameters identical. If the ADPPO advantage over DPPO in Table I disappears or drops below seed noise, the headline claim is an artifact of extra inner updates rather than of the ADAPG update rule.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing problem is that Algorithm 1 does not specify what is being optimized or how many gradient steps are taken. The inner loop 'for i = 1 to T do' uses T, which Table III defines as the environment horizon (1000 for GYM, 300 for ROBOMIMIC). If ADPO methods run T gradient steps per batch while the six baselines use the standard single optimizer update, the Y/-- scores in Table I could reflect extra gradient steps rather than ADAPG. The gradient itself is written as g_i = grad_theta V^{pi_theta}(rho), where rho is never defined and no derivation connects the actor losses in Eqs. (10), (13), (16), (17), (21), (22) to this objective. The ADAPG update in Eq. (24) also contains a free parameter lambda that is never set, along with unspecified beta1 and beta2. Without these values the algorithm is not reproducible, and with T as written the comparison is not controlled. This is a correctness and reproducibility risk in the empirical claim, not merely a missing hyperparameter table.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADPO, an Adam-based adaptive gradient framework for fine-tuning diffusion-model policies in reinforcement learning. It replaces the optimizer step in six diffusion-based RL methods (DPPO, DIPO, IDQL, DAWR, QSM, DQL) with an ADAPG update (Eq. 24) that interpolates between adaptive gradient estimators using a Katyusha-style momentum term. Experiments on Hopper-v2, HalfCheetah-v2, Walker2d-v2, Lift, Can, and Square compare the six ADPO variants against the original methods, reporting a qualitative Y/-- comparison in Table I and a hyperparameter sensitivity study for epsilon and omega. The paper claims faster and more stable training with better or comparable final performance.","tokens_in":12275,"tokens_out":5572,"duration_ms":50855,"significance":"The problem is timely: diffusion policies are expressive but notoriously hard to optimize in RL, and a generic optimizer-level drop-in that accelerates several existing methods would be practically useful. The paper also deserves credit for attempting a broad benchmark across six methods and two benchmark suites, and for explicitly studying epsilon and omega sensitivity. However, the empirical evidence as presented does not substantiate the central claim. The algorithm is under-specified (T, rho, lambda, beta1, beta2), the comparison is uncontrolled with respect to inner-loop gradient steps and per-task hyperparameter tuning, and the results are reported without numbers or significance tests. If the missing details and controls are supplied in a revision, the contribution could be of moderate practical value; as written, the load-bearing evidence is missing.","major_comments":[{"comment":"Algorithm 1 is under-specified in a way that undermines the controlled comparison. The inner loop 'for i = 1 to T do' uses T, which Table III defines as the environment horizon (1000 for GYM, 300 for ROBOMIMIC). If ADPO takes T gradient steps per batch while the baselines perform one optimizer update, the gains in Table I could be due to extra gradient steps rather than to the ADAPG rule. The gradient is written as g_i = grad_theta V^{pi_theta}(rho), where rho is never defined and no derivation connects this objective to the actor losses in Eqs. (10), (13), (16), (17), (21), and (22). Furthermore, lambda in Eq. (24) is never assigned a value, and beta1 and beta2 are not specified for the experiments. Please specify T (or rename the horizon), define rho, state lambda, beta1, and beta2, and run a controlled comparison with the same number of inner optimizer steps for ADPO and baselines.","section":"Algorithm 1, Eq. (24), Table III"},{"comment":"The headline empirical claim is supported only by a qualitative Y/-- table without numerical values, error bars, or significance tests. Table I assigns 'Y' or '--' but reports no quantities; Figures 2 and 3 show curves with overlapping shaded regions, and for several pairs (e.g., HalfCheetah ADPPO vs DPPO in Figure 2(c)) the curves are visually indistinguishable. The statement in Section V.B that 'ADPO can significantly improve the performance' is therefore not substantiated. Provide final mean +/- std tables, per-seed results, and a significance test (e.g., paired t-test or bootstrap) for each ADPO-baseline pair.","section":"Section V.B, Table I, Figures 2 and 3"},{"comment":"The comparison is not controlled because omega in Eq. (24) is tuned per task and per method (Table II), and the ablation in Section V.C shows omega is sensitive to the environment. If ADPO's advantage depends on per-task selection of omega while baselines use fixed default hyperparameters, the Y/-- scores could reflect hyperparameter tuning rather than the ADAPG update. Please report the search procedure, show that the same tuning budget is given to baselines, or provide results for a fixed omega.","section":"Table II and Section V.C"}],"minor_comments":[{"comment":"The text refers to 'Figures 1.a and 1.b' but the actual figures are numbered 2 and 3; update the cross-references.","section":"Section V.B"},{"comment":"The text states that GYM tasks are discrete actions, but Hopper-v2, HalfCheetah-v2, and Walker2d-v2 are continuous control tasks.","section":"Section III.C"},{"comment":"Equation (18) writes Q_phi1, Q_phi1 instead of Q_phi1, Q_phi2 for the two Q-networks; fix the typo.","section":"Eq. (18)"},{"comment":"Algorithm 1 initializes Q_phi3 only for double Q learning; since the paper uses double Q-learning, the reference to Q_phi3 is confusing and should be removed or justified.","section":"Algorithm 1"},{"comment":"The sentence 'This indicates that epsilon is robust and doesn't require extensive tuning for different environments' is duplicated verbatim; remove the repetition.","section":"Section V.C"},{"comment":"The line 'update actor policy pi_theta using Eq. (10), (11), (13), (16), (17), (21), (22)' lists all method losses together, but each ADPO variant presumably uses only its own loss; clarify how the correct loss is selected for each method.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"I recommend that the editor ask the authors to provide full code and configuration files, and to clarify or rerun the experiments with a controlled inner-loop step count. The current degree of algorithmic under-specification is unusual for a journal submission and prevents verification of the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the ADAPG update is a sensible minor twist on Adam, and the survey across six diffusion-policy RL methods is a reasonable scope, but the paper's central claim that ADPO accelerates and stabilizes these methods is not supported. The stress-test note lands: Algorithm 1 is under-specified to the point of being unreproducible, and the comparison may be uncontrolled.\n\nThe contribution is genuinely small: Eq. (24) merges Adam's bias-corrected moments with a Katyusha-style extrapolation weight omega. Nothing wrong with that, and applying it to DPPO, DIPO, IDQL, DAWR, QSM, and DQL is a useful test if the experiments were clean. The paper does show sensitivity analyses for epsilon and omega, which is more than many such papers do.\n\nBut the soft spots aren't minor. Algorithm 1's inner loop runs for i=1 to T, and Table III defines T as the environment horizon (1000 for GYM, 300 for ROBOMIMIC). If ADPO takes T gradient steps per batch while the baselines take the usual single step, the reported gains could come from extra compute, not from ADAPG. The gradient term g_i = grad_theta V^{pi_theta}(rho) is undefined: rho never appears anywhere in the paper, and no derivation connects the six actor losses (Eqs. 10, 13, 16, 17, 21, 22) to this value gradient. The discount lambda in Eq. (24) is never given a value. Table I is a qualitative Y/-- table with no error bars or significance tests; Figures 2-3 show curves that overlap substantially. Table II tunes omega per task and per method, with no evidence that baselines received equivalent tuning. No code or data are provided, only a video link.\n\nThese are load-bearing problems because the abstract's claim of significant improvement depends on a controlled comparison that the paper does not document. The underlying idea is plausible and could be a useful practical trick, but as written this is not a reproducible or well-controlled empirical study.\n\nI wouldn't cite it, wouldn't use it in a reading group, and wouldn't send it to referees as is. The right move is to send it back to the authors with a clear request to specify the algorithm, fix the T/lambda ambiguity, run equivalently controlled baselines, and report error bars. If they do that, it could be a minor but valid contribution.","headline":"The ADAPG twist is a plausible minor idea, but the paper's central empirical claim is unsupported because Algorithm 1 is under-specified and the comparison is uncontrolled.","tokens_in":12815,"tokens_out":4354,"would_cite":false,"duration_ms":39258,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the inner-loop optimizer with an adaptive-gradient update rule makes six diffusion-based reinforcement learning methods train faster and more stably on robotic control benchmarks.","keywords":["diffusion policy","reinforcement learning","adaptive gradient methods","policy gradient","robotic manipulation","continuous control","ADAPG","fine-tuning"],"falsifier":"Re-run ADPPO and DPPO on Hopper-v2 with identical inner-loop step counts and identical discount $\\lambda$, changing only whether the inner update is Eq. (23) or Eq. (24); if the reward gap reported in Table I does not appear, the central claim is not supported.","tokens_in":11812,"feed_emoji":"🤖","tokens_out":12130,"duration_ms":104862,"temperature":0.7,"pith_summary":"Diffusion models can represent robot policies as rich, multimodal action distributions, but optimizing those policies with reinforcement learning has been slow and unstable. This paper claims that part of the trouble is the optimizer rather than the objective, and proposes ADPO: a drop-in replacement of the inner-loop policy update with an adaptive-gradient rule called ADAPG. Applying ADPO to six existing diffusion-based RL methods (DPPO, DIPO, IDQL, DAWR, QSM, DQL) is reported to give faster convergence, more stable training, and better or equal final rewards or success rates across three locomotion and three manipulation tasks. A reader should care because this suggests many diffusion-based RL methods could be improved without redesigning their loss functions, merely by changing how each gradient step is computed.","feed_headline":"Adaptive optimizer makes diffusion-policy RL train faster and steadier","feed_subtitle":"A one-line optimizer swap makes six diffusion RL methods train faster and steadier on simulated robot tasks.","key_machinery":"The machinery is the ADAPG update, an adaptive-gradient rule whose step at iteration $i$ uses two running averages, $m_i = \\beta_1 m_{i-1} + (1-\\beta_1) g_i$ and $v_i = \\beta_2 v_{i-1} + (1-\\beta_2) g_i^2$, then forms an intermediate point $h_i = \\theta_{i-1} - \\eta \\frac{(1-\\lambda)g_i + \\lambda m_i}{\\sqrt{v_i}+\\varepsilon}$ and finally sets $\\theta_i = \\omega h_i + (1-\\omega) h_{i-1}$. The discount $\\lambda$ interpolates between the raw gradient and the momentum average, while $\\omega$ blends successive intermediate points, which is intended to damp oscillation and error accumulation from noisy gradients. This update is inserted as the inner loop of Algorithm 1 in place of the AdamW weight-decay update, so the actor and critic loss formulas of the six baseline methods are unchanged; only the optimizer minimizing them changes.","core_discovery":"The paper's central claim is that the update rule in Eq. (24), ADAPG, is a better inner-loop optimizer for diffusion-based reinforcement learning than the AdamW-style update in Eq. (23) that it replaces. In the authors' experiments, every one of the six wrapped methods matches or beats its baseline in the qualitative comparison of Table I; ADPPO, ADIPO, and ADQL are marked above baseline in most of the six tasks, while the remaining variants are on par in some tasks and above in others. The gains are reported as clearest in the harder manipulation tasks, where the ADPO variants preserve training stability and converge faster. The paper presents this as evidence that adaptive policy gradient methods, previously little used in RL, are a generally useful accelerator for diffusion policies.","pith_inferences":["A testable consequence the paper leaves implicit: if ADAPG's benefit comes from better per-gradient adaptation, the gap between ADPO and baseline should widen when gradient noise is high but the number of inner steps is held equal; this could be checked by measuring update variance on a single task.","Because Algorithm 1 does not state the inner-loop step count $T$ or the discount $\\lambda$, part of the reported speedup could come from additional gradient steps per batch; a controlled replication holding $T$ and $\\lambda$ fixed across optimizers would separate the rule's contribution from added computation.","Since Eq. (24) recovers different optimizer behaviors as $\\lambda$ and $\\omega$ vary, ADPO can be viewed as a family of adaptive steps; searching over $(\\lambda,\\omega)$ per task and comparing against tuned AdamW would test whether the benefit is the rule itself or simply better-tuned adaptive steps."],"forward_implications":["Swapping in ADAPG requires no change to the actor or critic loss of the wrapped method, so the framework should port directly to other diffusion-policy RL objectives.","The reported gains are largest on the long-horizon manipulation tasks, suggesting ADPO is most useful when training stability is hardest to maintain.","The stabilizer $\\varepsilon = 10^{-11}$ is reported to work across all tested tasks without tuning, removing one hyperparameter from the pipeline.","The momentum weight $\\omega$ is task- and method-dependent, with best values ranging from $0.6$ to $1.5$, so practical deployment still requires tuning it per setting."],"supporting_citations":[{"why":"Supplies DPPO, one of the six baseline methods; its policy-gradient actor loss and PPO clipping are wrapped by ADPPO.","marker":"[25]"},{"why":"Supplies DAWR, the advantage-weighted-regression baseline wrapped by ADAWR, using TD-bootstrapped advantages in the actor loss.","marker":"[20]"},{"why":"Supplies DIPO, the online diffusion-policy baseline wrapped by ADIPO, with action-gradient updates and a Bellman-residual critic.","marker":"[37]"},{"why":"Supplies IDQL, the implicit Q-learning baseline wrapped by AIDQL, with value-weighted action resampling.","marker":"[14]"},{"why":"Supplies DQL, the diffusion Q-learning baseline wrapped by ADQL, with a Q-weighted denoising objective.","marker":"[34]"},{"why":"Supplies QSM, the Q-score-matching baseline wrapped by AQSM, whose actor loss aligns the policy score with the Q gradient.","marker":"[23]"},{"why":"Provides the Adam/AdamW update that ADAPG replaces; Eq. (23) is the direct comparison baseline inside Algorithm 1.","marker":"[17]"},{"why":"Provides the accelerated momentum scheme that motivates the $\\omega$-blending in the ADAPG update of Eq. (24).","marker":"[3]"},{"why":"Supplies the three locomotion control environments used for the GYM experiments.","marker":"[5]"},{"why":"Supplies the three manipulation task benchmarks (Lift, Can, Square) used for the manipulation experiments.","marker":"[19]"}],"fun_headline_variants":["New optimizer beats AdamW for diffusion-policy RL training","ADPO's ADAPG update: faster, stabler diffusion RL","Swap in ADAPG to boost diffusion-policy RL training speed","Adaptive gradient update key to faster diffusion RL","One-line optimizer swap improves diffusion RL stability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on unstated details of the inner update loop: how many gradient steps are taken, what the discount factor is, and which objective the gradient is computed from, so the improvement could come from those choices rather than from the ADAPG rule itself.","fun_headline_variants_meta":{"raw":{"variants":["New optimizer beats AdamW for diffusion-policy RL training","ADPO's ADAPG update: faster, stabler diffusion RL","Swap in ADAPG to boost diffusion-policy RL training speed","Adaptive gradient update key to faster diffusion RL","One-line optimizer swap improves diffusion RL stability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1682,"prompt_tokens":911,"completion_tokens":771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":690}},"tokens_in":527,"tokens_out":771,"duration_ms":7592,"temperature":1.0,"reasoning_tokens":690,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:56:01.881507+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run ADPPO and DPPO on Hopper-v2 with identical inner-loop step counts and identical discount $\\lambda$, changing only whether the inner update is Eq. (23) or Eq. (24); if the reward gap reported in Table I does not appear, the central claim is not supported.","supporting_citations":[{"cited_title":"Katyusha: The first direct acceleration of stochastic gradient methods","cited_arxiv_id":null,"evidence_quote":"Provides the accelerated momentum scheme that motivates the $\\omega$-blending in the ADAPG update of Eq. (24)."}],"review_version":1}