{"id":"4ceb7eac-df05-4556-b859-b737fead5c29","arxiv_id":"2411.15370","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new incremental actor-critic algorithm, AVG, learns continuous-control policies from single samples using reparameterized gradients plus normalization and TD-error scaling, and works on real robots.","lead":"This paper introduces Action Value Gradient, a deep reinforcement learning method that updates from each new experience immediately, with no replay buffer, target network, or batch update. It reports strong simulated control results and the first real-robot learning runs using only incremental updates, on a UR5 arm and an iRobot Create.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2, the observation-normalization routine that AVG's stability rests on, is internally inconsistent: as printed it divides by a variance and produces an undefined 0/0 on the first sample, so the method described in the paper is not reproducible as written.","rationale":"The paper is fundamentally an empirical contribution: AVG, an incremental reparameterized actor-critic with three normalization/scaling techniques, is claimed to be the first incremental deep RL method that learns effectively on continuous control and on real robots. The reader's weakest assumption was that the normalization techniques might introduce bias or interact badly with adaptive scaling, a concern the convergence proof in Appendix I does not cover. My concern is more basic and more concrete: the printed observation-normalization routine is internally inconsistent and, taken literally, cannot be executed. This directly affects a load-bearing component of the central claim, because observation normalization is one of the three mechanisms credited for AVG's stability. I do not see evidence of bad faith: the authors share code, report detailed hyperparameters, and present broad ablations. The right posture is therefore not rejection but a conditional requirement that the algorithm specification be reconciled with the implementation, and that a from-scratch reader of the paper be able to reproduce the reported learning behavior. Because the reader's verdict is already CONDITIONAL, my read does not move the verdict; it sharpens one condition that should be checked before the strongest claims are accepted.","tokens_in":43906,"tokens_out":19380,"duration_ms":190105,"concrete_test":"Run the released Normalize function on the first observation of Ant-v4 starting from n=0. If it returns NaN, the implementation matches the pseudocode and AVG as printed cannot train at all. If it returns a finite value, inspect whether the implementation uses sqrt(M2/n) and an epsilon, and then re-run Ant-v4 for 1M timesteps using a from-scratch implementation that strictly follows Algorithm 2 to check whether the published algorithm, rather than the code variant, actually learns.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that AVG's three stabilizers — observation normalization, penultimate normalization, and TD-error scaling — make incremental deep policy gradient learning work. The weakest load-bearing link is the formal specification of the first stabilizer. In Algorithm 2, after the Welford update, the routine sets sigma to the biased variance (M2/n) and then returns Xnorm = delta2 / sigma. On the first observation, n=1, delta2=0, sigma=0, so Xnorm is 0/0 (NaN). Algorithm 1 immediately assigns this returned value to the current state S, meaning a literal implementation of the printed algorithm would feed NaN into the policy and critic on the first timestep and fail. In addition, Algorithm 3 squares the quantity returned as sigma_R, treating it as a standard deviation, while Algorithm 2 defines it as a variance; the two uses are inconsistent. If the released code inserts a square root and an epsilon, then the paper's pseudocode does not describe the algorithm that actually produced the reported results. If it does not, the reported results cannot be reproduced from the printed specification. Either way, a central stability mechanism of AVG is not pinned down by the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Action Value Gradient (AVG), an incremental deep policy gradient method that updates online from the most recent transition without a replay buffer, minibatch, or target network. AVG combines the reparameterized policy gradient with entropy regularization and three stabilizers: Welford-style observation normalization, penultimate-layer normalization, and TD-error scaling. The authors evaluate AVG against IAC, SAC-1, TD3-1, and full batch methods on MuJoCo/Gymnasium tasks, DeepMind Control Suite tasks, a visual reacher, and two physical robots, and they provide a convergence analysis for a simplified variant called RPG-TD in Appendix I. The main claims are that AVG is the only incremental method that learns effectively and that it enables the first effective real-robot incremental deep RL demonstration.","tokens_in":44077,"tokens_out":6058,"duration_ms":56562,"significance":"The empirical program is in many ways exemplary: 30 seeds, 10M-timestep runs, a 300-configuration random-search protocol, ablations of all three stabilizers, learning-from-pixels experiments, and real-robot demonstrations with reported computation times. The public code and videos are valuable assets. If the central claims hold, the paper makes a useful contribution to resource-limited and on-device reinforcement learning. However, two load-bearing issues currently undercut the strongest claims: the printed normalization routine is internally inconsistent, and the paper's own Figure 7 contradicts the abstract's 'only incremental method' assertion. The theoretical appendix, while transparent about its scope, analyzes a simplified variant rather than AVG itself. These problems are fixable and do not invalidate the empirical core, but they must be addressed before the paper can be accepted.","major_comments":[{"comment":"The normalization routine as printed is internally inconsistent. After the Welford update, Algorithm 2 sets sigma to the biased variance M2/n and returns Xnorm = delta2/sigma. On the first observation, n=1, delta2=0, and sigma=0, so Xnorm is 0/0 and Algorithm 1 would feed NaN into both the policy and the critic on the first timestep. In addition, Algorithm 3 treats the value returned as sigma_R as a standard deviation and squares it, while Algorithm 2 defines it as a variance; Schaul-style return scaling requires the variance itself, not its square. If the released code contains a square root and an epsilon, then the pseudocode does not describe the algorithm that produced the reported results; if it does not, the printed method is not reproducible. Because observation normalization is one of the three stabilizers on which AVG's stability claim rests, the pseudocode and text must be corrected to match the actual implementation.","section":"§3, Algorithms 2–3"},{"comment":"The abstract's claim that AVG is 'the only incremental method that learns effectively' is contradicted by the paper's own results. In Figure 7, IAC+, which is still an incremental method after receiving the same normalization and scaling techniques, 'performs in a mostly comparable manner to AVG,' as stated in Section 5.2. If the intended uniqueness claim is only about the raw IAC baseline without the proposed stabilizers, that should be stated explicitly and the abstract reworded; as written, the claim overstates what the experiments demonstrate.","section":"Abstract and §5.2, Figure 7"},{"comment":"The convergence analysis is proved for RPG-TD, a simplified variant that uses i.i.d. transition samples, has no entropy regularization, and does not include the normalization and scaling techniques. The authors explicitly acknowledge this does not match AVG's single-stream incremental setting. Yet the main text presents this analysis as 'On the Theory of AVG' and claims the first convergence result for model-free methods using the reparameterization gradient estimator. That theoretical claim should be qualified so that it is not read as a convergence guarantee for the actual AVG algorithm. Since the paper's primary contribution is empirical, this is not fatal, but the theory section currently promises more than the theorem delivers.","section":"Appendix I and 'On the Theory of AVG'"}],"minor_comments":[{"comment":"The notation in Algorithm 2 uses the same symbol for the running mean and for the sum of squared differences M2, which makes the Welford update hard to follow; distinct symbols such as bar-mu and M2 should be used and defined.","section":"§3, Algorithm 2"},{"comment":"The SAC reference line is trained for 1M timesteps while AVG and the other incremental curves are shown for 10M timesteps; the text saying AVG achieves performance 'comparable to SAC' should acknowledge the different training budgets.","section":"§4, Figure 3 caption"},{"comment":"The caption says the learning curves are for 'the best seed,' but the surrounding text describes a random search over hyperparameter configurations; the caption should say 'best hyperparameter configuration' to avoid confusion.","section":"§5.2, Figure 6 caption"},{"comment":"PPO does not use a replay buffer, so the x-axis label 'Replay Buffer Size' for PPO is misleading; the paper should define what is reduced for PPO (for example, update horizon or minibatch size) so the buffer-size comparison is meaningful across all three methods.","section":"§1, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is substantial and likely publishable after revision. The two main fixes are to align the printed normalization pseudocode with the actual code and to qualify the 'only incremental method' claim in the abstract in light of Figure 7. If the released code is verified to contain the square root and the epsilon guard, this is a straightforward major-revision case rather than a rejection; the theory section should also be scoped explicitly to RPG-TD."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. First, this is a substantial empirical study of an incremental reparameterized actor-critic (AVG) with three stabilizers, and it mostly delivers: 30-seed runs, ablations, real-robot demos, code, and honest limitations. Second, the printed Algorithm 2 is wrong: it divides the deviation δ2 by the variance σ, and on the first sample that is 0/0, so a literal implementation feeds NaN into the policy. Algorithm 3 then treats the output σ as a standard deviation while Algorithm 2 defines it as a variance. The released code may fix this with a square root and epsilon, but the manuscript does not describe the algorithm that produced the results. That is a genuine reproducibility defect in the paper's central stability mechanism, not a cosmetic typo.\n\nWhat is new: AVG is a sensible combination—reparameterized policy gradient, learned critic, same action for both updates, no replay buffer or target network—plus observation normalization, penultimate normalization, and TD-error scaling. The empirical package is strong: six Gymnasium/DMC tasks, sparse-reward dog tasks, a visual reacher, and first-of-their-kind incremental deep RL runs on two robots. The ablation study cleanly shows all three stabilizers matter. The corrected and extended convergence analysis for reparameterized policy gradients (Appendix I) is a useful contribution, though it is for RPG-TD, a simplified variant, and the authors say so.\n\nThe soft spots, in order. (1) The pseudocode bug above. A referee must ask for corrected algorithms and a check against the code. (2) The headline claim that AVG is 'the only incremental method' is contradicted by the paper's own Figure 7: IAC with the same normalization and scaling (IAC+) is mostly comparable. The concurrent streaming method of Elsayed et al. (2024b) is cited but not benchmarked. (3) Hyperparameters are per-task selected via large random search; the limitations section acknowledges sensitivity. (4) The convergence proof's assumptions (i.i.d. transitions, no entropy or normalization) do not cover AVG itself; the authors are transparent about this, but it means theory is not load-bearing for the main claims.\n\nOverall, the central empirical claim—with the stabilizers, an incremental RG actor-critic can learn across a range of continuous-control tasks and on robots—appears supported, modulo the need to verify that the actual code matches the intended algorithm. The 'only' claim should be softened. This paper deserves a serious referee; I would send it to review and ask for a corrected pseudocode, a comparison against concurrent streaming methods, and a sharper statement of what is and is not covered by the theory.","headline":"Solid empirical work on incremental deep policy gradient learning, but the printed Algorithm 2 has a 0/0 divide and inconsistent variance/standard-deviation usage, so the method as written is not reproducible.","tokens_in":44696,"tokens_out":3211,"would_cite":true,"duration_ms":27696,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a deep policy gradient method called Action Value Gradient (AVG) can learn continuous-control policies from a single stream of experience, with no replay buffer, batch update, or target network, and can run on real…","keywords":["action value gradient","incremental reinforcement learning","reparameterization gradient","deep policy gradient","real-time robot learning","replay-free learning","observation normalization","TD error scaling"],"falsifier":"Train AVG on a standard continuous-control task after multiplying all rewards by a factor of 100, and also in a version where the reward scale shifts mid-training, while recording the critic gradient norm and episodic return; if learning diverges or collapses despite the normalization stack in either setting, the paper's central claim about stability would be falsified.","tokens_in":43622,"feed_emoji":"🤖","tokens_out":7590,"duration_ms":62868,"temperature":0.7,"pith_summary":"The paper tries to establish that deep reinforcement learning for continuous control does not require the usual stabilizing machinery of large replay buffers, minibatch updates, and target networks. It introduces Action Value Gradient (AVG), an incremental actor-critic that updates from the most recent transition only, using the reparameterization gradient and three normalization and scaling techniques. On standard continuous-control benchmarks, the authors report that AVG is the only incremental method that learns effectively, often reaching final performance comparable to batch soft actor-critic. On two physical robots, they report the first effective deep reinforcement learning using only incremental updates. If true, this would make real-time, on-device deep RL feasible on memory-limited hardware.","feed_headline":"Deep RL learns on real robots with no replay buffer","feed_subtitle":"One-step updates match batch deep RL on benchmarks and run onboard a robot.","key_machinery":"The load-bearing object is a three-part stabilization stack. Observation normalization maintains running mean and variance online with Welford's algorithm; penultimate normalization projects the network's penultimate-layer features to a unit vector; and TD-error scaling divides each temporal difference error by $\\sigma_\\delta$ with $\\sigma_\\delta^2 = \\mathbb{V}[R] + \\mathbb{V}[\\gamma]\\mathbb{E}[G^2]$, computed from online statistics. The update itself is the reparameterized, squashed-normal policy gradient with sample entropy regularization, and the same sampled action is used for both the critic and actor updates. This stack is what replaces replay buffers, batch updates, and target networks.","core_discovery":"The paper's central claim is that an incremental, single-sample actor-critic can be made stable enough to learn deep continuous-control policies, provided the update uses the reparameterization gradient and a specific set of normalization and scaling tricks. The actor update is the entropy-regularized action-value gradient $\\nabla_\\theta J(\\theta) \\propto \\mathbb{E}_{S\\sim d_{\\pi,\\gamma}, A\\sim\\pi_\\theta}[\\nabla_\\theta f_\\theta(\\xi; S)|_{\\xi=h_\\theta(A;S)} \\nabla_A (q_{\\pi_\\theta}(S,A) - \\eta \\log \\pi_\\theta(A|S))]$, with the same action reused for the critic update. The paper demonstrates empirically that AVG is the only incremental method tested that avoids catastrophic failure on the benchmarks, and that removing target networks does not hurt it. The convergence analysis in the appendix is for a simplified variant called RPG-TD and is not a proof for AVG itself.","pith_inferences":["The results suggest that the instability commonly blamed on missing replay buffers and target networks may be addressable at the level of gradient and target scaling, a hypothesis the paper tests only for one architecture and one set of benchmarks.","A natural extension, flagged by the authors as future work, is to combine AVG with eligibility traces to close the sample-efficiency gap to batch methods; that would make the approach more competitive on harder tasks.","Because the convergence theorem applies only to RPG-TD with i.i.d. samples and no normalization, a cautious reader should treat the theorem as support for the reparameterization estimator, not as a guarantee for AVG's normalization stack.","The reported sensitivity to hyperparameters suggests that AVG's practical success may depend on per-task tuning; a testable extension would be an automated or adaptive scheme for the entropy coefficient and learning rates."],"forward_implications":["AVG learns from a single stream of experience, so memory use collapses to the network weights and a few running statistics, making onboard learning practical on devices like a Jetson Nano.","Batch methods SAC, PPO, and TD3 degrade catastrophically when their replay buffer is shrunk toward size 1, whereas AVG succeeds with a buffer of exactly 1, i.e., no buffer at all.","Target Q-networks provide no benefit for AVG; only near-instant Polyak averaging ($\\tau$ close to 1) matches performance, so they can be dropped without cost.","The normalization and scaling stack also helps other incremental methods, with IAC+ performing comparably to AVG on several tasks, though SAC-1+ and TD3-1+ remain unreliable.","The robot experiments demonstrate for the first time effective deep reinforcement learning on real robots using only incremental updates, on a manipulator and a mobile robot."],"supporting_citations":[{"why":"Supplies the reparameterization policy gradient theorem that AVG's update is built on.","marker":"Lan et al. 2022"},{"why":"Source of the TD-error scaling rule used as scaled_td in AVG.","marker":"Schaul et al. 2021"},{"why":"Provides the online mean and variance algorithm used for observation normalization.","marker":"Welford 1962"},{"why":"Proposed the penultimate-layer feature normalization AVG uses as pnorm.","marker":"Bjorck et al. 2022"},{"why":"Defines the incremental one-step actor-critic (IAC) baseline AVG is compared against.","marker":"Sutton and Barto 2018"},{"why":"Defines SAC, the batch off-policy baseline whose entropy-augmented objective and squashed normal policy AVG adapts.","marker":"Haarnoja et al. 2018"},{"why":"PPO baseline that motivates the observation normalization component norm_obs.","marker":"Schulman et al. 2017"},{"why":"TD3 baseline, a reparameterization-based batch method whose target-network mechanism AVG drops.","marker":"Fujimoto et al. 2018"},{"why":"Convergence analysis for deterministic policy gradient that the appendix extends to reparameterized policies (RPG-TD).","marker":"Xiong et al. 2022"},{"why":"Provides the two real-robot task setups, UR-Reacher-2 and Create-Mover, used for the physical experiments.","marker":"Mahmood et al. 2018"}],"fun_headline_variants":["Incremental deep RL learns robots without replay","Single-sample updates make deep RL run on robots","AVG: deep policy gradient with one-step updates","No batch updates, no replay: robot learns anyway","Real robots trained with incremental deep RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The stability of AVG rests on the unproven assumption that the three online normalization and scaling techniques do not distort the reparameterized policy gradient enough to prevent convergence; the paper's convergence theorem is proven only for a simplified variant with i.i.d. transition samples, no entropy regularization, and no normalization, so nothing in the paper guarantees AVG will learn on new tasks or reward scales.","fun_headline_variants_meta":{"raw":{"variants":["Incremental deep RL learns robots without replay","Single-sample updates make deep RL run on robots","AVG: deep policy gradient with one-step updates","No batch updates, no replay: robot learns anyway","Real robots trained with incremental deep RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1236,"prompt_tokens":879,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":495,"tokens_out":357,"duration_ms":3967,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:22:05.692849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train AVG on a standard continuous-control task after multiplying all rewards by a factor of 100, and also in a version where the reward scale shifts mid-training, while recording the critic gradient norm and episodic return; if learning diverges or collapses despite the normalization stack in either setting, the paper's central claim about stability would be falsified.","supporting_citations":[],"review_version":1}