{"id":"a399ae68-d58b-4297-aec7-571c77077223","arxiv_id":"2501.14513","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ABPT averages a zero-step value gradient with an N-step backpropagation gradient so that non-differentiable reward components do not fully block policy learning.","lead":"Robotics researchers propose a training method for quadrotor policies that mixes a model-based gradient with a learned value-function gradient, reducing the damage caused by rewards that are only partially smooth. In simulated hovering, tracking, landing, and racing tasks, the method learns faster and reaches higher rewards than standard alternatives.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ABPT is never compared to AGPO/AOBG, the closest existing gradient-mixing methods, so the claim that it converges faster and achieves higher rewards than existing algorithms is untested against its direct competitors.","rationale":"Read in good faith: the paper proposes a clear, simple idea—optimize J = (J_N + J_0)/2 so the learned Q-value can inject gradient through non-differentiable reward terms—and supports it with ablations and real-world demos. The internal equations mostly hang together, and the empirical story on four tasks is plausible. However, the proof appendix contains a real algebraic error: the direct gradient of (9) is 0.5(∇G + γ^N∇V_end + ∇V_t), which using (17) equals 0.5∇^qG + 0.5∇G + γ^N∇V_end, not the printed 0.5(∇^qG + ∇G + γ^N∇V_end). This does not break the algorithm if Eq. (9) is differentiated directly, but it means the theoretical derivation as printed is inconsistent. The more load-bearing issue for the central claim is that AGPO and AOBG—the closest prior work on mixing value and first-order gradients for non-smooth differentiable simulation—are never included in the comparisons. The paper even concedes AGPO is 'probably effective' and dismisses it on computational-cost grounds without measurement. Because ABPT is essentially a fixed-ratio mixture of the same ingredients AGPO adaptively mixes, the absence of this comparison leaves the headline superiority claim untested. This is a concrete, checkable omission rather than a philosophical disagreement. The reader's weakest-assumption concern about critic accuracy is valid but is partially mitigated by the empirical success of the method across tasks; the missing-baseline issue is not mitigated at all, so I elevate it. The verdict remains conditional: the paper should be accepted only after the AGPO/AOBG comparison is added or the claim is narrowed to the evaluated baselines.","tokens_in":13056,"tokens_out":16386,"duration_ms":141247,"concrete_test":"Run AGPO and AOBG (with released code or faithful reimplementation) on the four VisFly tasks using the same parallel differentiable simulator, horizon, batch size, and hyperparameter tuning budget as ABPT; compare wall-time to reach a fixed reward threshold and final asymptotic reward across 5 seeds. If either method matches or exceeds ABPT's convergence speed or final reward, the paper's central claim that ABPT is superior to existing gradient-mixing algorithms does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that ABPT 'converges significantly faster and achieves higher ultimate rewards than existing learning algorithms'—is not tested against the two existing methods explicitly designed for the same problem of biased first-order gradients in differentiable simulation. AGPO [36] already mixes critic/Q-value gradients with first-order gradients using an adaptive ratio, and AOBG [11] adaptively mixes zeroth- and first-order gradients; both are discussed in related work and Appendix E as 'probably effective' at reducing gradient bias from non-differentiable rewards. Yet neither appears in Figure 4 or in any table, with only a one-sentence assertion that they 'require extremely long training times due to the need for variance estimation at each step.' That is an untested claim, and ABPT's fixed 0.5 mixture could be worse, equal, or better than AGPO's adaptive mixture. Since the paper's novelty and headline result depend on beating existing gradient-mixing methods, omitting them makes the core empirical claim unverifiable. (Secondary: the derivation in Appendix A has an algebraic slip—Eq. (19)'s second line drops a factor of 2 on the terminal-value gradient—but the algorithm as written in Eq. (9) can still be implemented directly.)","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ABPT (Amended Backpropagation through Time), an on-policy actor-critic method for training quadrotor policies when the reward function is only partially differentiable. The core idea is to combine a 0-step return (a value term computed through a learned Q-function) with the N-step return used by first-order gradient methods such as SHAC, and to update the actor by differentiating the averaged objective in Eq. (9). Entropy regularization and a state-only replay buffer are added to improve exploration and sample efficiency. The method is evaluated on four quadrotor tasks (hovering, tracking, landing, racing) in the VisFly simulator, with qualitative real-world demonstrations, and is compared against PPO, BPTT, and SHAC. The paper claims that ABPT converges faster and reaches higher final rewards than these baselines, especially when rewards contain non-differentiable components.","tokens_in":13249,"tokens_out":6802,"duration_ms":61032,"significance":"If the empirical claims hold, ABPT is a useful and simple modification of the SHAC/BPTT pipeline for differentiable-simulation policy learning: it is easy to implement, preserves first-order gradient efficiency, and appears to mitigate the bias caused by non-differentiable reward terms. The paper includes four tasks with varying reward differentiability, ablations of the three proposed components, reward-robustness and learning-rate robustness experiments, and real-world video demonstrations. These are genuine strengths. The significance is moderated, however, by the absence of comparisons with the closest prior gradient-mixing methods (AGPO and AOBG), by an algebraic inconsistency in the central derivation in Appendix A, and by the lack of released code or complete implementation details, which makes the empirical claims hard to verify independently.","major_comments":[{"comment":"The central empirical claim that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms is not tested against AGPO [36] and AOBG [11], the two prior methods explicitly designed for biased or high-variance first-order gradients in differentiable simulation. The manuscript only asserts in §6.2 and Appendix E that these methods 'require extremely long training times due to the need for variance estimation at each step' without reporting any experiment or wall-clock measurement. Since ABPT's update is a fixed equal-weight average of a value gradient and a first-order gradient, and AGPO already mixes Q-gradients with first-order gradients using an adaptive ratio, the claimed superiority over existing gradient-mixing methods is currently unverified. The authors should either add these baselines or explicitly restrict the headline claim to the evaluated baselines.","section":"§6.2, Appendix E"},{"comment":"The bias-correction argument hinges on the unproven approximation G_{t:t+N} ≈ V_t - (1-d)γ^N V_{t+N+1} in Eq. (16). The paper gives no formal statement of the approximation error, no condition under which the critic is accurate enough for the value-gradient term to be reliable, and no bound on the residual bias after the equal-weight average. Furthermore, Eq. (19) is algebraically inconsistent: the first line writes ∇J as \\bar∇G + (1-d)γ^N ∇V_{t+N+1} with \\bar∇G = (∇[q]G + ∇G)/2, while the second line writes it as (∇[q]G + ∇G + (1-d)γ^N ∇V_{t+N+1})/2; these two expressions differ by a factor of two on the terminal-value term. This does not prevent direct implementation of Eq. (9), but it invalidates the derivation as written and should be corrected, together with a precise statement of what gradient is actually being averaged.","section":"Appendix A, Eqs. (16) and (19)"},{"comment":"The empirical evaluation is not reproducible from the manuscript as submitted. The abstract states that code will be released at a GitHub URL, while the main text refers to a placeholder 'Anonymous' link, and no code or supplementary material is provided. Network architectures, critic update schedules, the exact adaptive entropy coefficient computation, the TD-λ target details beyond Eq. (13), and the state-buffer sampling mechanism are not specified. For a paper whose contributions are primarily empirical, these details are necessary for verification; please provide complete implementation details or the actual code.","section":"§6.2, Appendix F"}],"minor_comments":[{"comment":"The sentence 'Though reinforcement learning (RL) can tackle such limitation by self-exploration of agents, it relies on RL relies on zero-order gradient (ZOG) approximations' contains a duplicated phrase 'it relies on RL relies on' and should be reworded.","section":"§1"},{"comment":"The notation '∇[0]θ Jθ' is nonstandard and the definition of the advantage A^{πθ}(s_t, a_t) is not given; please define it explicitly and use a consistent gradient notation throughout.","section":"Eq. (4)"},{"comment":"The first-order gradient formulas omit the expectation over trajectories and the minibatch index; this makes it unclear whether the gradient is computed per trajectory or averaged over the batch.","section":"Eqs. (6) and (7)"},{"comment":"In the Racing row, the observation is described as 'state & next 2 gates' but the task text in Appendix C describes four gates; please clarify whether the policy observes only the next two gates and, if so, how this is reconciled with the task description.","section":"Table 1"},{"comment":"The real-world validation is qualitative only; no quantitative metrics such as success rate, tracking error, or number of trials are reported. If real-world performance is part of the claimed contribution, please add quantitative results.","section":"§6.3 and Figure 1"},{"comment":"The abstract and main text are inconsistent about code availability: the abstract gives a GitHub URL, while the body says the code is 'to be released' at an anonymous placeholder, and the acknowledgments section contains boilerplate text that appears to be from a template; please clean this up.","section":"Abstract and §6.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty relative to AGPO is incremental: ABPT is essentially a fixed equal-weight mixture of a value gradient and a first-order gradient, whereas AGPO adaptively mixes Q-gradients with first-order gradients. Before this paper can be judged, the authors should be required to compare against AGPO and AOBG and to fix the derivation in Appendix A. The absence of code is also a serious concern for an empirically driven paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper proposes ABPT, which is an equal-weight average of a zero-step value gradient and an N-step first-order gradient for policy optimization in differentiable simulators. That specific combination is not in the prior work I know, and the paper does a decent job of explaining why it should reduce bias from non-differentiable rewards. The core idea is straightforward and worth testing.\n\nThe strengths are real: the method is simple, clearly stated, and the ablations show each component (0-step term, entropy, state replay) contributes. The four quadrotor tasks are a reasonable testbed, and the real-world validation, though qualitative, is a plus. The derivation in Appendix A is mostly transparent, but there is a small algebraic slip in Eq. (19) where a factor of 2 is dropped on the terminal-value gradient. Since the algorithm as written in Eq. (9) can be implemented directly, this is a minor issue.\n\nThe main problem is the missing comparison to AGPO and AOBG. These are the two methods explicitly designed to mix value or zeroth-order gradients with first-order gradients in non-smooth settings. The paper dismisses them in a single sentence, claiming they require extremely long training times due to variance estimation, but provides no experimental evidence. That makes the headline claim of 'converges significantly faster and achieves higher ultimate rewards than existing learning algorithms' unverifiable against the methods that matter most. The fixed 0.5 average might be better or worse than an adaptive mixture, and without data we cannot know. The lack of released code compounds this.\n\nThe theoretical justification also rests on the critic being accurate enough in Eq. (16); there is no bias bound or convergence guarantee. This is common in actor-critic papers, but here the bias-correction argument depends on it, so it deserves a formal statement or at least a sensitivity analysis.\n\nAll that said, this deserves a serious referee. The core idea is clear, the experiments are reasonably designed, and the omissions are addressable. I would like to see the authors add AGPO and AOBG to the comparison (even on a subset of tasks), release the code, and fix the algebra. If those are addressed, this becomes a useful incremental contribution for the differentiable-simulation community.\n\nFor a reading group, this is a reasonable paper to look at, though not a must-discuss. I probably would not cite it in my own work until the baselines are added, but it is a legitimate piece of incremental research.","headline":"A simple and plausibly useful gradient-mixing trick for differentiable simulators, but the empirical claim is weakened by missing the two closest baselines and by the absence of released code.","tokens_in":13835,"tokens_out":4587,"would_cite":false,"duration_ms":37480,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ABPT combines the N-step return with a 0-step value gradient so that non-differentiable reward components no longer bias backpropagation-through-time, yielding faster convergence and higher final rewards on quadrotor flight tasks.","keywords":["differentiable simulation","backpropagation through time","partially differentiable rewards","gradient bias","value gradient","quadrotor control","actor-critic","entropy regularization"],"falsifier":"Compare ABPT with plain BPTT on the racing task while scaling up the non-differentiable gate-passing bonus: if the bonus dominates and ABPT's converged gate-crossing rate or reward falls to BPTT's level, the claimed compensation is bounded. A sharper test is to replace the learned critic with a deliberately poor one at initialization and measure the angle between ABPT's parameter update and the exact analytical gradient of a fully differentiable surrogate reward; if the update direction is no closer than BPTT's, the value-gradient correction is not carrying the argument.","tokens_in":12807,"feed_emoji":"🚁","tokens_out":4892,"duration_ms":44043,"temperature":0.7,"pith_summary":"Backpropagation-through-time (BPTT) trains policies through a differentiable simulator, but quadrotor reward functions usually contain non-differentiable pieces such as success bonuses, and those pieces silently drop out of the gradient, biasing training. This paper proposes Amended Backpropagation-through-Time (ABPT), which optimizes an objective that averages the N-step return with a 0-step return: the learned value function's gradient is computed through the action-value function, so it is differentiable even when the reward is not. That value-gradient term compensates the missing reward gradients and keeps parameter updates heading in the right direction. Across hovering, tracking, landing, and racing, ABPT is reported to converge faster and reach higher final rewards than PPO, plain BPTT, and SHAC, with real-world flight validation. The paper also adds entropy regularization and state replay to improve exploration.","feed_headline":"Mix in a value gradient to fix biased BPTT training","feed_subtitle":"On quadrotor tasks, combining 0-step and N-step returns speeds convergence and lifts final rewards.","key_machinery":"The load-bearing object is the 0-step return $J_0^\\theta = V_\\phi(s_t)$, defined through the action-value function with an entropy bonus: $V_\\phi(s) = \\mathbb{E}_{a\\sim\\pi}[Q_\\phi(s,a)] + \\kappa H(\\pi_\\theta(\\cdot|s))$. Unlike a state-only value $V_\\phi(s)$, this form depends on $\\theta$ through the policy, so its gradient with respect to $\\theta$ is nonzero and can be backpropagated. The N-step return $J_N^\\theta = \\sum_{k=0}^{N-1}\\gamma^k R(s_{t+k}) + (1-d)\\gamma^N V_\\phi(s_{t+N})$ supplies the direct first-order reward path. Averaging the two yields a gradient whose value-gradient half is immune to non-differentiable rewards; a target critic, TD-$\\lambda$ return estimation, entropy regularization, and a state replay buffer stabilize the learned value and exploration.","core_discovery":"The central claim is that the biased first-order gradient caused by partially differentiable rewards can be corrected by averaging in the value gradient. With the objective $J_\\theta = \\frac{1}{2|B|}\\sum_i \\left(J_N^\\theta + J_0^\\theta\\right)$, where $J_0^\\theta = V_\\phi(s_t)$ is the state value computed from the learned action-value function $Q_\\phi(s, \\pi_\\theta(s))$, backpropagation produces the average of the value gradient $\\nabla^{[q]}_\\theta G_{t:t+N}$ and the first-order gradient plus the terminal value term. Because $Q_\\phi$ is a differentiable function of $\\theta$ regardless of reward differentiability, the 0-step term supplies gradient information about the non-differentiable part of the reward (for example, gate-passing bonuses) that the direct reward path drops. The paper states that this combination 'remains effective in guiding the parameter updates toward the correct direction, when the first-order gradient is biased,' and supports it with a parameter-residual experiment in Appendix A.","pith_inferences":["If the bias-correction story holds generally, the same 0-step/value-gradient averaging should transfer to other differentiable simulators with contact-rich or hybrid rewards, such as legged locomotion with foot-contact bonuses, not just quadrotors.","The method depends on the critic's accuracy early in training; a testable extension would adaptively weight the value-gradient term by critic uncertainty instead of a fixed one-half, which the paper explicitly leaves open.","Because ABPT stores only states rather than transitions, its replay buffer could be augmented with goal-conditioned or adversarially sampled states to stress-test generalization to corner cases."],"forward_implications":["Where rewards are partially differentiable, ABPT's update should stay closer to the true policy gradient than plain BPTT because the 0-step value gradient carries information about non-differentiable reward components.","The equal-weight average, rather than a variance-based adaptive mixture, keeps the per-step overhead small, making the method practical on GPU-parallel differentiable simulators.","On fully differentiable tasks, ABPT should retain BPTT-level convergence speed while gaining exploration from entropy and state replay, as demonstrated for hovering and tracking.","When non-differentiable rewards dominate the total reward, performance may still degrade; the paper's own limitation statement says the bias may not be fully eliminated in that regime."],"supporting_citations":[{"why":"Supplies the truncated N-step return with a terminal value and the actor-critic BPTT baseline that ABPT extends and compares against.","marker":"[35]"},{"why":"Introduces the Q-gradient (value gradient through the action-value function) that the 0-step term is built on, and motivates mixing first-order gradients with value predictions.","marker":"[36]"},{"why":"Gives the k-step and lambda-return definitions and the policy-gradient background used in Equations (4), (13), and (14).","marker":"[4]"},{"why":"Supplies the maximum-entropy framework, the adaptive entropy coefficient, and the target-critic stabilization adopted in Equations (12) through (15).","marker":"[15]"},{"why":"Analyzes when differentiable simulators give better policy gradients and motivates the treatment of gradient bias and variance from non-differentiable rewards.","marker":"[11]"},{"why":"Provides the differentiable quadrotor simulator in which all four tasks are trained and evaluated.","marker":"[29]"}],"fun_headline_variants":["ABPT: mix in value gradient to fix BPTT bias","Value-gradient amendment for biased BPTT rewards","Amend BPTT with value gradients for quadrotor tasks","Blend 0-step and N-step returns to fix BPTT bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's correction depends on the learned value function being accurate enough in the states actually visited; if the critic is wrong early in training, the added value-gradient term can push the policy in the wrong direction, and the paper provides no guarantee that the equal-weight average stays unbiased.","fun_headline_variants_meta":{"raw":{"variants":["ABPT: mix in value gradient to fix BPTT bias","Value-gradient amendment for biased BPTT rewards","Amend BPTT with value gradients for quadrotor tasks","Blend 0-step and N-step returns to fix BPTT bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1353,"prompt_tokens":951,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":567,"tokens_out":402,"duration_ms":4109,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:04:57.742867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare ABPT with plain BPTT on the racing task while scaling up the non-differentiable gate-passing bonus: if the bonus dominates and ABPT's converged gate-crossing rate or reward falls to BPTT's level, the claimed compensation is bounded. A sharper test is to replace the learned critic with a deliberately poor one at initialization and measure the angle between ABPT's parameter update and the exact analytical gradient of a fully differentiable surrogate reward; if the update direction is no closer than BPTT's, the value-gradient correction is not carrying the argument.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Q-gradient (value gradient through the action-value function) that the 0-step term is built on, and motivates mixing first-order gradients with value predictions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the k-step and lambda-return definitions and the policy-gradient background used in Equations (4), (13), and (14)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Analyzes when differentiable simulators give better policy gradients and motivates the treatment of gradient bias and variance from non-differentiable rewards."}],"review_version":1}