REVIEW 3 major objections 8 minor 1 cited by
Reactive Aerobatic Flight via Reinforcement Learning
T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A reinforcement learning policy, trained in simulation with an automated curriculum and domain randomization, enables a quadrotor to perform continuous inverted flight while reactively navigating a moving gate—reported as a first.
desk verdict The reset-state curriculum is a real contribution, but the 'first demonstration' claim needs quantitative real-world evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the combination of a biased-reset curriculum and domain randomization wrapped around a PPO-trained multilayer perceptron. The curriculum uses differential flatness to integrate a goal state backward in time, generating physically feasible reset states of increasing difficulty; a critic then selects seed states of moderate difficulty to expand for the next iteration, and a replay buffer preserves old states to prevent catastrophic forgetting. This makes the sparse aerobatic reward reachable early in training. Domain randomization—perturbing the drag matrix, control inputs, and latency—ensures the policy sees a distribution of dynamics, which is what allows the zero-shot transfer of the learned controller to the real quadrotor. The action space of thrust plus body rates is itself a mechanism: it is high-bandwidth enough for agility yet low-level enough to avoid the worst sim-to-real discrepancies of direct motor commands.
What would settle it
If the same trained policy were deployed on a quadrotor whose control latency exceeds 36 ms, whose drag coefficient differs from the nominal by more than 50%, or whose rotor time constant is outside the simulated range, and the drone then fails to complete a continuous inverted pass through a translating gate at roughly 1.0 m/s, the zero-shot transfer claim would be falsified.
Extended reading notes
Core claim
The central discovery is that a model-free policy, trained with Proximal Policy Optimization in a quadrotor simulator and regularized by a biased-reset curriculum plus domain randomization, can learn continuous inverted maneuvers and execute them reactively on a real quadrotor without any fine-tuning. The policy's observation is a composite of the next two aerobatic waypoints expressed in the drone's body frame, the drone's proprioceptive state, and the previous action; its action is mass-normalized collective thrust and body rates, a choice that mimics human piloting and keeps the sim-to-real gap manageable. The curriculum performs backward integration of a differentially flat model to spawn reset states near the goal, implicitly progressing from easy to hard, while randomization varies drag coefficients by ±50%, control inputs by ±20%, and latency from 4 to 36 ms. The paper's headline result—continuous inverted flight through a translating gate—is presented as a first for a reactive, autonomous system, and benchmarks against a recent trajectory-optimization method show lower position and attitude errors at waypoints and a response time of 0.3 ms versus roughly 1.5 seconds.
Load-bearing premise
The load-bearing premise is that the domain randomization ranges used in training—drag perturbed by ±50%, control inputs by ±20%, and latency from 4 to 36 ms—span the real quadrotor's actual dynamics, so that a policy trained in simulation transfers to the real drone without further tuning.
Editorial extensions
If this is right
- Aerobatic flight control becomes a reactive skill: the policy responds to moving waypoints in 0.3 ms, so dynamic gates and other moving targets can be handled online.
- Manual reward shaping is unnecessary: the automated curriculum alone yields efficient and stable convergence from a sparse aerobatic reward.
- The learned policy can repeat continuous inverted traversals multiple times, suggesting aerobatic maneuvers can be composed as reusable motor skills rather than replanned trajectories.
- The approach extrapolates beyond its training distribution, with a 78 percent success rate at a gate speed of 3.4 m/s, exceeding the 2 m/s maximum seen in training.
Reading between the lines
- If the recipe generalizes, the same curriculum-plus-randomization setup could be applied to other sparse-reward aggressive skills, such as perching, obstacle-dodging flips, or high-speed gap squeezes, without re-engineering rewards.
- Because state estimation and gate tracking currently rely on an external motion-capture system, the claimed reactivity is not yet about onboard perception; replacing motion capture with a vision pipeline is the natural next test, and the paper's latency numbers would grow with perception delay.
- The domain-randomization ranges were not validated against measured hardware discrepancies, so an immediate extension would be to identify the real drone's drag, delay, and actuation distributions and check whether the policy remains robust when the randomization envelope is shrunk to match reality.
- The benchmark comparison is against a static-waypoint planner; a direct comparison with a method that replans online on moving gates would isolate whether the 0.3 ms latency or the absence of tracking error is the main source of the performance gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a reinforcement learning (RL) framework for quadrotor aerobatic flight that maps drone state and aerobatic waypoint observations directly to thrust and body-rate commands. Training uses a biased reset-state curriculum that back-integrates from goal waypoints, plus domain randomization for zero-shot sim-to-real transfer. The authors validate the approach in the Flightmare simulator with ablations, a benchmark against a prior planning-based method, and real-world flights, including a claim of the first demonstration of continuous inverted flight while reactively navigating a moving gate.
Significance. If the central claim holds, the paper demonstrates a notable advance in agile quadrotor control: end-to-end RL can produce reactive aerobatic behavior that combines continuous inverted flight with moving-gate avoidance, going beyond prior planning-based and imitation-learning systems. The simulation results support the value of the curriculum in improving convergence over sparse-reward and reward-shaping baselines, and the zero-shot real-world transfer is an ambitious goal. However, the real-world evidence for the headline claim is currently qualitative, and the domain-randomization ranges are not empirically justified, so the significance is conditional on additional evidence.
major comments (3)
- [Section IV-B (Real-World Experiments)] The real-world dynamic-gate demonstration, which underpins the claim of "first demonstration of continuous inverted flight while reactively navigating a moving gate," is reported without any quantitative success metrics. The text provides only a verbal description and snapshots, with no number of attempts, successes, failures, per-trial gate-crossing errors, or a defined success criterion. Because the policy is stochastic and the MCS-plus-EKF pipeline introduces latency and noise, a single successful run could be an unrepresentative sample. Please report trial counts, a precise success criterion (e.g., gate-plane crossing within specified position/attitude tolerances), and failure analysis. This is load-bearing because the abstract's central claim is a real-world capability, and the current evidence does not establish that the capability is reliable.
- [Section III-D (Sim-to-Real Transfer)] The zero-shot sim-to-real transfer claim relies on the domain randomization ranges: drag coefficients perturbed by ±50%, control inputs by ±20%, and latency sampled from 4–36 ms. The paper provides no empirical justification that these ranges cover the actual discrepancies between the Flightmare model and the real quadrotor (including rotor delays, battery sag, or aerodynamic coupling during inverted flight), nor does it report a sensitivity analysis. If the real-world gap lies outside these perturbations, the transfer claim could fail. Please justify the chosen ranges with measured model discrepancies or add an ablation showing the policy's sensitivity to these ranges.
- [Abstract and Section IV (Benchmarks)] The claim of being the "first demonstration" of continuous inverted flight while reactively navigating a moving gate is not supported by a systematic comparison with prior aerobatic RL and planning literature. The paper cites Kaufmann et al. [3] and Song et al. [11] but does not analyze whether their methods or other existing systems could achieve the demonstrated capability, nor does it explain why the cited works are insufficient. The benchmark in Table I compares only against the authors' own prior method [5] in a static-waypoint setting. Please add a focused literature comparison or soften the novelty claim to match the actual evidence.
minor comments (8)
- [Equation (4)] Equation (4) is typeset confusingly: the equation number appears inside the expression, and the indicator function C is placed after the number, making the formula look like a multiplication by "(4)". Please reformat to read r_aer = (F(p_error) + F(theta_error) + c) * C(...).
- [Table II] The table header says "Aer.A (%)" but the values are reported in degrees (e.g., 11.5, 12.4). This appears to be a typo; the unit should be degrees.
- [Section IV-A (Ablation Studies)] The ablation results in Fig. 3 are presented only as reward curves without statistical details. Please report the number of training seeds, final average rewards, and standard deviations to substantiate the claim of "fastest training speed and best convergence."
- [Section III (Methodology)] Several hyperparameters are not reported: the reward weights w0–w3, the completion threshold L, the activation constants a and b in Eq. (5), the biased sampling probabilities ρ1 and ρ2 in Algorithm 1, and the PPO training hyperparameters (learning rate, batch size, etc.). Providing these values is necessary for reproducibility.
- [Section III-B1 (Observation Space)] The notation O_env = [δep1, eq1, δep2, eq2] is not fully defined. Please clarify what δep denotes (presumably a position difference in the body frame) and define the subscripts consistently.
- [References] References [3] and [6] appear to be the same paper (Kaufmann et al., "Deep drone acrobatics") listed with different venue formats. Please consolidate into one citation.
- [Section IV-B (Real-World Experiments)] The real-world experimental setup would benefit from more detail: the gate dimensions, the distance between consecutive gates, the MCS update rate, and the total flight distance. Currently only the gate velocity (~1.0 m/s) is given, which is well within the training distribution of 0–2 m/s; a test at or above the training maximum would better support the "reactive" claim.
- [Throughout] There are several typos and grammatical errors, including "achieveing" (Contribution 1), "expandsion" (text after Eq. (8)), "reseting" (Section IV-A), and "suboptimal" (Section IV-A). A careful proofread is recommended.
Circularity Check
No significant circularity: the RL pipeline is trained and evaluated on its own stated objective, and the central real-world claim is an empirical demonstration, not a prediction derived from fitted inputs.
full rationale
The paper's derivation chain is self-contained. The policy is trained with PPO on the reward defined in Eq. (3)-(7), whose completion criterion and error terms are the same geometric waypoint objective used for evaluation in Tables I and II; this is standard task evaluation, not a fitted parameter renamed as a prediction. The curriculum in Algorithm 1 selects reset states via backward flat-state integration and critic evaluation, and it does not presuppose the final policy's success. Domain randomization in Section III-D intentionally broadens the training distribution with fixed perturbation ranges rather than fitting those ranges to the real-world result, and the real-world flights are presented as demonstrations rather than as quantitative predictions from fitted constants. The only self-referential element is the benchmark against reference [5], which is the authors' own prior Science Robotics paper, and the observation-space formulation is described as 'similar to our previous work [5]'. However, [5] is implemented and run as a baseline in Table I, so the comparison is empirical and not a citation substituted for evidence. The 'first demonstration' claim is supported by qualitative real-world snapshots and video rather than trial statistics, but that is an evidence-strength and overclaim concern, not circularity. No equation reduces to its own input, and no load-bearing argument depends on an unverified self-citation. Score 1 reflects only the minor presence of self-citation in the baseline choice and problem formulation.
Assumptions & free parameters
free parameters (6)
- reward weights w0..w3 =
not specified
- completion threshold L =
not specified
- reward activation constants a and b =
not specified
- biased sampling probabilities rho1 and rho2 =
not specified
- domain randomization ranges =
drag +/-50%, control +/-20%, latency 4-36 ms
- real-world action bounds =
thrust 0-20 m/s2, roll/pitch +/-5 rad/s, yaw +/-3.14 rad/s
assumptions (3)
- domain assumption Rigid-body quadrotor dynamics with mass-normalized thrust, neglecting rotor dynamics and most aerodynamic effects (Eq. 1).
- domain assumption Differential flatness with heading aligned to the horizontal velocity projection (Eq. 9).
- ad hoc to paper Domain randomization distributions cover the real-world dynamics gap (Section III-D).
Cite this review
Pith. "Pith review of Reactive Aerobatic Flight via Reinforcement Learning." pith.science (2026). https://pith.science/paper/NSTF5CGC
@misc{pith2026250524396,
author = {Pith},
title = {Pith review of: Reactive Aerobatic Flight via Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/NSTF5CGC}},
note = {Machine review of arXiv:2505.24396}
}
read the original abstract
Quadrotors have demonstrated remarkable versatility, yet their full aerobatic potential remains largely untapped due to inherent underactuation and the complexity of aggressive maneuvers. Traditional approaches, separating trajectory optimization and tracking control, suffer from tracking inaccuracies, computational latency, and sensitivity to initial conditions, limiting their effectiveness in dynamic, high-agility scenarios. Inspired by recent breakthroughs in data-driven methods, we propose a reinforcement learning-based framework that directly maps drone states and aerobatic intentions to control commands, eliminating modular separation to enable quadrotors to perform end-to-end policy optimization for extreme aerobatic maneuvers. To ensure efficient and stable training, we introduce an automated curriculum learning strategy that dynamically adjusts aerobatic task difficulty. Enabled by domain randomization for robust zero-shot sim-to-real transfer, our approach is validated in demanding real-world experiments, including the first demonstration of a drone autonomously performing continuous inverted flight while reactively navigating a moving gate, showcasing unprecedented agility.
Figures
Forward citations
Cited by 1 Pith paper
-
High-Speed Vision-Based Flight in Clutter with Safety-Shielded Reinforcement Learning
A reinforcement-learning quadrotor policy trained with Dijkstra and control-barrier rewards, plus a high-order CBF safety filter, is claimed to navigate cluttered indoor and forest environments at up to 7.5 m/s.
Reference graph
Works this paper leans on
-
[5]
Unlocking aerobatic potential of quadcopters: Autonomous freestyle flight generation and execution,
M. Wang, Q. Wang, Z. Wang, Y . Gao, J. Wang, C. Cui, Y . Li, Z. Ding, K. Wang, C. Xu, and F. Gao, “Unlocking aerobatic potential of quadcopters: Autonomous freestyle flight generation and execution,”Science Robotics, vol. 10, no. 101, p. eadp9905,
-
[3]
E. Kaufmann, A. Loquercio, R. Ranftl, M. Mueller, V . Koltun, and D. Scaramuzza, “Deep drone acrobatics,” 07 2020
work page 2020
-
[11]
Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,
Y . Song, A. Romero, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,”Science Robotics, vol. 8, no. 82, p. eadg1462, 2023
2023
-
[1]
Whole-Body Control Through Narrow Gaps From Pixels To Action
T. Wu, Y . Chen, T. Chen, G. Zhao, and F. Gao, “Whole-body control through narrow gaps from pixels to action,”arXiv preprint arXiv:2409.00895, 2024
work page Pith review arXiv 2024
-
[2]
Controller synthesis and perfor- mance optimization for aerobatic quadrotor flight,
Y . Chen and N. O. P ´erez-Arancibia, “Controller synthesis and perfor- mance optimization for aerobatic quadrotor flight,”IEEE Transactions on Control Systems Technology, vol. 28, no. 6, pp. 2204–2219, 2020
work page 2020
-
[4]
Minimum snap trajectory generation and control for quadrotors,
D. Mellinger and V . Kumar, “Minimum snap trajectory generation and control for quadrotors,” in2011 IEEE international conference on robotics and automation. IEEE, 2011, pp. 2520–2525
work page 2011
-
[6]
E. Kaufmann, A. Loquercio, R. Ranftl, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Deep drone acrobatics,”arXiv preprint arXiv:2006.05768, 2020
arXiv 2006
-
[7]
M. Kamel, T. Stastny, K. Alexis, and R. Siegwart, “Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system,”Robot Operating System (ROS) The Complete Reference (V olume 2), pp. 3–39, 2017
work page 2017
Show all 23 references
-
[8]
Fast-racing: An open-source strong baseline forSE(3)planning in autonomous drone racing,
Z. Han, Z. Wang, N. Pan, Y . Lin, C. Xu, and F. Gao, “Fast-racing: An open-source strong baseline forSE(3)planning in autonomous drone racing,”IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 8631–8638, 2021
2021
-
[9]
Improving local trajectory optimisation using probabilistic movement primitives,
R. A. Shyam, P. Lightbody, G. Das, P. Liu, S. Gomez-Gonzalez, and G. Neumann, “Improving local trajectory optimisation using probabilistic movement primitives,” in2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 2666– 2671
2019
-
[10]
Learning high-speed flight in the wild,
A. Loquercio, E. Kaufmann, R. Ranftl, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Learning high-speed flight in the wild,”Science Robotics, vol. 6, no. 59, p. eabg5810, 2021
2021
-
[12]
Champion-level drone racing using deep reinforce- ment learning,
E. Kaufmann, L. Bauersfeld, A. Loquercio, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforce- ment learning,”Nature, vol. 620, no. 7976, pp. 982–987, 2023
2023
-
[13]
Automated curriculum learning for neural net- works,
A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu, “Automated curriculum learning for neural net- works,” ininternational conference on machine learning. Pmlr, 2017, pp. 1311–1320
2017
-
[14]
Re- verse curriculum generation for reinforcement learning,
C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel, “Re- verse curriculum generation for reinforcement learning,” inConference on robot learning. PMLR, 2017, pp. 482–495
2017
-
[15]
Automatic curriculum learning for deep rl: A short survey,
R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y . Oudeyer, “Automatic curriculum learning for deep rl: A short survey,”arXiv preprint arXiv:2003.04664, 2020
2003 arXiv
-
[16]
Model predictive contouring control for time-optimal quadrotor flight,
A. Romero, S. Sun, P. Foehn, and D. Scaramuzza, “Model predictive contouring control for time-optimal quadrotor flight,”IEEE Transac- tions on Robotics, vol. 38, no. 6, pp. 3340–3356, 2022
2022
-
[17]
Trajectory generation and tracking control for aggressive tail-sitter flights,
G. Lu, Y . Cai, N. Chen, F. Kong, Y . Ren, and F. Zhang, “Trajectory generation and tracking control for aggressive tail-sitter flights,”The International Journal of Robotics Research, vol. 43, no. 3, pp. 241– 280, 2024
2024
-
[18]
Differential flatness transformations for aggressive quadrotor flight,
B. Morrell, M. Rigter, G. Merewether, R. Reid, R. Thakker, T. Tzane- tos, V . Rajur, and G. Chamitoff, “Differential flatness transformations for aggressive quadrotor flight,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 5204–5210
2018
-
[19]
Back to newton’s laws: Learning vision-based agile flight via differentiable physics,
Y . Zhang, Y . Hu, Y . Song, D. Zou, and W. Lin, “Back to newton’s laws: Learning vision-based agile flight via differentiable physics,” arXiv preprint arXiv:2407.10648, 2024
2024 arXiv
-
[20]
Proximal policy optimization algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”CoRR, vol. abs/1707.06347, 2017. [Online]. Available: http://arxiv.org/abs/ 1707.06347
2017 arXiv
-
[21]
Nonlinear dynamic modeling for high performance control of a quadrotor,
M. Bangura and R. Mahony, “Nonlinear dynamic modeling for high performance control of a quadrotor,” inProceedings of the 2012 Australasian Conference on Robotics and Automation, ACRA 2012, ser. Australasian Conference on Robotics and Automation, ACRA, 2012, 2012 Australasian C...
2012
-
[22]
Flightmare: A flexible quadrotor simulator,
Y . Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” inConference on Robot Learning, 2020
2020
-
[2025]
Available: https://www.science.org/doi/abs/10.1126/ scirobotics.adp9905
[Online]. Available: https://www.science.org/doi/abs/10.1126/ scirobotics.adp9905
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.