Pith. sign in

REVIEW 3 major objections 8 minor 1 cited by

Reactive Aerobatic Flight via Reinforcement Learning

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A reinforcement learning policy, trained in simulation with an automated curriculum and domain randomization, enables a quadrotor to perform continuous inverted flight while reactively navigating a moving gate—reported as a first.

desk verdict The reset-state curriculum is a real contribution, but the 'first demonstration' claim needs quantitative real-world evidence. read the letter →

arxiv 2505.24396 v1 pith:NSTF5CGC submitted 2025-05-30 cs.RO

classification cs.RO
keywords quadrotoraerobaticsreinforcementlearningcurriculumdomainrandomizationsim-to-realtransferinvertedflightaggressivecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single end-to-end reinforcement learning policy can replace the traditional trajectory-optimization-plus-tracking pipeline for extreme aerobatic flight. It reports the first demonstration of a quadrotor performing continuous inverted flight while reactively passing through a moving gate, trained entirely in simulation and transferred zero-shot to real hardware. The authors argue that this works because the policy maps drone state and aerobatic waypoints directly to thrust and body-rate commands at 100 Hz, reacting in 0.3 ms, while an automated curriculum and domain randomization make the sparse-reward task learnable and robust to sim-to-real gaps. If correct, this establishes model-free reinforcement learning as a viable route to reactive agility that modular optimization-based methods cannot match.

What carries the argument

The load-bearing mechanism is the combination of a biased-reset curriculum and domain randomization wrapped around a PPO-trained multilayer perceptron. The curriculum uses differential flatness to integrate a goal state backward in time, generating physically feasible reset states of increasing difficulty; a critic then selects seed states of moderate difficulty to expand for the next iteration, and a replay buffer preserves old states to prevent catastrophic forgetting. This makes the sparse aerobatic reward reachable early in training. Domain randomization—perturbing the drag matrix, control inputs, and latency—ensures the policy sees a distribution of dynamics, which is what allows the zero-shot transfer of the learned controller to the real quadrotor. The action space of thrust plus body rates is itself a mechanism: it is high-bandwidth enough for agility yet low-level enough to avoid the worst sim-to-real discrepancies of direct motor commands.

What would settle it

If the same trained policy were deployed on a quadrotor whose control latency exceeds 36 ms, whose drag coefficient differs from the nominal by more than 50%, or whose rotor time constant is outside the simulated range, and the drone then fails to complete a continuous inverted pass through a translating gate at roughly 1.0 m/s, the zero-shot transfer claim would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a model-free policy, trained with Proximal Policy Optimization in a quadrotor simulator and regularized by a biased-reset curriculum plus domain randomization, can learn continuous inverted maneuvers and execute them reactively on a real quadrotor without any fine-tuning. The policy's observation is a composite of the next two aerobatic waypoints expressed in the drone's body frame, the drone's proprioceptive state, and the previous action; its action is mass-normalized collective thrust and body rates, a choice that mimics human piloting and keeps the sim-to-real gap manageable. The curriculum performs backward integration of a differentially flat model to spawn reset states near the goal, implicitly progressing from easy to hard, while randomization varies drag coefficients by ±50%, control inputs by ±20%, and latency from 4 to 36 ms. The paper's headline result—continuous inverted flight through a translating gate—is presented as a first for a reactive, autonomous system, and benchmarks against a recent trajectory-optimization method show lower position and attitude errors at waypoints and a response time of 0.3 ms versus roughly 1.5 seconds.

Load-bearing premise

The load-bearing premise is that the domain randomization ranges used in training—drag perturbed by ±50%, control inputs by ±20%, and latency from 4 to 36 ms—span the real quadrotor's actual dynamics, so that a policy trained in simulation transfers to the real drone without further tuning.

Editorial extensions

If this is right

  • Aerobatic flight control becomes a reactive skill: the policy responds to moving waypoints in 0.3 ms, so dynamic gates and other moving targets can be handled online.
  • Manual reward shaping is unnecessary: the automated curriculum alone yields efficient and stable convergence from a sparse aerobatic reward.
  • The learned policy can repeat continuous inverted traversals multiple times, suggesting aerobatic maneuvers can be composed as reusable motor skills rather than replanned trajectories.
  • The approach extrapolates beyond its training distribution, with a 78 percent success rate at a gate speed of 3.4 m/s, exceeding the 2 m/s maximum seen in training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the recipe generalizes, the same curriculum-plus-randomization setup could be applied to other sparse-reward aggressive skills, such as perching, obstacle-dodging flips, or high-speed gap squeezes, without re-engineering rewards.
  • Because state estimation and gate tracking currently rely on an external motion-capture system, the claimed reactivity is not yet about onboard perception; replacing motion capture with a vision pipeline is the natural next test, and the paper's latency numbers would grow with perception delay.
  • The domain-randomization ranges were not validated against measured hardware discrepancies, so an immediate extension would be to identify the real drone's drag, delay, and actuation distributions and check whether the policy remains robust when the randomization envelope is shrunk to match reality.
  • The benchmark comparison is against a static-waypoint planner; a direct comparison with a method that replans online on moving gates would isolate whether the 0.3 ms latency or the absence of tracking error is the main source of the performance gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper presents a reinforcement learning (RL) framework for quadrotor aerobatic flight that maps drone state and aerobatic waypoint observations directly to thrust and body-rate commands. Training uses a biased reset-state curriculum that back-integrates from goal waypoints, plus domain randomization for zero-shot sim-to-real transfer. The authors validate the approach in the Flightmare simulator with ablations, a benchmark against a prior planning-based method, and real-world flights, including a claim of the first demonstration of continuous inverted flight while reactively navigating a moving gate.

Significance. If the central claim holds, the paper demonstrates a notable advance in agile quadrotor control: end-to-end RL can produce reactive aerobatic behavior that combines continuous inverted flight with moving-gate avoidance, going beyond prior planning-based and imitation-learning systems. The simulation results support the value of the curriculum in improving convergence over sparse-reward and reward-shaping baselines, and the zero-shot real-world transfer is an ambitious goal. However, the real-world evidence for the headline claim is currently qualitative, and the domain-randomization ranges are not empirically justified, so the significance is conditional on additional evidence.

major comments (3)
  1. [Section IV-B (Real-World Experiments)] The real-world dynamic-gate demonstration, which underpins the claim of "first demonstration of continuous inverted flight while reactively navigating a moving gate," is reported without any quantitative success metrics. The text provides only a verbal description and snapshots, with no number of attempts, successes, failures, per-trial gate-crossing errors, or a defined success criterion. Because the policy is stochastic and the MCS-plus-EKF pipeline introduces latency and noise, a single successful run could be an unrepresentative sample. Please report trial counts, a precise success criterion (e.g., gate-plane crossing within specified position/attitude tolerances), and failure analysis. This is load-bearing because the abstract's central claim is a real-world capability, and the current evidence does not establish that the capability is reliable.
  2. [Section III-D (Sim-to-Real Transfer)] The zero-shot sim-to-real transfer claim relies on the domain randomization ranges: drag coefficients perturbed by ±50%, control inputs by ±20%, and latency sampled from 4–36 ms. The paper provides no empirical justification that these ranges cover the actual discrepancies between the Flightmare model and the real quadrotor (including rotor delays, battery sag, or aerodynamic coupling during inverted flight), nor does it report a sensitivity analysis. If the real-world gap lies outside these perturbations, the transfer claim could fail. Please justify the chosen ranges with measured model discrepancies or add an ablation showing the policy's sensitivity to these ranges.
  3. [Abstract and Section IV (Benchmarks)] The claim of being the "first demonstration" of continuous inverted flight while reactively navigating a moving gate is not supported by a systematic comparison with prior aerobatic RL and planning literature. The paper cites Kaufmann et al. [3] and Song et al. [11] but does not analyze whether their methods or other existing systems could achieve the demonstrated capability, nor does it explain why the cited works are insufficient. The benchmark in Table I compares only against the authors' own prior method [5] in a static-waypoint setting. Please add a focused literature comparison or soften the novelty claim to match the actual evidence.
minor comments (8)
  1. [Equation (4)] Equation (4) is typeset confusingly: the equation number appears inside the expression, and the indicator function C is placed after the number, making the formula look like a multiplication by "(4)". Please reformat to read r_aer = (F(p_error) + F(theta_error) + c) * C(...).
  2. [Table II] The table header says "Aer.A (%)" but the values are reported in degrees (e.g., 11.5, 12.4). This appears to be a typo; the unit should be degrees.
  3. [Section IV-A (Ablation Studies)] The ablation results in Fig. 3 are presented only as reward curves without statistical details. Please report the number of training seeds, final average rewards, and standard deviations to substantiate the claim of "fastest training speed and best convergence."
  4. [Section III (Methodology)] Several hyperparameters are not reported: the reward weights w0–w3, the completion threshold L, the activation constants a and b in Eq. (5), the biased sampling probabilities ρ1 and ρ2 in Algorithm 1, and the PPO training hyperparameters (learning rate, batch size, etc.). Providing these values is necessary for reproducibility.
  5. [Section III-B1 (Observation Space)] The notation O_env = [δep1, eq1, δep2, eq2] is not fully defined. Please clarify what δep denotes (presumably a position difference in the body frame) and define the subscripts consistently.
  6. [References] References [3] and [6] appear to be the same paper (Kaufmann et al., "Deep drone acrobatics") listed with different venue formats. Please consolidate into one citation.
  7. [Section IV-B (Real-World Experiments)] The real-world experimental setup would benefit from more detail: the gate dimensions, the distance between consecutive gates, the MCS update rate, and the total flight distance. Currently only the gate velocity (~1.0 m/s) is given, which is well within the training distribution of 0–2 m/s; a test at or above the training maximum would better support the "reactive" claim.
  8. [Throughout] There are several typos and grammatical errors, including "achieveing" (Contribution 1), "expandsion" (text after Eq. (8)), "reseting" (Section IV-A), and "suboptimal" (Section IV-A). A careful proofread is recommended.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the RL pipeline is trained and evaluated on its own stated objective, and the central real-world claim is an empirical demonstration, not a prediction derived from fitted inputs.

full rationale

The paper's derivation chain is self-contained. The policy is trained with PPO on the reward defined in Eq. (3)-(7), whose completion criterion and error terms are the same geometric waypoint objective used for evaluation in Tables I and II; this is standard task evaluation, not a fitted parameter renamed as a prediction. The curriculum in Algorithm 1 selects reset states via backward flat-state integration and critic evaluation, and it does not presuppose the final policy's success. Domain randomization in Section III-D intentionally broadens the training distribution with fixed perturbation ranges rather than fitting those ranges to the real-world result, and the real-world flights are presented as demonstrations rather than as quantitative predictions from fitted constants. The only self-referential element is the benchmark against reference [5], which is the authors' own prior Science Robotics paper, and the observation-space formulation is described as 'similar to our previous work [5]'. However, [5] is implemented and run as a baseline in Table I, so the comparison is empirical and not a citation substituted for evidence. The 'first demonstration' claim is supported by qualitative real-world snapshots and video rather than trial statistics, but that is an evidence-strength and overclaim concern, not circularity. No equation reduces to its own input, and no load-bearing argument depends on an unverified self-citation. Score 1 reflects only the minor presence of self-citation in the baseline choice and problem formulation.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

All numerical knobs in this paper are manually selected or randomized rather than derived. The free parameters are reward weights, thresholds, and randomization ranges. No new physical entities are introduced.

free parameters (6)
  • reward weights w0..w3 = not specified
    Hand-tuned weights in Eq. (3) balancing aerobatic completion, action magnitude, action change, and yaw alignment.
  • completion threshold L = not specified
    Used in the completion criterion in Eq. (4) to determine how far the drone can deviate laterally from the waypoint.
  • reward activation constants a and b = not specified
    Shape the sparse reward gradient in Eq. (5).
  • biased sampling probabilities rho1 and rho2 = not specified
    Control the mix of reset states in Algorithm 1 among current expanded states, replay buffer, and start states.
  • domain randomization ranges = drag +/-50%, control +/-20%, latency 4-36 ms
    Hyperparameters chosen to bridge the sim-to-real gap, but not justified from measured discrepancies.
  • real-world action bounds = thrust 0-20 m/s2, roll/pitch +/-5 rad/s, yaw +/-3.14 rad/s
    Constraints imposed on the policy output during real-world deployment.
assumptions (3)
  • domain assumption Rigid-body quadrotor dynamics with mass-normalized thrust, neglecting rotor dynamics and most aerodynamic effects (Eq. 1).
    The dynamics used in simulation and curriculum expansion assume this model; the real quadrotor may differ.
  • domain assumption Differential flatness with heading aligned to the horizontal velocity projection (Eq. 9).
    Used to convert flat states back to full states during curriculum reset generation; not guaranteed for aggressive maneuvers.
  • ad hoc to paper Domain randomization distributions cover the real-world dynamics gap (Section III-D).
    The chosen randomization ranges are not derived from measured sim-to-real discrepancies; the claim of zero-shot transfer hinges on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reactive Aerobatic Flight via Reinforcement Learning." pith.science (2026). https://pith.science/paper/NSTF5CGC

@misc{pith2026250524396,
  author       = {Pith},
  title        = {Pith review of: Reactive Aerobatic Flight via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSTF5CGC}},
  note         = {Machine review of arXiv:2505.24396}
}
read the original abstract

Quadrotors have demonstrated remarkable versatility, yet their full aerobatic potential remains largely untapped due to inherent underactuation and the complexity of aggressive maneuvers. Traditional approaches, separating trajectory optimization and tracking control, suffer from tracking inaccuracies, computational latency, and sensitivity to initial conditions, limiting their effectiveness in dynamic, high-agility scenarios. Inspired by recent breakthroughs in data-driven methods, we propose a reinforcement learning-based framework that directly maps drone states and aerobatic intentions to control commands, eliminating modular separation to enable quadrotors to perform end-to-end policy optimization for extreme aerobatic maneuvers. To ensure efficient and stable training, we introduce an automated curriculum learning strategy that dynamically adjusts aerobatic task difficulty. Enabled by domain randomization for robust zero-shot sim-to-real transfer, our approach is validated in demanding real-world experiments, including the first demonstration of a drone autonomously performing continuous inverted flight while reactively navigating a moving gate, showcasing unprecedented agility.

Figures

Figures reproduced from arXiv: 2505.24396 by the authors.

Figure 1
Figure 1. Visualization of the large-scale aerobatic flight trajectory, rendered using [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The training framework. The term ryaw denotes the yaw alignment reward, which encourages the UAV’s heading direction to align closely with its velocity vector. This alignment results in more natural flight trajectories that better match human pilots’ intuitive control strategies and facilitates first-person-view recording: ryaw = − arccos( vB − (vB · e3)e3 ||vB − (vB · e3)e3|| · e1), (7) where e3 = [0, 0, 1] and e1 … view at source ↗
Figure 3
Figure 3. Trajectory visualizations from ablation studies and reward [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of flight trajectories. The center of each black [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Real-world experiments. The top row shows the static [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. High-Speed Vision-Based Flight in Clutter with Safety-Shielded Reinforcement Learning

    cs.RO 2026-02 reject novelty 5.0 of 10

    A reinforcement-learning quadrotor policy trained with Dijkstra and control-barrier rewards, plus a high-order CBF safety filter, is claimed to navigate cluttered indoor and forest environments at up to 7.5 m/s.

Reference graph

Works this paper leans on

23 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [5]

    Unlocking aerobatic potential of quadcopters: Autonomous freestyle flight generation and execution,

    M. Wang, Q. Wang, Z. Wang, Y . Gao, J. Wang, C. Cui, Y . Li, Z. Ding, K. Wang, C. Xu, and F. Gao, “Unlocking aerobatic potential of quadcopters: Autonomous freestyle flight generation and execution,”Science Robotics, vol. 10, no. 101, p. eadp9905,

  2. [3]

    Deep drone acrobatics,

    E. Kaufmann, A. Loquercio, R. Ranftl, M. Mueller, V . Koltun, and D. Scaramuzza, “Deep drone acrobatics,” 07 2020

  3. [11]

    Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,

    Y . Song, A. Romero, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,”Science Robotics, vol. 8, no. 82, p. eadg1462, 2023

  4. [1]

    Whole-Body Control Through Narrow Gaps From Pixels To Action

    T. Wu, Y . Chen, T. Chen, G. Zhao, and F. Gao, “Whole-body control through narrow gaps from pixels to action,”arXiv preprint arXiv:2409.00895, 2024

  5. [2]

    Controller synthesis and perfor- mance optimization for aerobatic quadrotor flight,

    Y . Chen and N. O. P ´erez-Arancibia, “Controller synthesis and perfor- mance optimization for aerobatic quadrotor flight,”IEEE Transactions on Control Systems Technology, vol. 28, no. 6, pp. 2204–2219, 2020

  6. [4]

    Minimum snap trajectory generation and control for quadrotors,

    D. Mellinger and V . Kumar, “Minimum snap trajectory generation and control for quadrotors,” in2011 IEEE international conference on robotics and automation. IEEE, 2011, pp. 2520–2525

  7. [6]

    Deep drone acrobatics,

    E. Kaufmann, A. Loquercio, R. Ranftl, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Deep drone acrobatics,”arXiv preprint arXiv:2006.05768, 2020

  8. [7]

    Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system,

    M. Kamel, T. Stastny, K. Alexis, and R. Siegwart, “Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system,”Robot Operating System (ROS) The Complete Reference (V olume 2), pp. 3–39, 2017

Show all 23 references
  1. [8]

    Fast-racing: An open-source strong baseline forSE(3)planning in autonomous drone racing,

    Z. Han, Z. Wang, N. Pan, Y . Lin, C. Xu, and F. Gao, “Fast-racing: An open-source strong baseline forSE(3)planning in autonomous drone racing,”IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 8631–8638, 2021

  2. [9]

    Improving local trajectory optimisation using probabilistic movement primitives,

    R. A. Shyam, P. Lightbody, G. Das, P. Liu, S. Gomez-Gonzalez, and G. Neumann, “Improving local trajectory optimisation using probabilistic movement primitives,” in2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 2666– 2671

  3. [10]

    Learning high-speed flight in the wild,

    A. Loquercio, E. Kaufmann, R. Ranftl, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Learning high-speed flight in the wild,”Science Robotics, vol. 6, no. 59, p. eabg5810, 2021

  4. [12]

    Champion-level drone racing using deep reinforce- ment learning,

    E. Kaufmann, L. Bauersfeld, A. Loquercio, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforce- ment learning,”Nature, vol. 620, no. 7976, pp. 982–987, 2023

  5. [13]

    Automated curriculum learning for neural net- works,

    A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu, “Automated curriculum learning for neural net- works,” ininternational conference on machine learning. Pmlr, 2017, pp. 1311–1320

  6. [14]

    Re- verse curriculum generation for reinforcement learning,

    C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel, “Re- verse curriculum generation for reinforcement learning,” inConference on robot learning. PMLR, 2017, pp. 482–495

  7. [15]

    Automatic curriculum learning for deep rl: A short survey,

    R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y . Oudeyer, “Automatic curriculum learning for deep rl: A short survey,”arXiv preprint arXiv:2003.04664, 2020

  8. [16]

    Model predictive contouring control for time-optimal quadrotor flight,

    A. Romero, S. Sun, P. Foehn, and D. Scaramuzza, “Model predictive contouring control for time-optimal quadrotor flight,”IEEE Transac- tions on Robotics, vol. 38, no. 6, pp. 3340–3356, 2022

  9. [17]

    Trajectory generation and tracking control for aggressive tail-sitter flights,

    G. Lu, Y . Cai, N. Chen, F. Kong, Y . Ren, and F. Zhang, “Trajectory generation and tracking control for aggressive tail-sitter flights,”The International Journal of Robotics Research, vol. 43, no. 3, pp. 241– 280, 2024

  10. [18]

    Differential flatness transformations for aggressive quadrotor flight,

    B. Morrell, M. Rigter, G. Merewether, R. Reid, R. Thakker, T. Tzane- tos, V . Rajur, and G. Chamitoff, “Differential flatness transformations for aggressive quadrotor flight,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 5204–5210

  11. [19]

    Back to newton’s laws: Learning vision-based agile flight via differentiable physics,

    Y . Zhang, Y . Hu, Y . Song, D. Zou, and W. Lin, “Back to newton’s laws: Learning vision-based agile flight via differentiable physics,” arXiv preprint arXiv:2407.10648, 2024

  12. [20]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”CoRR, vol. abs/1707.06347, 2017. [Online]. Available: http://arxiv.org/abs/ 1707.06347

  13. [21]

    Nonlinear dynamic modeling for high performance control of a quadrotor,

    M. Bangura and R. Mahony, “Nonlinear dynamic modeling for high performance control of a quadrotor,” inProceedings of the 2012 Australasian Conference on Robotics and Automation, ACRA 2012, ser. Australasian Conference on Robotics and Automation, ACRA, 2012, 2012 Australasian C...

  14. [22]

    Flightmare: A flexible quadrotor simulator,

    Y . Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” inConference on Robot Learning, 2020

  15. [2025]

    Available: https://www.science.org/doi/abs/10.1126/ scirobotics.adp9905

    [Online]. Available: https://www.science.org/doi/abs/10.1126/ scirobotics.adp9905

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.