REVIEW 4 major objections 6 minor 28 references
Multi-Task Reinforcement Learning for Quadrotors
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read One policy learns to stabilize, track, and race a quadrotor.
desk verdict Genuine first MTRL-for-quadrotors paper with real-world transfer; the central sample-efficiency claim needs error bars before I'd fully trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the shared dynamics encoder, a small neural network that maps the platform-state part of the observation, namely position, rotation, linear and angular velocity, and previous action, into a 32-dimensional latent embedding used by every task. Because the paper assumes the transition probability is identical across tasks for a fixed quadrotor, this shared embedding is meant to capture everything about the vehicle's dynamics, while task-specific encoders contribute only task-dependent cues. The policy is trained with PPO using a multi-critic scheme: one critic per task evaluates that task's value function, and the actor is shared. The paper also relies on not sharing the feature encoder with the critics, citing improved policy performance.
What would settle it
Train the same architecture on two tasks with deliberately different dynamics, for example one task with a heavy payload and another without, or two different motor time constants, and compare the shared-encoder policy against single-task policies. If the multi-task policy's sample-efficiency or converged performance no longer beats single-task training once dynamics differ, the identical-transition assumption is load-bearing; the paper provides no such ablation.
Extended reading notes
Core claim
The paper claims to present the first multi-task reinforcement learning architecture for quadrotor control, in which a single policy with a shared actor, a shared dynamics encoder, and separate critics per task learns three control tasks more sample-efficiently than training each task independently, without giving up converged performance. The shared encoder consumes the observation dimensions common to all tasks (position, orientation, linear and angular velocity, and previous action), while task-specific encoders process inputs such as gate positions or desired velocity; the resulting embeddings are concatenated and fed to the shared actor. Ablations show that fusing shared and task-specific information is essential: sharing only the actor gives results close to single-task learning, while separating the encoders entirely fails at racing. Real-world deployment of one policy on the same platform achieved 100 percent success on all three tasks, with metrics close to simulation.
Load-bearing premise
The whole architecture banks on the claim that the quadrotor's transition dynamics are identical across all three tasks because the platform is fixed; if real conditions such as battery voltage, payload, or aerodynamic regime change the dynamics between tasks, the shared encoder could be forced to conflate them and the transfer benefit could vanish or reverse.
Editorial extensions
If this is right
- Training one policy for several flight skills is feasible with less data than training each skill separately, which matters when real-world experience is expensive.
- Adding a new task on the same quadrotor platform may require only a new task-specific encoder and critic, reusing the learned dynamics representation.
- The sample-efficiency advantage is largest at intermediate training budgets; at 20 million steps the multi-task policy reduced stabilization time by 18 percent and gate-passing error by 16 percent relative to single-task RL.
- The design warns that naively sharing state information without fusing task-specific inputs can collapse on conflicting tasks, so architecture details matter.
- A converged multi-task policy can match specialized policies on all measured metrics, lowering the incentive to train per-task controllers.
Reading between the lines
- If the identical-dynamics assumption holds only approximately, the shared encoder may entangle task-specific dynamic effects; a natural extension is testing with varying payloads, battery voltages, or motor constants to see when sharing starts to hurt.
- The same shared-encoder-plus-task-specific-encoder recipe could be applied to other platforms with invariant dynamics, such as fixed-wing aircraft or underwater vehicles, where tasks differ mainly in objectives.
- The framework's reliance on task identifiers by observation length suggests a path to continual learning: adding a task with a novel observation type may require minimal retraining of the shared encoder.
- The real-world tests used motion capture; the approach's benefit may interact with perception noise, so vision-based multi-task flight is an unproven but plausible next step.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multi-task reinforcement learning (MTRL) framework for quadrotor control. The method uses a shared encoder for dynamics-related observations (position, orientation, velocities, previous action), task-specific encoders for task-dependent observations, a shared actor, and per-task critic networks. It is evaluated on three tasks—autonomous racing, high-speed stabilization, and velocity tracking—using the Flightmare simulator and the Agilicious quadrotor platform. The central claim is that this MTRL approach achieves better sample efficiency and final task performance than single-task RL baselines, and that the same policy transfers to real-world flight. The paper reports 10-seed averages for all methods, plus a real-world deployment of the MTRL policy.
Significance. If the central claim is supported, the work is a useful contribution: a single control policy for multiple quadrotor tasks is practically relevant, and the architecture—shared dynamics encoder, per-task critics, fused task-specific observations—is well motivated by the invariant physics of a fixed quadrotor platform. The evaluation includes a fair comparison (same reward functions, same simulator, same hyperparameters for all methods), an ablation of sharing strategies, and real-world validation. The main weakness is that the headline superiority claim rests entirely on point estimates: no standard deviations, confidence intervals, or significance tests are reported, and several 40M-sample margins are small. This must be fixed before the sample-efficiency conclusion can be considered established.
major comments (4)
- [IV.E, Table III, Fig. 4] The central claim of sample-efficiency superiority is supported only by averages over 10 seeds, with no standard deviations, confidence intervals, or pairwise significance tests. This is load-bearing because several converged 40M-sample differences are small: racing MGE is 0.152 vs 0.159 vs 0.163, tracking error is 1.21 vs 1.44 vs 1.18, and thalf is 0.35 vs 0.39 vs 0.41. Without seed-level distributions, the reader cannot determine whether the 20M-sample advantages (which are larger) reflect a real effect or run-to-run PPO noise, and the 'not worse at convergence' claim cannot be verified. Please report per-seed results, error bars in Fig. 4, and statistical tests (e.g., paired bootstrap or Wilcoxon signed-rank) for the comparisons in Table III.
- [IV.C and IV.G] The MTRL-Separate ablation is used to argue that fusing shared and task-specific information is necessary, but the paper only reports that this baseline 'cannot fly at all' in racing and then gives a speculative explanation ('the policy will then prefer learning the rather easier task'). No training curves, task-gradient diagnostics, or other evidence are provided for this mechanism. Since the necessity of the fusion design is one of the main architectural conclusions, please either add supporting analyses (e.g., per-task loss curves, gradient-conflict measures, or a controlled variant) or weaken the causal claim to a descriptive statement.
- [IV.F and Abstract] The abstract states that the framework is 'validated both in simulation and real-world scenarios' and 'outperforms baseline approaches'; however, Section IV.F only evaluates the MTRL policy in the real world and does not run any baseline policy there. The real-world results demonstrate transfer and feasibility, not superiority over baselines. Please rephrase the abstract and Section IV.F to distinguish the simulation-based superiority claim from the real-world feasibility demonstration, or add baseline comparisons in the real-world experiments.
- [III.D] The framework's knowledge-sharing premise is the identical-transition-probability assumption stated in Section III.D. The paper correctly identifies this as the primary motivation, but it does not test robustness to dynamics variation (e.g., battery voltage drop, changing mass, or aerodynamic effects at different speeds). If the transition dynamics differ between tasks or over time, the shared encoder could entangle task-specific dynamics with shared state, potentially negating the transfer. Please add a discussion of this limitation and, ideally, an ablation with perturbed dynamics (e.g., mass or inertia variation) to bound the applicability of the method.
minor comments (6)
- [IV.B and IV.C] The spelling 'MTRL-Seperate' is inconsistent; it should be 'MTRL-Separate' throughout.
- [III.C.2] The definitions of δp1 and δp2 are confusing: the text first says they are relative position differences of gate corners with respect to the drone, then says δp1 is the difference 'between the current quadrotor position' and gate corners, and δp2 is the positional difference 'between the next gate to pass and the gate after the next gate.' Please rewrite these two sentences to unambiguously define the two vectors.
- [III.B, Eq. (1)] The symbol ⊙ in qW B⊙c/m is not defined. Please specify whether this is a quaternion product and how the body-frame vector c is transformed.
- [IV.E.2] In the sentence 'our MTRL approach demonstrates a much better task performance than all of the baseline approaches,' the phrase 'much better' is not supported without uncertainty quantification; please use quantitative language consistent with the statistical analysis.
- [IV.A] The sentence 'for the shared dynamic encoder, we use a three-layer MLP with 19 neurons in the input layers' is imprecise: it should say '19-dimensional input' or '19 neurons in the input layer,' not 'layers.'
- [Table III] In the MTRL-Separate row, 'crash crash' appears under MGE and LT; please use a dash or 'N/A' for metrics that are undefined when the policy never completes a lap.
Circularity Check
No significant circularity: the multi-task advantage is measured against independent single-task baselines, with self-citations confined to infrastructure and reward design.
full rationale
The paper's central claim is an empirical superiority result: MTRL is more sample-efficient and no worse at convergence than single-task RL. This is supported by Table III and Fig. 4, where all methods use the same PPO algorithm, the same per-task reward functions, and the same number of training samples; the MTRL hyperparameters were fixed based on single-task performance, so the comparison is not constructed in favor of MTRL. The reward functions (Eqs. 3-8) and the identical-transition-probability assumption (Section III.D) are inputs to the method, not outputs derived from the claimed result. The racing reward is 'adjusted based on [1]', which is a self-citation, but Eq. 4 spells out the reward components explicitly and the constants in Table I are fixed; citing one's own prior reward design is not circular because the reward is a specification, not a prediction. Flightmare [27] and Agilicious [28] are self-cited infrastructure, but using one's own simulator and hardware does not force the comparative outcome, and the baselines run in the same infrastructure. The shared-dynamics assumption is justified by the physical model in Eq. 1 and by the fixed quadrotor platform, and it is stated explicitly as an assumption rather than derived from the target result. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via self-citation. The absence of seed-level variance or significance tests in Table III is a statistical reporting weakness, but that is a correctness risk, not circularity.
Assumptions & free parameters
free parameters (6)
- Racing reward coefficients alpha1-alpha7 =
0.5, 0.025, -1.0, -2e-4, -5e-4, -5, -10 (Table I)
- Stabilization reward coefficients beta1-beta6 =
-2e-3, -2e-4, -4e-5, -1e-5, -1e-4, 10 (Table I)
- Tracking reward coefficients lambda1-lambda3 =
-2e-4, -1.2e-3, -1e-4 (Table I)
- Curriculum parameters =
10% speed increase per 100k samples (stabilization); 1 m/s per 100k samples (tracking); upper limits not fully specified
- Network hyperparameters =
Shared encoder 3-layer MLP, 19-128-32; task encoder 3-layer MLP, variable-128-32; actor 2x256
- PPO hyperparameters =
not reported
assumptions (4)
- domain assumption Quadrotor dynamics follow the rigid-body model in Eq. (1) with collective thrust and body torques.
- domain assumption Transition probabilities are identical across all tasks (Section III.D).
- standard math PPO reliably optimizes the multi-task objective in Eq. (2).
- domain assumption The racing reward decomposition from [1] is a valid proxy for racing performance.
Cite this review
Pith. "Pith review of Multi-Task Reinforcement Learning for Quadrotors." pith.science (2026). https://pith.science/paper/SN64RNZN
@misc{pith2026241212442,
author = {Pith},
title = {Pith review of: Multi-Task Reinforcement Learning for Quadrotors},
year = {2026},
howpublished = {\url{https://pith.science/paper/SN64RNZN}},
note = {Machine review of arXiv:2412.12442}
}
read the original abstract
Reinforcement learning (RL) has shown great effectiveness in quadrotor control, enabling specialized policies to develop even human-champion-level performance in single-task scenarios. However, these specialized policies often struggle with novel tasks, requiring a complete retraining of the policy from scratch. To address this limitation, this paper presents a novel multi-task reinforcement learning (MTRL) framework tailored for quadrotor control, leveraging the shared physical dynamics of the platform to enhance sample efficiency and task performance. By employing a multi-critic architecture and shared task encoders, our framework facilitates knowledge transfer across tasks, enabling a single policy to execute diverse maneuvers, including high-speed stabilization, velocity tracking, and autonomous racing. Our experimental results, validated both in simulation and real-world scenarios, demonstrate that our framework outperforms baseline approaches in terms of sample efficiency and overall task performance.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,
Y . Song, A. Romero, M. Mueller, V . Koltun, and D. Scaramuzza, “Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,” Science Robotics, p. adg1462, 2023
work page 2023
-
[2]
Autonomous power line inspection with drones via perception-aware mpc,
J. Xing, G. Cioffi, J. Hidalgo-Carri ´o, and D. Scaramuzza, “Autonomous power line inspection with drones via perception-aware mpc,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 1086–1093
2023
-
[3]
Con- trastive learning for enhancing robust scene transfer in vision-based agile flight,
J. Xing, L. Bauersfeld, Y . Song, C. Xing, and D. Scaramuzza, “Con- trastive learning for enhancing robust scene transfer in vision-based agile flight,” in 2024 IEEE international conference on robotics and automation (ICRA). IEEE, 2024
work page 2024
-
[4]
A benchmark comparison of learned control policies for agile quadrotor flight,
E. Kaufmann, L. Bauersfeld, and D. Scaramuzza, “A benchmark comparison of learned control policies for agile quadrotor flight,” in International Conference on Robotics and Automation (ICRA) , 2022
work page 2022
-
[5]
Champion-level drone racing using deep reinforcement learning,
E. Kaufmann, L. Bauersfeld, A. Loquercio, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforcement learning,” Nature, vol. 620, no. 7976, pp. 982–987, Aug 2023
work page 2023
-
[6]
Environ- ment as policy: Learning to race in unseen tracks,
H. Wang, J. Xing, N. Messikommer, and D. Scaramuzza, “Environ- ment as policy: Learning to race in unseen tracks,” arXiv preprint arXiv:2410.22308, 2024
-
[7]
Conflict-averse gradi- ent descent for multi-task learning,
B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu, “Conflict-averse gradi- ent descent for multi-task learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 18 878–18 890, 2021
work page 2021
-
[8]
Scaling up multi-task robotic reinforcement learning,
D. Kalashnikov, J. Varley, Y . Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, “Scaling up multi-task robotic reinforcement learning,” in Conference on Robot Learning , 2022
work page 2022
Show all 28 references
-
[9]
Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention,
A. Gupta, J. Yu, T. Z. Zhao, V . Kumar, A. Rovinsky, K. Xu, T. De- vlin, and S. Levine, “Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention,” in 2021 IEEE International Conference on Robotics and Auto...
2021
-
[10]
Sharing knowledge in multi-task deep reinforcement learning,
C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters, “Sharing knowledge in multi-task deep reinforcement learning,” International Conference on Learning Representations , 2020
2020
-
[11]
Limt: Language-informed multi-task visual world models,
E. Aljalbout, N. Sotirakis, P. van der Smagt, M. Karl, and N. Chen, “Limt: Language-informed multi-task visual world models,” arXiv preprint arXiv:2407.13466, 2024
2024 arXiv
-
[12]
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on robot learning , 2020
2020
-
[13]
Control of a quadrotor with reinforcement learning,
J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control of a quadrotor with reinforcement learning,” IEEE Robotics and Automation Letters , vol. 2, no. 4, pp. 2096–2103, 2017
2017
-
[14]
Low-level control of a quadrotor with deep model-based reinforcement learning,
N. O. Lambert, D. S. Drew, J. Yaconelli, S. Levine, R. Calandra, and K. S. Pister, “Low-level control of a quadrotor with deep model-based reinforcement learning,” IEEE Robotics and Automation Letters , vol. 4, no. 4, pp. 4224–4230, 2019
2019
-
[15]
Learning speed adaptation for flight in clutter,
G. Zhao, T. Wu, Y . Chen, and F. Gao, “Learning speed adaptation for flight in clutter,” IEEE Robotics and Automation Letters , 2024
2024
-
[16]
Collision avoidance and navigation for a quadrotor swarm using end-to-end deep reinforcement learning,
Z. Huang, Z. Yang, R. Krupani, B. S ¸enbas ¸lar, S. Batra, and G. S. Sukhatme, “Collision avoidance and navigation for a quadrotor swarm using end-to-end deep reinforcement learning,” in IEEE International Conference on Robotics and Automation (ICRA) , 2024
2024
-
[17]
Learning to fly in seconds,
J. Eschmann, D. Albani, and G. Loianno, “Learning to fly in seconds,” IEEE Robotics and Automation Letters , 2024
2024
-
[18]
CAD2RL: Real single-image flight without a single real image,
F. Sadeghi and S. Levine, “CAD2RL: Real single-image flight without a single real image,” in Robotics: Science and Systems (RSS) , 2017
2017
-
[19]
Bootstrapping reinforcement learning with imitation for vision-based agile flight,
J. Xing, A. Romero, L. Bauersfeld, and D. Scaramuzza, “Bootstrapping reinforcement learning with imitation for vision-based agile flight,” Conference on Robot Learning , 2024
2024
-
[20]
Demonstrating agile flight from pixels without state estimation,
I. Geles, L. Bauersfeld, A. Romero, J. Xing, and D. Scaramuzza, “Demonstrating agile flight from pixels without state estimation,” in Proceedings of Robotics: Science and Systems , 2024
2024
-
[21]
Multi-task reinforcement learning with soft modularization,
R. Yang, H. Xu, Y . Wu, and X. Wang, “Multi-task reinforcement learning with soft modularization,” Advances in Neural Information Processing Systems, vol. 33, pp. 4767–4777, 2020
2020
-
[22]
Lifelong robotic reinforcement learning by retain- ing experiences,
A. Xie and C. Finn, “Lifelong robotic reinforcement learning by retain- ing experiences,” in Conference on Lifelong Learning Agents , 2022
2022
-
[23]
Autonomous drone racing with deep reinforcement learning,
Y . Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza, “Autonomous drone racing with deep reinforcement learning,” in IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS) , 2021
2021
-
[24]
On the continuity of rotation representations in neural networks,
Y . Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity of rotation representations in neural networks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019
2019
-
[25]
What matters in on-policy reinforcement learning? a large-scale empir- ical study,
M. Andrychowicz, A. Raichuk, P. Sta ´nczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, et al. , “What matters in on-policy reinforcement learning? a large-scale empir- ical study,” arXiv preprint arXiv:2006.05990 , 2020
2006 arXiv
-
[26]
Proximal policy optimization algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv, 2017
2017
-
[27]
Flightmare: A flexible quadrotor simulator,
Y . Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” in Conference on Robot Learning, 2020
2020
-
[28]
Agilicious: Open-source and open-hardware agile quadrotor for vision- based flight,
P. Foehn, E. Kaufmann, A. Romero, R. Penicka, S. Sun, L. Bauersfeld, T. Laengle, G. Cioffi, Y . Song, A. Loquercio, and D. Scaramuzza, “Agilicious: Open-source and open-hardware agile quadrotor for vision- based flight,” Science Robotics, vol. 7, no. 67, 2022
2022
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.