{"id":"db6b8a3a-3273-439b-8a4a-ff51a2434689","arxiv_id":"2507.10914","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A non-episodic, model-based online optimizer, M-GAPS, tunes quadrotor and car controllers on hardware faster than episodic baselines and adapts to wind and payload changes.","lead":"This paper tests an online algorithm, M-GAPS, that retunes a robot's controller parameters during a single continuous flight or drive, without resets or episodes. It shows on quadrotors and a small car that this approach adapts faster than episodic baselines and a hand-tuned controller, including under unmodeled wind and added payload.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central empirical claim is credible, but its 'principled' framing rests on an unverified contractiveness conjecture; if the gradient approximation G_t (Eq. 6) is biased, the theoretical guarantees do not apply and the reported speed may not be robust.","rationale":"The reader's weakest_assumption identifies the unverified contractiveness conjecture as the load-bearing premise for the theoretical framing. I agree that this is the most significant soft spot: the algorithm's practical success is the paper's central contribution, but the claim that it is a principled extension of online policy optimization depends on the accuracy of G_t, which is guaranteed only under contractiveness. The paper itself acknowledges the conjecture in Section IV-C and the limitations section, so the concern is not hidden; it is a genuine gap in support. I considered whether the single-trial large-disturbance results (Figures 5 and 7) are a more direct threat to the empirical claim, but the strong per-lap patterns and the parameter-evolution evidence make the hardware results reasonably convincing even as single trials. The contractiveness issue instead undermines the generalization argument: if G_t is only accidentally accurate on these trajectories, the method's good performance cannot be expected on other platforms or disturbances. The proposed re-simulation test would settle empirically whether the gradient approximation is faithful, which is the key mechanism behind both the practical success and the theoretical justification. Since the paper already frames the contractiveness assumption as a conjecture and asks for empirical validation, the reader's conditional acceptance is appropriate; publishing the gradient-accuracy comparison or a direct check of contraction factors would either support or refute the framing, so I recommend no change to the verdict.","tokens_in":15607,"tokens_out":12725,"duration_ms":158350,"concrete_test":"Using the logged hardware state-action trajectories from the detuned-initialization experiment and the same nominal model, re-simulate for a sample of timesteps t (e.g., every 100 steps) the exact surrogate gradient ∇F_t(θ_t) by replaying the trajectory from the measured initial state with a constant policy parameter θ_t, following Eqs. 3-4. Compare the cosine similarity and relative magnitude of ∇F_t(θ_t) with the online G_t from Eq. 6. If the cosine similarity is consistently above 0.9 and the magnitude ratio stays bounded, the gradient approximation is empirically validated for these runs; if it is frequently low or negative, contractiveness is not holding in practice, and the theoretical guarantees of M-GAPS cannot be invoked for the hardware results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"M-GAPS updates parameters with G_t from Eq. 6, which is positioned in Section III-A1 as an approximation to the exact surrogate gradient ∇F_t(θ_t), valid when the closed-loop dynamics are contractive. The paper does not prove contractiveness for the logarithmic policy class; Section IV-C only conjectures it, citing the SO(3) log-parameterization and actuation limits as unresolved issues. If contractiveness fails, the O(1) sensitivity recursion (5) can accumulate error, and G_t may be a biased or even misdirected gradient. The local regret bound (2) and policy regret (1) from [20] then do not apply, so the empirical speed of adaptation is left without theoretical grounding. This matters because the abstract and introduction claim that M-GAPS extends online optimization in a principled way; without contractiveness, the hardware results are only anecdotes about a particular trajectory and initialization. The concern is not that the experiments are wrong, but that their interpretation as evidence for a principled, general-purpose method is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies non-episodic online policy optimization for tuning robot controllers, with a focus on nonlinear geometric quadrotor controllers. It implements the M-GAPS algorithm, compares it against two episodic baselines (DiffTune and OPRF), and introduces a logarithmic reparameterization of the policy gains intended to improve the optimization landscape. The experimental evaluation is conducted on real hardware: a Crazyflie 2.0 quadrotor in three scenarios (detuned initialization, time-varying wind, heavy payload) and a 1:6-scale Ackermann-steered car. The central claims are that M-GAPS finds near-optimal parameters more quickly than the episodic baselines, especially when their episode length is not well chosen, and that it substantially outperforms an expert-tuned baseline under unmodeled disturbances.","tokens_in":15856,"tokens_out":3941,"duration_ms":45457,"significance":"If the empirical results hold, this is a valuable demonstration that a non-episodic, model-based online policy optimization algorithm can run in real time on embedded hardware and adapt to unmodeled disturbances without modifying the dynamics model. The detuned-initialization comparison is carefully executed with five trials and error bars (Fig. 2), and the episode-length sensitivity study (Fig. 4) provides a useful practical insight. The logarithmic reparameterization is a simple and potentially transferable contribution. The paper is also honest in its Limitations section about the unverified contractiveness assumption, which is a point in its favor. The main weaknesses are the absence of trial counts and error bars for the large-disturbance experiments and the gap between the theoretical framing and the empirically verified conditions.","major_comments":[{"comment":"The large-disturbance experiments that support the central adaptivity claim (wind, payload, and car) report no trial counts or error bars. Only Fig. 2 explicitly states '±1 standard deviation over 5 trials.' If Figs. 5, 7, and 9 are single traces, the claim that M-GAPS 'substantially outperforms' the expert baseline under wind and payload is not statistically supported, and this is load-bearing for the paper's main conclusion. Please report the number of trials and per-trial variability (e.g., mean ± std over at least three to five runs for each condition), and state whether the disturbance phase and initial conditions were held fixed across trials.","section":"Section V-C, Figs. 5, 7, 9"},{"comment":"The paper explicitly leaves contractiveness as a conjecture in Section IV-C, yet Section III-A1 states that 'this, in turn, leads to the regret guarantees of M-GAPS [20].' Because the approximation G_t in Eq. (6) is only justified when the closed-loop dynamics are contractive, the theoretical grounding of the quadrotor experiments is not established. The Limitations section (VIII) acknowledges this, but the main-text phrasing still overstates the connection between the theory and the empirical results. The authors should either (a) provide empirical evidence of contractiveness (e.g., measure the decay of the sensitivity y_t or closed-loop contraction during the experiments), or (b) explicitly decouple the empirical claims from the regret framework and avoid implying that the regret bounds apply to the systems tested.","section":"Section IV-C and Section III-A1"},{"comment":"The 'hindsight best' and 'hindsight worst' episode lengths for DiffTune appear to be selected using the same experimental data shown in Fig. 4. This is a post-hoc selection and makes the comparison in Fig. 2 somewhat circular: DiffTune⋆ is the best-performing variant selected from the same runs. The sensitivity analysis itself is valuable, but the claim that 'DiffTune is close when the episode length is optimal' would be stronger if the selection protocol were specified in advance or if the full episode-length sweep were presented as a hyperparameter study with appropriate caveats about multiple comparisons.","section":"Section V-B, Fig. 4"}],"minor_comments":[{"comment":"There is a typo: 'data-inefficient learning that that impedes adaptivity' should be 'that impedes adaptivity.'","section":"Section I, first paragraph"},{"comment":"The statement 'The logarithmic map log : SO(n) 7→ so(n) is a continuous function' is imprecise for n=3 because the logarithm is multi-valued; please specify the chosen branch (e.g., the principal branch with rotation angle in [0, π]) or clarify that a branch is fixed.","section":"Section IV-A"},{"comment":"In the definition softclamp(x, y) = y tanh(x/y), the behavior for y = 0 is undefined; since Bxy and Bz are positive in context, this is not a logical error, but a brief note would improve clarity.","section":"Section IV-B"},{"comment":"The caption says 'Row: Optimizer. Left: Gains for horizontal motion axes. Right: Gains for vertical motion axis,' but each optimizer row contains two panels that are not explicitly labeled in the caption; consider referring to panel letters or adding explicit subfigure labels.","section":"Fig. 3 caption"},{"comment":"The line 'where δ >0 is the discretization interval, 0 < Cξ, Cτ ≪ 1/δ are time delay constants' appears to have a typo: it should likely read '0 < Cξ, Cτ δ ≪ 1' or similar; please check the intended inequality.","section":"Appendix A.1"},{"comment":"The term 'quasi-regret' is used informally; consider giving a formal definition (e.g., cumulative cost difference from the expert parameter θm) at first use.","section":"Section V-A"}],"recommendation":"major_revision","confidential_remarks":"This is a strong empirical paper that is likely to be of interest to the robotics and learning-for-control communities. The self-citation to the authors' own prior work [20] is appropriate and not a concern. The main revision need is experimental transparency: the large-disturbance results need trial counts and variability measures, and the theoretical framing should be reconciled with the admitted lack of contractiveness verification. The car experiment, while a nice extension, also needs at least a statement of the number of runs. The paper is within the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nShort version: this is a credible hardware study of a non-episodic model-based online policy optimization algorithm, and it does what it says. It's not a breakthrough in theory, but the empirical work is careful enough to be worth a serious referee.\n\nWhat's actually new: M-GAPS itself is from Lin et al. [20]. The new content is the practical deployment on Crazyflie hardware, the logarithmic gain reparameterization, the so(3) state parameterization, and the comparison with DiffTune and OPRF under detuned initialization, wind, and payload. The detuned-initialization experiment is the strongest part: five trials, error bars, and the result that M-GAPS converges to near-expert quasi-regret faster than the episodic baselines, especially when DiffTune's episode length is not favorably chosen. The wind and payload experiments support the adaptivity claim qualitatively, and the car demo shows the method generalizes.\n\nThe paper is honest about its main theoretical gap: the contractiveness assumption needed for the regret bounds from [20] is not verified for this controller and state parameterization; Section IV-C says only a conjecture. The limitations section repeats this. So the framing as a principled extension of online optimization is partly an act of faith, but the paper does not hide it. I would not call the empirical results anecdotes; they are real comparisons with a carefully chosen baseline. The larger issue is reproducibility: no code, no data release, single-trace figures for the large-disturbance experiments, and several hyperparameters (learning rates, episode lengths, OPRF's epsilon, cost weights) that are not fully specified in the text. That is fixable.\n\nOne more concern: the claim 'finds near-optimal parameters' uses the manually tuned θm as a proxy for optimality, which is reasonable in context but is quasi-regret, not policy regret. The paper says this explicitly.\n\nOverall: the core claim is plausible and the experiments are mostly well run. The soft spots are reproducibility and the unverified contractiveness, both acknowledged or addressable. I'd send it to review; it deserves a serious look, and the authors should be pushed to release code/data and to state hyperparameters. I would cite this if I work on online policy optimization for robots.","headline":"A credible hardware study of non-episodic online policy optimization with honest limitations, a reproducibility gap, and an unverified but acknowledged contractiveness assumption; worth a serious referee.","tokens_in":16424,"tokens_out":2811,"would_cite":true,"duration_ms":28968,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a single continuous trajectory with no resets, M-GAPS tunes a nonlinear quadrotor controller to near-expert performance in about 15 seconds, beats episodic baselines, and adapts to unmodeled wind and payloads.","keywords":["non-episodic online policy optimization","M-GAPS","quadrotor geometric control","controller parameter tuning","model-based policy gradient","hardware experiments","online gradient descent","adaptive control"],"falsifier":"Simulate or run the same figure-8 experiment while comparing the approximate gradient $G_t$ from the paper's equation (6) with the true surrogate gradient $\\nabla F_t(\\theta_t)$ computed by full re-simulation at each timestep; if their inner product is often negative or the sensitivity error grows without bound, the contractiveness assumption fails and the theory no longer covers the method.","tokens_in":15384,"feed_emoji":"🚁","tokens_out":10495,"duration_ms":106517,"temperature":0.7,"pith_summary":"Tuning a nonlinear robot controller normally means either hand-tuning or episodic optimization that resets the problem, but a deployed robot faces a single continuous trajectory with time-varying dynamics, costs, and goals. This paper argues that model-based online policy optimization can work non-episodically on real hardware: it implements M-GAPS, an algorithm that keeps one sensitivity matrix per timestep and updates the policy parameters with an approximate gradient that costs $O(1)$ per step. On a quadrotor with a nonlinear geometric controller, M-GAPS recovers near-expert tracking after roughly 15 seconds from a detuned start, beats episodic model-based and model-free baselines, and keeps improving under a periodic fan array and a heavy payload. The paper also shows the same method tunes a 1:6-scale Ackermann car, cutting tracking error about fivefold within one lap. The significance is a practical middle path between adaptive control, which is too restrictive in policy class, and model-free reinforcement learning, which is too data-hungry for fast adaptivity.","feed_headline":"No episodes needed: optimizer tunes a quadrotor in 15 seconds","feed_subtitle":"Single-trajectory model-based tuning beats episodic baselines on hardware and adapts to wind and heavy payloads.","key_machinery":"The load-bearing object is the $O(1)$ sensitivity recursion $y_{t+1} = (\\partial g_t/\\partial x_t + \\partial g_t/\\partial u_t\\, \\partial \\pi_t/\\partial x_t) y_t + \\partial g_t/\\partial u_t\\, \\partial \\pi_t/\\partial \\theta_t$, together with the resulting approximate gradient $G_t = (\\partial f_t/\\partial x_t + \\partial f_t/\\partial u_t\\, \\partial \\pi_t/\\partial x_t) y_t + \\partial f_t/\\partial u_t\\, \\partial \\pi_t/\\partial \\theta_t$. This is an online, non-episodic approximation of the ideal surrogate-cost gradient $\\nabla F_t(\\theta_t)$, where $F_t(\\theta)$ is the cost at time $t$ of having used the same parameter $\\theta$ at every earlier timestep; computing the true gradient would require re-simulating the whole trajectory from scratch at every step. M-GAPS instead treats the deployed parameter as if it had been constant and differentiates along the actually visited trajectory. The validity of this approximation rests on closed-loop contractiveness: each fixed parameter value steers the state toward its own preferred trajectory at an exponential rate, so slow parameter changes keep the sensitivity matrix accurate. Two reparameterizations improve the landscape: the rotation state lives in the Lie algebra $\\mathfrak{so}(3)$ so the state space is Euclidean, and the positive feedback gains are optimized through $\\theta = \\log \\vartheta$, matching the empirically observed scaling of gradient sensitivity.","core_discovery":"The paper's central claim is that M-GAPS, a model-based non-episodic online policy optimizer, is practical on real quadrotor hardware and finds near-optimal controller parameters more quickly than episodic baselines, even when the dynamics model is imperfect. The algorithm maintains a sensitivity state and updates $\\theta_{t+1} = \\theta_t - \\eta G_t$, using a gradient estimate taken along the actually visited trajectory while ignoring the fact that the policy parameter has been changing. With a logarithmic reparameterization of the feedback gains and an $\\mathfrak{so}(3)$-valued rotation state, the paper reports that M-GAPS closes most of the gap to an expert-tuned parameter in about 15 seconds, is less sensitive than the model-based episodic baseline to the choice of episode length, and substantially outperforms the expert parameter under periodic fan wind and a 60 percent payload increase; the same method cuts tracking error by about five times within one lap on a 1:6-scale car. The theory carries local-regret guarantees if the closed-loop dynamics are contractive, which the paper conjectures rather than proves for this controller.","pith_inferences":["A natural extension, not tested in the paper, is to feed an online dynamics estimate into the same sensitivity recursion, letting the optimizer track both policy parameters and model parameters.","The differentiability requirement will likely bind on contact-rich systems such as walking robots, and a testable next step is to benchmark M-GAPS on such systems and measure how often gradient steps worsen the cost near discontinuities.","The parameter trajectories differ sharply between wind and payload scenarios, which suggests the adapted gains could serve as a diagnostic for what disturbance is present, a use the paper does not explore.","Because the local regret guarantee does not rule out bad local minima, a deployed implementation should monitor gradient norm or cost and re-initialize, as with any nonconvex optimizer."],"forward_implications":["M-GAPS removes episode length as a hyperparameter and avoids the large parameter steps that episodic updates cause.","Because it uses analytic derivatives of an approximate model rather than model-free exploration, it runs at 500 hertz on a 168 MHz microcontroller with 192 kB of memory.","The wind and payload experiments show that a fixed nominal dynamics model can support continued improvement under unmodeled disturbances, so model error does not by itself block online policy optimization.","The car experiment indicates the method transfers across robot platforms and can serve as automatic controller initialization on new hardware even when online adaptivity is not the main goal.","If the contractiveness conjecture is verified, M-GAPS inherits regret guarantees that the episodic baselines do not have."],"supporting_citations":[{"why":"Introduces M-GAPS and supplies its regret and local-regret guarantees for non-episodic online policy optimization; this is the algorithm the paper implements on hardware.","marker":"[20]"},{"why":"Supplies the surrogate-cost formulation and the no-regret online gradient descent analysis under contractive perturbations that motivates the approximate update.","marker":"[19]"},{"why":"Provides the model-based episodic baseline used for comparison and the sensitivity recursion that M-GAPS shares.","marker":"[7]"},{"why":"Provides the model-free episodic baseline OPRF and the residual-feedback gradient estimator used in the comparison.","marker":"[42]"},{"why":"Supplies the geometrically stabilizing quadrotor controller structure that the paper's policy class extends, and the exponential-stability result behind the contractiveness conjecture.","marker":"[18]"},{"why":"Supplies differential flatness for the quadrotor, used to compute desired angular velocity and shape the policy class.","marker":"[25]"},{"why":"Generates the embedded C++ code for the analytic derivatives that M-GAPS and the model-based baseline need at 500 hertz.","marker":"[24]"},{"why":"Supplies the expert-tuned baseline parameters and the hardware testbed used for the quadrotor experiments.","marker":"[30]"},{"why":"Supplies the car's lateral dynamics model used in the Ackermann-steered car experiment.","marker":"[22]"}],"fun_headline_variants":["No-reset tuning: quadrotor adapts in 15 seconds flat","Single-pass optimizer tunes quadrotor in 15s","Online policy optimizer tunes drone in 15s without episodes","Non-episodic optimizer adapts quadrotor to wind and payload"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything theoretical rests on the closed-loop quadrotor under the logarithmic policy class being contractive, so the one-step sensitivity recursion tracks the true surrogate-cost gradient; the paper conjectures this rather than proving it.","fun_headline_variants_meta":{"raw":{"variants":["No-reset tuning: quadrotor adapts in 15 seconds flat","Single-pass optimizer tunes quadrotor in 15s","Online policy optimizer tunes drone in 15s without episodes","Non-episodic optimizer adapts quadrotor to wind and payload"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001126,"raw_usage":{"total_tokens":4697,"prompt_tokens":978,"completion_tokens":3719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3645}},"tokens_in":594,"tokens_out":3719,"duration_ms":27829,"temperature":1.0,"reasoning_tokens":3645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:21:59.164438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate or run the same figure-8 experiment while comparing the approximate gradient $G_t$ from the paper's equation (6) with the true surrogate gradient $\\nabla F_t(\\theta_t)$ computed by full re-simulation at each timestep; if their inner product is often negative or the sensitivity error grows without bound, the contractiveness assumption fails and the theory no longer covers the method.","supporting_citations":[{"cited_title":"Preiss, Fengze Xie, Emile Anand, Soon-Jo Chung, Yisong Yue, and Adam Wierman","cited_arxiv_id":null,"evidence_quote":"Introduces M-GAPS and supplies its regret and local-regret guarantees for non-episodic online policy optimization; this is the algorithm the paper implements on hardware."},{"cited_title":"Preiss, Emile Anand, Yingying Li, Yisong Yue, and Adam Wierman","cited_arxiv_id":null,"evidence_quote":"Supplies the surrogate-cost formulation and the no-regret online gradient descent analysis under contractive perturbations that motivates the approximate update."},{"cited_title":"Difftune: Autotuning through autodifferentiation","cited_arxiv_id":null,"evidence_quote":"Provides the model-based episodic baseline used for comparison and the sensitivity recursion that M-GAPS shares."},{"cited_title":"Zavlanos","cited_arxiv_id":null,"evidence_quote":"Provides the model-free episodic baseline OPRF and the residual-feedback gradient estimator used in the comparison."},{"cited_title":"Harris McClamroch","cited_arxiv_id":null,"evidence_quote":"Supplies the geometrically stabilizing quadrotor controller structure that the paper's policy class extends, and the exponential-stability result behind the contractiveness conjecture."},{"cited_title":"Minimum snap trajectory generation and control for quadrotors","cited_arxiv_id":null,"evidence_quote":"Supplies differential flatness for the quadrotor, used to compute desired angular velocity and shape the policy class."},{"cited_title":"SymForce: Symbolic Com- putation and Code Generation for Robotics","cited_arxiv_id":null,"evidence_quote":"Generates the embedded C++ code for the analytic derivatives that M-GAPS and the model-based baseline need at 500 hertz."},{"cited_title":"Preiss, Wolfgang H ¨onig, Gaurav S","cited_arxiv_id":null,"evidence_quote":"Supplies the expert-tuned baseline parameters and the hardware testbed used for the quadrotor experiments."},{"cited_title":"Preiss, Jedidiah Alindogan, Matthew Anderson, and Soon-Jo Chung","cited_arxiv_id":null,"evidence_quote":"Supplies the car's lateral dynamics model used in the Ackermann-steered car experiment."}],"review_version":1}