{"id":"ecff255c-a027-4a44-b796-93b64e8a182a","arxiv_id":"2502.10429","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"CRL2RT combines classical controllers with RL in a time-interleaved Cloud-Edge design, reporting over 2500 Hz online-update control on CPUs and tracking gains of 18.3% to 60.7% in simulation.","lead":"This paper presents CRL2RT, a reinforcement learning controller that alternates classical and learned control actions to run at over 2500 Hz on a CPU while updating weights online. It reports 18.3% to 60.7% tracking improvements over PID, adaptive PID, and MRAC on a simulated dragonfly-like tandem-wing testbed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finite-time convergence and safety are asserted via an invalid gradient-domain Laplace transform and a circular Lyapunov inequality; the central theoretical claim is unsupported as written.","rationale":"The paper's central claim is that CRL2RT ensures finite-time convergence and safety while achieving over 2500 Hz control and 18.3%-60.7% tracking improvements. The empirical components are further weakened by self-referential simulation validation, but the theoretical guarantee is the least secure condition because it fails internally: the authors' equations do not imply their conclusion. Even if the semi-physical model were perfect, the abstract's 'ensures finite-time convergence and single-life adaptability' would remain unsupported. The timing and tracking results are engineering observations, but the theoretical contribution is what distinguishes CRL2RT from generic RL, and a reader cannot verify the proof from the manuscript. The reader's verdict is therefore unchanged: without a corrected derivation, the paper should not be accepted with its current claims.","tokens_in":28939,"tokens_out":14995,"duration_ms":149178,"concrete_test":"Re-derive Eq. (31) from Eq. (30) while retaining the K_{i+1} term and all initial-value terms; if the solution of the recurrence is not K_i = R_i/(1-gamma), then the Q-estimation and policy-composer rule (Eqs. 32-34 and 59-61) on which the finite-time proof rests are unproven. This single symbolic check would settle whether the central convergence claim has a valid derivation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central guarantee in the abstract depends on the Q-estimation of Eqs. (22)-(34). The step from Eq. (30) to Eq. (31) is not a valid simplification: Eq. (30) is a recurrence involving K_{i+1}, Q_i(0), r_i(0), and Q_{i+1}(0); dropping these terms and writing K_i = R_i/(1-gamma) is unjustified. Consequently, the inverse transform giving Q_i ≈ r_i/(1-gamma) in Eq. (32) and the composer update ΔQ in Eq. (34) lack a foundation. The Lyapunov computation in Eq. (72) then assumes dQ/dS < 0 and dQ/dθ < 0 (Eqs. 68-69) without proof, and Eq. (73) is imposed rather than derived; the statement that an exponentially growing product compared with polynomially growing Q^alpha proves finite-time convergence is not a valid Lyapunov argument. The parameters m_g, c*, and the existence of θ_CP satisfying Eq. (59) are unconstrained. The Lipschitz safety argument is also conditional on a finite maximum P_E,rl that is never bounded. Thus the claim that CRL2RT 'ensures finite-time convergence' is not supported by the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRL2RT, a reinforcement learning framework for real-time control of the Direct-Drive Tandem-Wing Experimental Platform (DDTWEP). The algorithm combines classical controllers (PID, Adaptive PID, MRAC) with an RL controller via a time-interleaved module and a rule-based policy composer, with cloud-side weight updates and edge-side inference. The authors claim a control frequency above 2500 Hz on standard CPUs, finite-time convergence and safety guarantees, and tracking improvements of 18.3% to 60.7% when combined with the three classical controllers, based on experiments in a semi-physical simulation.","tokens_in":29404,"tokens_out":6090,"duration_ms":59541,"significance":"If the claims were fully supported, the paper would represent a useful engineering contribution to real-time RL control, particularly the latency-focused decomposition of edge/cloud computation and the attempt to integrate RL with classical controllers at high frequency. The timing breakdown in Table 7 is informative. However, the theoretical guarantees for finite-time convergence and safety are not established by the derivations as written, and the headline performance claim is contradicted by the paper's own results in the MRAC 20 Hz configurations. The reliance on an unvalidated simulation model further limits the current significance.","major_comments":[{"comment":"The step from Eq. (30) to Eq. (31) is not a valid algebraic simplification: Eq. (30) contains the terms Q_i(0), r_i(0), gamma K_{i+1}, and Q_{i+1}(0), all of which are discarded without justification. Consequently, the Q-value estimator Q_i(theta) = r_i(theta)/(1-gamma_i) in Eq. (32) and the composer update in Eq. (34) are not derived from the Bellman equation. Since the finite-time convergence proof in Section V.C.2 relies directly on this Q-estimator, the theoretical foundation of the central claim is unsupported.","section":"Section IV.B, Eqs. (30)-(32)"},{"comment":"The finite-time Lyapunov proof is circular and incomplete. Equation (59) is stated as a relation that the policy composer must satisfy, but the existence of theta_CP and the sequence m_g is never demonstrated, and no constraints on m_g are given. Equations (68)-(69) assert the sign conditions dQ/dS < 0 and dQ/dtheta < 0 without proof, and Eq. (73) is imposed rather than derived. The final argument that an exponentially growing product dominates a polynomially growing Q^alpha does not establish the pointwise inequality in Eq. (63) that is required for finite-time Lyapunov convergence. These gaps are load-bearing for the claim that CRL2RT 'ensures finite-time convergence.'","section":"Section V.C.2, Eqs. (59)-(74)"},{"comment":"The safety guarantee of the time-interleaved module depends on a finite upper bound P_E,rl,max on the RL policy's ability to increase the tracking error. The paper never establishes such a bound for the actual policy class used in the experiments. Without this bound, the Lipschitz inequality in Eq. (51) is conditional, and the claim that the algorithm 'ensures safety' through the time-interleaved module is not supported.","section":"Section V.B, Eqs. (50)-(53)"},{"comment":"The abstract states that CRL2RT 'enhances tracking performance by 18.3% to 60.7%' when integrated with PID, Adaptive PID, and MRAC, but Table 11 reports errors of -172.0% and -115.3% for the CRL2RT(MRAC) configuration in the two 20 Hz conditions. The paper acknowledges this degradation in Section VI.D, but the unqualified abstract claim and the contribution statement in Section II are not accurate over the full set of reported experiments.","section":"Abstract and Table 11"},{"comment":"The experimental results are generated entirely in a semi-physical simulation, and the subsystem models are delegated to the authors' prior work [21] and to an unreviewed preprint [62]. No physical hardware experiments are reported, and no independent validation of the 8-DOF model is provided. The timing measurements in Table 7 appear to be single trials without error bars or code release, so the headline 2534 Hz control frequency is not established as a robust, reproducible result.","section":"Section VI.B.1 and Table 1"}],"minor_comments":[{"comment":"Section numbering is inconsistent: the text refers to 'Section 3.2' and 'Section 4.3' when the relevant sections are actually IV.B and V.C, and Section IV is titled 'Primarily,' which appears to be a typographical error.","section":"General"},{"comment":"The gradient-domain Laplace transform is not rigorously defined: Eq. (22) is dimensionally unclear, and the exponent in Eqs. (24)-(25) uses both s and the newly introduced g_d without a clear relation between these variables.","section":"Eqs. (22)-(25)"},{"comment":"The three column headers in Table 7 are ambiguous, and the corresponding frameworks are not defined precisely in the text; the table also appears to report a single measurement per item with no statistics, which should be stated explicitly.","section":"Table 7"},{"comment":"The plots are labeled 'Rewards,' but the reward function in Eq. (17) is a sum of tracking errors that the algorithm minimizes; the y-axis label and the direction of improvement should be clarified to avoid confusion.","section":"Figures 10-12"},{"comment":"There are several typographical errors, including Eq. (2) repeating T_M,2 and omitting T_M,4, Eq. (10) using inconsistent subscripts (J_MZZ, C), and reference [56] being incomplete.","section":"Editorial"}],"recommendation":"reject","confidential_remarks":"The manuscript's theoretical core and simulation fidelity depend heavily on the authors' own in-press or unreviewed prior work (refs. [21] and [62]), and the experimental data are not shared. Given the load-bearing algebraic error in the Q-estimator derivation, the circular Lyapunov argument, and the abstract's contradiction of Table 11, I do not see a straightforward revision that would make the central claims sound within the scope of a single paper. The engineering timing breakdown might be salvageable as a separate, more modest contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The 2500 Hz online-update figure is a concrete engineering data point worth taking seriously. The finite-time convergence and safety guarantee in the abstract is not: the gradient-domain Laplace transform derivation is invalid, and the Lyapunov argument is circular. The stress-test note holds up on a direct reading.\n\nWhat is actually new: the time-interleaved module and the rule-based policy composer come from the authors' earlier CRL work [21]. The new material is the Cloud-Edge load balancing, the NumPy-based matrix inference on the edge, and the integration experiment with PID, Adaptive PID, and MRAC. The timing table is the best part: a slowest control frequency of 2534.21 Hz with a breakdown of where the time goes. That is useful for anyone trying to get online-updating RL to kilohertz rates on a CPU. The authors also report the negative MRAC numbers in Table 11, which is more transparent than most papers in this area.\n\nThe soft spots are substantial. The step from Eq. (30) to Eq. (31) drops K_{i+1}, Q_i(0), and r_i(0) without justification, so Eq. (32), Q_i = r_i/(1-gamma_i), has no foundation. Everything built on it—the composer update, the 'finite-time' argument—inherits the problem. The Lyapunov proof then assumes dQ/dS<0 and dQ/dtheta<0 without proof, imposes Eq. (73) rather than deriving it, and the exponential-versus-polynomial comparison is not a valid finite-time Lyapunov argument. Eq. (59) is set by fiat with m_g unconstrained. So the 'ensures finite-time convergence and safety' claim is unsupported as written.\n\nEmpirically, all results come from a semi-physical simulation whose subsystem models are delegated to the authors' own prior work [21] and their preprint [62]. No code, no data, no error bars. The 2534 Hz number appears to be a single slowest-case measurement, not a distribution. The abstract's 18.3–60.7% range omits the two 20 Hz MRAC rows, which are -172.0% and -115.3%. That is a real overstatement, though contained: those are two of eighteen cells, and the table itself discloses them.\n\nWho this is for: people working on real-time online RL for fast mechanical systems, specifically the CPU inference optimization. They should read the timing table and ignore the proof. The citation pattern leans heavily on the authors' own inaccessible preprints for model fidelity, which makes the simulation hard to check.\n\nBottom line: I would send it to review rather than desk-reject—the timing result deserves referee scrutiny and the paper is not incoherent, just overclaimed. But I would expect the outcome to be major revision unless the theory is dropped or made correct and code/data are released. If the authors reposition it as an empirical systems note, it could be a solid contribution.","headline":"Useful CPU timing result buried under an unsupported finite-time convergence proof; the paper overclaims as written.","tokens_in":29815,"tokens_out":5307,"would_cite":false,"duration_ms":52456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning controller can run above 2500 Hz on ordinary CPUs while still updating online, and it cuts tracking error by 18.3% to 60.7% when paired with PID, Adaptive PID, or MRAC.","keywords":["concerto reinforcement learning","real-time control","online weight updates","flapping-wing aircraft","tandem-wing platform","finite-time Lyapunov convergence","edge inference"],"falsifier":"Run the trained CRL2RT policy on the physical DDTWEP with its six-component balance under the same 40 Hz and 60 Hz flapping commands and compare tracking error with the semi-physical simulation. If the 18.3% to 60.7% improvement over the classical controllers does not appear, or if the slowest control frequency drops below 2000 Hz once weight updates and data transmission are included, the paper's central claim is falsified.","tokens_in":28697,"feed_emoji":"⚙️","tokens_out":8165,"duration_ms":74008,"temperature":0.7,"pith_summary":"The paper proposes CRL2RT, a reinforcement-learning control algorithm designed to run online at high frequency on a dragonfly-inspired, direct-drive tandem-wing experimental platform. It claims CRL2RT sustains a control frequency above 2500 Hz on ordinary CPUs while still updating network weights from incoming data, and that when paired with PID, Adaptive PID, or MRAC it lowers tracking error by 18.3% to 60.7% under 40 Hz and 60 Hz flapping conditions. The core argument is that safety and finite-time convergence can be guaranteed by alternating classical and learned actions and by composing past policy updates according to a Lyapunov condition, so real-time RL need not sacrifice online learning.","feed_headline":"Online RL control breaks 2500 Hz on standard CPUs","feed_subtitle":"On a dragonfly-inspired tandem-wing rig, the same learner lifts PID, Adaptive PID, and MRAC tracking by up to 60.7%.","key_machinery":"The two load-bearing mechanisms are the Time-Interleaved module and the Rule-Based Policy Composer. The Time-Interleaved module alternates classical-control steps with RL steps so the classical controller's bounded error-reduction capability keeps the combined error Lipschitz continuous and bounds worst-case deviation. The Rule-Based Policy Composer, using a gradient-domain Laplace transform estimate of Q-values from reward histories without an edge-side critic, adds posterior-screened policy weights at the start of each descent segment so that the Lyapunov function satisfies $dQ/dt \\le -c Q^{\\alpha}$, yielding finite-time convergence. On the edge, inference is restructured as direct numerical array operations, removing tensorization and de-tensorization overhead.","core_discovery":"The central claim is that real-time reinforcement learning does not have to trade away online weight updates: by running weight updates on the cloud side, running direct matrix-operation inference on the edge side, interleaving classical and learned controllers, and injecting policy-composer weights that satisfy a finite-time Lyapunov condition, CRL2RT reaches a slowest control frequency around 2534 Hz on a general-purpose CPU while still updating from experience. The paper further claims the same framework works with three different classical controllers, improving tracking performance by 18.3% to 60.7% in the 40 Hz and 60 Hz configurations, with finite-time convergence and safety provided by the two theoretical modules.","pith_inferences":["A natural extension is to test the architecture on other high-rate mechatronic systems, such as quadruped joints or precision motion stages; that would show whether the 2500 Hz result is a property of the controller structure or of this particular eight-DOF platform.","The 20 Hz MRAC case suggests that when the classical partner already performs well the exploration signal weakens; an altered exploration schedule might recover gains in that regime, but this is not examined in the paper.","Because the same simulation model is used for training and evaluation, a physical flight or tethered hardware trial is the decisive next step, and the platform's own six-component balance makes that trial directly feasible.","The gradient-domain Laplace estimator obtains Q-values from reward histories without an edge-side critic, a trick that may carry over to other latency-sensitive RL deployments."],"forward_implications":["A 2000 Hz motion-control requirement can be met by an RL controller that updates its weights online on commodity CPUs, without FPGA or accelerator hardware.","The same learned framework can be layered onto PID, Adaptive PID, or MRAC baselines, upgrading existing classical controllers rather than replacing them.","Under 40 Hz and 60 Hz flapping conditions, tracking error falls by 18.3% to 60.7%; at 20 Hz with MRAC the improvement is not significant because MRAC alone already performs well.","Removing tensor operations from edge inference cuts per-step execution time by more than a factor of three, making I/O the dominant cost.","The Lipschitz and finite-time Lyapunov conditions imply that, during training, worst-case tracking error stays bounded on every two-step interleaved cycle."],"supporting_citations":[{"why":"Supplies the CRL algorithm, the subsystem models of the tandem-wing platform, and the gradient-domain Laplace Q-estimator that CRL2RT extends.","marker":"[21]"},{"why":"Validates the realism of the semi-physical simulation used for all training and experimental results.","marker":"[62]"},{"why":"Defines the 2000 Hz high-frequency control target and the PID controller baseline for flapping-wing platforms.","marker":"[8]"},{"why":"Introduces adaptive policy learning with a classical default policy, the baseline CRL2RT improves upon by adding online updates and safety.","marker":"[60]"},{"why":"Provides the deterministic actor-critic and Sarsa update rules forming the learning core.","marker":"[68]"},{"why":"The Bellman equation and Sarsa foundations used to derive critic updates and reward-based Q estimates.","marker":"[69]"},{"why":"States the finite-time Lyapunov convergence condition the Rule-Based Policy Composer is designed to satisfy.","marker":"[73]"},{"why":"Provides the array-programming basis for the edge-inference matrix operations that remove tensorization overhead.","marker":"[79]"}],"fun_headline_variants":["Dragonfly-inspired RL hits 2534 Hz on standard CPUs","Hybrid RL boosts tracking 18-61%","RL on flapping wings reaches 2534 Hz","Real-time RL: 2500 Hz without sacrificing updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All measured frequencies and tracking gains come from a semi-physical simulation whose eight-degree-of-freedom tandem-wing model and realism validation are taken from earlier work cited in the paper; if that model does not reproduce the real unsteady aerodynamic loads on the actual platform, the numbers will not transfer to hardware.","fun_headline_variants_meta":{"raw":{"variants":["Dragonfly-inspired RL hits 2534 Hz on standard CPUs","Hybrid RL boosts tracking 18-61%","RL on flapping wings reaches 2534 Hz","Real-time RL: 2500 Hz without sacrificing updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2396,"prompt_tokens":920,"completion_tokens":1476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1421}},"tokens_in":536,"tokens_out":1476,"duration_ms":13546,"temperature":1.0,"reasoning_tokens":1421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:21:45.309114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained CRL2RT policy on the physical DDTWEP with its six-component balance under the same 40 Hz and 60 Hz flapping commands and compare tracking error with the semi-physical simulation. If the 18.3% to 60.7% improvement over the classical controllers does not appear, or if the slowest control frequency drops below 2000 Hz once weight updates and data transmission are included, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"Development of Minimal Biorobotic Stealth Distance and Its Application in the Design of Direct-Drive Dragonfly-Inspired Aircraft","cited_arxiv_id":"2410.15559","evidence_quote":"Validates the realism of the semi-physical simulation used for all training and experimental results."},{"cited_title":"HiFly -Dragon: A Dragonfly Inspired Flapping Flying Robot with Modified, Resonant, Direct-Driven Flapping Mechanisms,","cited_arxiv_id":null,"evidence_quote":"Defines the 2000 Hz high-frequency control target and the PID controller baseline for flapping-wing platforms."},{"cited_title":"Adaptive policy learning for data -driven powertrain control with eco-driving,","cited_arxiv_id":null,"evidence_quote":"Introduces adaptive policy learning with a classical default policy, the baseline CRL2RT improves upon by adding online updates and safety."},{"cited_title":"Deterministic policy gradient algorithms,","cited_arxiv_id":null,"evidence_quote":"Provides the deterministic actor-critic and Sarsa update rules forming the learning core."},{"cited_title":"S., and Barto, A","cited_arxiv_id":null,"evidence_quote":"The Bellman equation and Sarsa foundations used to derive critic updates and reward-based Q estimates."}],"review_version":1}