{"id":"dfd9e418-ba1d-46e7-bd3b-a385350dfde0","arxiv_id":"2505.07321","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A residual reinforcement learning controller trained entirely on a physical 1:10 race car beats a pursuit controller's lap time by up to 11.5% after about 20 minutes of on-track practice.","lead":"This paper trains a reinforcement learning controller directly on a small autonomous race car, with no simulation pre-training, and reports lap times up to 11.5% faster than a classical racing controller after about 20 minutes. The result suggests that on-board learning can match or beat classical control in high-speed robotics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'SotA' claim rests on in-house baselines that the paper itself shows to be conservative; without an external comparison, the 11.5% headline is not established.","rationale":"The reader's weakest assumption correctly identifies the SotA baseline as the most load-bearing premise. The paper's own Appendix J undermines the MPC baseline, calling it conservative and uncompetitive on high-grip tires; MAP is an in-house controller from the same research group. The claimed 11.5% improvement is computed against these baselines, not against an independent, verifiable state of the art. This matters because the paper's positioning, title, abstract, and Contribution 2 all assert that on-board RL surpasses classical SotA; if the baselines are under-tuned, the headline conclusion is overstated even though the internal comparisons remain valid. The alternative concerns (20 clean-lap selection, three seeds, no significance testing) are real but secondary: the clean-lap protocol is disclosed with boundary-violation counts, the three-seed improvements are consistent across seeds, and the paper does not claim statistical significance. A parameter sweep of the MAP baseline directly tests whether the gap is an artifact of conservative tuning. If the sweep finds a faster MAP, the verdict should remain conditional but with the SotA language removed from the contribution. If the sweep confirms MAP is near its tuned limit, the central claim stands. In either case, the reader's CONDITIONAL verdict is the appropriate recommendation, so no adjustment is needed.","tokens_in":21541,"tokens_out":15843,"duration_ms":158242,"concrete_test":"Request from the authors the C-track map, MAP's speed-profile lookup table, and raw deployment logs. Run a controlled sweep of the MAP baseline on the same physical setup: scale the speed-profile lookup table by 0.9, 1.0, 1.05, 1.1, and 1.15, and vary look-ahead distance by +/-20%, reporting t_min and t_mean over 20 clean laps for each configuration. If any configuration within a plausibly tuned envelope (e.g., speed scaling <= 10%) achieves t_min <= 9.6 s, the residual improvement over a well-tuned classical controller is below the claimed 11.5% and the 'SotA' comparison is not substantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two legs: the residual policy improves its own base controller, and it beats the state of the art. The first leg is well supported by the within-paper comparisons. The second leg rests entirely on in-house baselines: MAP is from the authors' own group, and the MPC baseline is acknowledged in Appendix J to be conservative ('induced unnecessary conservativity') and uncompetitive on high-grip tires, performing worse than MAP on both Turbo-tire scenarios. No quantitative comparison to independent prior on-board RL systems (RLPP, FastRLAP) or to external F1TENTH results is provided, despite Contribution 2 asserting the method yields 'the fastest F1TENTH algorithm when compared to previous best results.' If the MAP speed-profile lookup table is conservative, the 'up to 11.5% vs SotA' improvement largely measures the gap to an under-tuned reference rather than an advance over the field. The residual-improvement claim survives, but the SotA framing is unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an on-board reinforcement learning framework for a 1:10 F1TENTH autonomous racing car. The method is a residual Soft Actor-Critic policy that augments a classical base controller (PP, MAP, or FTG), trained entirely on the physical vehicle without simulation pre-training. Additional components include multi-step TD learning, an asynchronous acting/training pipeline, a curriculum safety filter, and a Heuristic Delayed Reward Adjustment (HDRA) mechanism. Experiments on a real car across two tracks and two tire types show that the residual policy consistently improves its base controller, with the RL MAP configuration reducing the minimum lap time by up to 11.5% relative to the MAP baseline and achieving a 9.26 s minimum lap on the C-track with Turbo tires. An end-to-end variant, trained for 82 minutes, also surpasses the MAP baseline. A simulator ablation studies the contribution of the asynchronous pipeline, multi-step TD, and HDRA. The paper claims that this is the fastest F1TENTH algorithm reported to date and that on-board RL can surpass classical state-of-the-art racing controllers without sim-to-real transfer.","tokens_in":21677,"tokens_out":3660,"duration_ms":35280,"significance":"If the empirical claims hold, this is a meaningful advance for physical robot learning in autonomous racing: it demonstrates that a policy trained directly on a real vehicle, with no simulation pre-training and no external GPU, can consistently beat the classical controller it is built on, and can do so within a single battery charge. The paper provides real physical experiments on multiple tracks, two tire compounds, and multiple random seeds, with per-seed tables, training curves, and speed-profile visualizations. The within-paper comparison to the base controllers is convincing and is the core strength of the paper. The claim to surpass the state of the art, however, rests on in-house implementations of the comparison controllers, and the evidence for that external-facing claim is currently incomplete. The simulator ablation is a useful addition, though it is limited to one setting and does not establish a statistically significant benefit for HDRA.","major_comments":[{"comment":"The state-of-the-art comparison is not established. The MAP baseline is from the authors' own group (Becker et al. 2023), the MPC baseline is an in-house implementation of Vázquez et al. (2020) that Appendix J admits is conservative and 'induced unnecessary conservativity' on high-grip tires, and no quantitative comparison is provided to independent on-board RL systems such as RLPP (Ghignone et al. 2025) or FastRLAP (Stachowicz et al. 2023). The abstract and Contribution 2 claim the fastest F1TENTH algorithm, but the 'up to 11.5% vs SotA' headline measures the gap to these in-house baselines. Please either provide external comparisons on comparable tracks or carefully rephrase the SotA claims to refer to the provided baseline controllers.","section":"Section 4.2, Table 1, Contribution 2"},{"comment":"The statistical evidence for repeatability and robustness is thin. Only three random seeds are used, no significance tests or confidence intervals are reported, and the lap-time statistics in Table 1 are computed only over the 20 violation-free deployment laps, while boundary violations during deployment (up to 2 on C-Track Turbo, 1 on C-Track TPU, and 5 on Y-Track) are excluded from those statistics. The claim that the method is robust across seeds and environments would be substantially strengthened by reporting per-seed pairwise comparisons, effect sizes, or confidence intervals, and by stating the sensitivity of the reported lap times to the violation-exclusion rule.","section":"Section 5.1, Table 1"},{"comment":"The zero-shot generalization claim is overstated. The text states that zero-shot results were mixed and that Seeds 2 and 3 experienced significant instability, and Table 4 shows that Seed 2 required 14 boundary violations and Seed 3 required 7 before completing 20 violation-free laps, with 22 violations in the combined deployment. The conclusion that 'a single policy can be tuned to drive consistently on a different track' is supported by the few-shot results, not by the zero-shot results. The section should be reworded to distinguish the two settings clearly and to present zero-shot transfer as a limitation rather than as demonstrated capability.","section":"Section 5.2, Table 4"},{"comment":"The ablation does not support HDRA as a core algorithmic improvement. Table 5 shows that HDRA+TD3 Async achieves an average lap time of 9.53 s versus 9.68 s for TD3 Async, and the text itself says this is not a statistically relevant improvement, with the effect appearing mainly as reduced lap-time spread. This is acceptable for an engineering paper, but the contribution list and abstract should present HDRA as a variance-reduction heuristic rather than as one of the main reasons for the performance gain, or the authors should provide statistical evidence for a benefit.","section":"Section 5.3, Figure 5, Table 5"}],"minor_comments":[{"comment":"The abstract states 'up to an 11.5% reduction ... with only 20 min of training,' but the 11.5% improvement appears to refer to the Y-Track from-scratch comparison in Table 1, which used 21–25 minutes of training, while the 20-minute C-Track Turbo result is about 11.05%. Please make the training-time and scenario attribution of the headline number explicit.","section":"Abstract and Section 5.1"},{"comment":"The legend for the RL MAP panel repeats 'RL MAP' for the training curve and for what appears to be the deployed policy or baseline reference; please clarify the legend labels and use distinct names for the training and deployment curves.","section":"Figure 2"},{"comment":"The paper relies on several hand-chosen hyperparameters (reward multiplier lambda, penalty p, HDRA steps N, safety filter threshold range, residual action limits, curriculum speed increment) without sensitivity analysis. A short discussion of which parameters are critical and how sensitive performance is to them would improve reproducibility.","section":"Table 6 and Section 3"},{"comment":"The pseudocode in Algorithm 1 has an indentation ambiguity in the condition 'if terminal_state is True and replay_buffer.reward[current_step] = 0', and the relation between the pseudocode variable 'adjustment_steps N' and Eq. (1) should be stated explicitly. Also, there is a minor typo in the abstract: '20 minof training' should be '20 min of training'.","section":"Appendix G and Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The central residual-improvement result is credible and well supported by physical experiments. The main weakness is external benchmarking: the SotA claim rests on the authors' own baselines, and the related-work section already lists RLPP and FastRLAP, which are the natural external comparators. I would encourage the editor to request either a comparison with those prior systems on a shared track or a substantial re-framing of the claims. The paper would also benefit from a data-availability statement for the raw per-lap times."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper shows a residual SAC policy trained entirely on a physical F1TENTH car, with no sim-to-real pre-training, consistently beating the classical controller it wraps. That is a real result, and it is the first time I have seen it done on a 1:10 car at these speeds. The RL MAP agent's 9.26 s vs 10.41 s minimum lap time improvement over the MAP baseline is not a fluke: it repeats across three seeds, two tracks, and two tire types, and the speed-profile plots make the mechanism believable. The E2E controller catching up after 82 minutes is a nice secondary point.\n\nWhat the paper does well: it is unusually complete for a robotics paper. Real hardware experiments, multiple tracks and tires, a real ablation in simulation, and honest reporting of training curves and violation counts. The residual structure is simple but effective, and the asynchronous training plus multi-step TD choices are sensible and supported by the ablation. The paper also does not oversell its zero-shot generalization; it reports mixed results and pivots to few-shot, which is the right scientific move.\n\nThe soft spots are in the framing, not the core result. The \"up to 11.5% vs SotA\" headline rests on baselines that are in-house or acknowledged to be conservative. MAP is from the authors' own group, and the MPC is admitted in Appendix J to have \"induced unnecessary conservativity\" on high-grip tires. There is no quantitative comparison to independent prior RL racing systems like RLPP or FastRLAP, so Contribution 2's claim about being the fastest F1TENTH algorithm is not established. The lap-time statistics also only count violation-free laps, which can exclude the controller's worst behavior, and three seeds with no significance testing is thin for the repeatability claims. No code or data are released. None of this falsifies the central residual-improvement result, but it does mean the headline magnitude is a best-case number against a weak reference.\n\nI would send this to peer review. The empirical core is solid and the field needs more on-board learning results like this. The revision should fix the SotA framing, add statistical care, and ideally release data or code. A serious referee will get value from engaging with it, and I would cite it if I were working on on-policy learning for scaled race cars.","headline":"Real on-board residual RL that beats its own classical baselines; the 'SotA' headline is softer than it looks because the baselines are in-house.","tokens_in":22309,"tokens_out":1007,"would_cite":true,"duration_ms":11581,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning race car trained only on the physical track beat its classical state-of-the-art baseline by up to 11.5% in lap time.","keywords":["autonomous racing","reinforcement learning","on-board learning","residual policy","soft actor-critic","F1TENTH","sim-to-real","sample efficiency"],"falsifier":"Run the same training pipeline on the same physical track against an independently implemented, publicly benchmarked MAP or MPC baseline that has itself been tuned to the track, and compare minimum lap times over 20 laps; if the classical controller matches or beats the trained RL policy, or the improvement disappears against an equally strong baseline, the central claim would be refuted. Replicating the Y-track zero-shot experiment and finding that no seed reliably completes 20 laps would likewise weaken the generalization claim.","tokens_in":21276,"feed_emoji":"🏎️","tokens_out":8179,"duration_ms":63831,"temperature":0.7,"pith_summary":"This paper claims that a reinforcement-learning agent can be trained entirely on a physical race car, with no simulator, and still beat carefully tuned classical controllers. On the F1TENTH 1:10 platform, a residual Soft Actor-Critic policy that adds small corrections on top of a classical Model- and Acceleration-based Pursuit controller reached a 9.26-second minimum lap time versus the baseline's 10.41 seconds on the C-track with Turbo tires, an 11.5% improvement, after about 20 minutes of on-track training. The same architecture improved every classical baseline it was paired with (PP, MAP, FTG), and an end-to-end version trained without any base controller also beat the classical baseline after 82 minutes. The significance is that it removes the simulation-to-reality transfer step that often limits RL in real-time autonomous systems, learning directly from the platform's own dynamics.","feed_headline":"On-board RL shaves 11% off race car lap times","feed_subtitle":"A residual policy trained only on a physical F1TENTH car overtakes classical controllers in 20 minutes.","key_machinery":"The load-bearing mechanism is the residual policy structure: the RL agent learns only a small additive correction $u_{RL}$ to the base controller's command $u_{base}$, with steering corrections bounded to $\\pm 0.15$ rad and speed corrections in $[-0.5, 2]$ m/s, so an untrained agent starts near the base behavior and improves it. Three supporting mechanisms carry the fast on-board training: asynchronous acting/training (the acting policy is synced once per second while training runs at 32 Hz), multi-step TD learning over three steps to handle delayed rewards, and HDRA, which linearly redistributes the crash penalty $p$ over the $N$ previous transitions to discourage the whole sequence leading to failure. A curriculum safety filter with an adaptive heading threshold gates unsafe commands and triggers a recovery state machine. Together these let a physical car collect and learn from its own experience at racing speed without simulation.","core_discovery":"The discovery is that a deliberately small residual policy can learn the delta between what a classical controller does and what is actually fastest, and that this delta is learnable on-board and in real time. Rather than learning the full control mapping, the agent outputs bounded corrections to steering and velocity that are added to the base controller's commands; the base controller keeps the car safe while the residual policy explores faster lines. With Soft Actor-Critic as the learner, the paper adds three mechanisms: multi-step temporal-difference updates to spread delayed reward signal, an asynchronous acting/training pipeline that decouples data collection at 10 Hz from policy updates at 32 Hz, and a Heuristic Delayed Reward Adjustment that retroactively amplifies penalties on the transitions leading to a crash. The result is consistent improvement: across C-track and Y-track, Turbo and TPU tires, and three random seeds, residual RL MAP improved minimum lap times by 5.4% to 11.5% and mean lap times by 5.9% to 9.0% over its baseline, with speeds above 6.5 m/s.","pith_inferences":["If this recipe generalizes beyond F1TENTH, the same residual-plus-safety-filter structure should work for drone racing or legged locomotion: start from any stabilizing base policy, train the residual directly on the platform, and let the filter act as a curriculum; the paper does not test this, but nothing in the architecture is platform-specific.","The ablation suggests the asynchronous pipeline and multi-step TD are the main drivers of convergence speed, while HDRA mostly reduces variance across seeds, so a user with limited time might drop HDRA without losing much average lap-time performance.","The zero-shot transfer results (mixed, with two of the three seeds unstable) imply that the practical recipe is few-shot fine-tuning rather than direct transfer, and an independent comparison against an externally tuned classical racing controller is the natural next test of the 'state-of-the-art' claim."],"forward_implications":["If the result holds, on-board RL can replace sim-to-real transfer for agile racing robots: the learned controller outperforms the classical baseline it is attached to, so classical controllers can be kept as safety nets while RL refines them in real time.","The asynchronous pipeline means a resource-constrained robot without a GPU can train on-board at racing speeds, so the approach should transfer to other real-time platforms with limited compute.","The residual structure's consistency across the PP, MAP, and FTG baselines suggests the method is controller-agnostic: any classical controller, even a simple one, can be upgraded by the same learned residual.","The end-to-end result, reaching a 9.65-second minimum lap time after 82 minutes with no base controller, shows the same framework can eventually learn racing from scratch, though more slowly."],"supporting_citations":[{"why":"Defines the MAP baseline controller that the residual RL policy is trained on top of and that is the main classical state-of-the-art opponent.","marker":"Becker et al., 2023"},{"why":"Supplies the MPC formulation used as a reference performance baseline and the cost structure for the MPC experiments.","marker":"Vázquez et al., 2020"},{"why":"Defines the Soft Actor-Critic algorithm that the on-board training loop is built on.","marker":"Haarnoja et al., 2018"},{"why":"Provides the asynchronous acting/training architecture that decouples data collection from policy updates on physical robots.","marker":"Yuan & Mahmood, 2022"},{"why":"Introduces residual reinforcement learning, the policy structure that lets an untrained agent start near base behavior and refine it.","marker":"Johannink et al., 2019"},{"why":"Shows residual policy learning works for autonomous racing in simulation, the direct precedent the paper extends to real-world training.","marker":"Trumpp et al., 2023"},{"why":"Defines the F1TENTH platform and evaluation environment used for all physical experiments.","marker":"O'Kelly et al., 2020"},{"why":"Previous on-vehicle RL racing system whose performance this work claims to surpass; supplies the on-robot learning context.","marker":"Stachowicz et al., 2023"},{"why":"Supports the use of multi-step temporal-difference learning for training stability.","marker":"Barth-Maron et al., 2018"}],"fun_headline_variants":["On-board residual RL cuts race lap times by up to 11.5%","20-minute on-board RL beats classical racing controllers","No simulation: on-board RL learns faster laps in 20 min","Residual RL learns the racing delta in just 20 minutes","On-board RL: up to 11.5% faster laps with 20-min training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim hinges on the baselines labeled state-of-the-art (MAP and MPC) being strong, fairly tuned representatives of classical racing control; if they are under-tuned or conservative, the measured lap-time win over them overstates the advantage over genuinely best classical controllers.","fun_headline_variants_meta":{"raw":{"variants":["On-board residual RL cuts race lap times by up to 11.5%","20-minute on-board RL beats classical racing controllers","No simulation: on-board RL learns faster laps in 20 min","Residual RL learns the racing delta in just 20 minutes","On-board RL: up to 11.5% faster laps with 20-min training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3171,"prompt_tokens":1016,"completion_tokens":2155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2061}},"tokens_in":632,"tokens_out":2155,"duration_ms":15572,"temperature":1.0,"reasoning_tokens":2061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:19:49.922821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training pipeline on the same physical track against an independently implemented, publicly benchmarked MAP or MPC baseline that has itself been tuned to the track, and compare minimum lap times over 20 laps; if the classical controller matches or beats the trained RL policy, or the improvement disappears against an equally strong baseline, the central claim would be refuted. Replicating the Y-track zero-shot experiment and finding that no seed reliably completes 20 laps would likewise weaken the generalization claim.","supporting_citations":[{"cited_title":"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor","cited_arxiv_id":null,"evidence_quote":"Defines the Soft Actor-Critic algorithm that the on-board training loop is built on."},{"cited_title":"Residual reinforcement learning for robot control","cited_arxiv_id":null,"evidence_quote":"Introduces residual reinforcement learning, the policy structure that lets an untrained agent start near base behavior and refine it."},{"cited_title":"FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing","cited_arxiv_id":null,"evidence_quote":"Previous on-vehicle RL racing system whose performance this work claims to surpass; supplies the on-robot learning context."},{"cited_title":"Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap","cited_arxiv_id":null,"evidence_quote":"Supports the use of multi-step temporal-difference learning for training stability."}],"review_version":1}