{"id":"7dd058d9-e92b-4fff-a9d8-37a779526458","arxiv_id":"2504.17857","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"An RL policy trained in Isaac Sim and tuned with Wasserstein/MMD distributional gap optimization runs on Spot at over 5.2 m/s, tripling the stock controller's speed.","lead":"This paper reports the first open end-to-end reinforcement learning controller for Boston Dynamics Spot, trained in NVIDIA Isaac Sim and deployed on the real robot. It uses distributional statistics (Wasserstein distance and MMD) to tune simulator friction and torque limits, reaching over 5.2 m/s, more than three times Spot's stock controller speed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Optimized actuator parameters may absorb unmodeled sensor noise: Section IV.A excludes noise from simulation while hardware rollouts include it, so the distributional score does not isolate physical sim-to-real mismatch.","rationale":"The reader's weakest_assumption identifies essentially the same concern: distributional matching without sensor noise in simulation may not separate sensor noise from true actuator parameters, so the optimized values could be policy- and command-specific. I agree and sharpen the point by grounding it in an internal inconsistency: Section III.A introduces a noise model for training, but Section IV.A explicitly excludes sensor noise from the simulated data used in the scoring objective. Since hardware data contains state-estimation noise, the Wasserstein/MMD score is biased by that noise component, and CMA-ES has no way to attribute it to the correct source. This is a real soft spot in the causal claim that the optimized parameters are the right physical parameters. However, the paper's primary empirical contribution -- a deployed RL policy on Spot reaching 5.2 m/s with robustness demonstrations -- is supported by video and qualitative plots, and the concern does not invalidate that demonstration. It does warrant the conditional verdict: the method's generalizability and the identifiability of its parameters remain unproven. I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":11207,"tokens_out":2836,"duration_ms":33858,"concrete_test":"Re-run the Section III.B distributional scoring with simulated rollouts corrupted by the same Gaussian observation-noise model described in Section III.A, then repeat the CMA-ES optimization from the same initialization. If the optimized friction and torque-speed parameters differ substantially from Table I, or if the relative ranking of candidate parameter sets changes, the current optimization is confounded by sensor noise and the Table I values are not identifiable physical parameters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the Wasserstein/MMD score, computed on joint positions, joint velocities, and policy actions, identifies the eight friction and torque-speed parameters as the causes of the sim-to-real gap. Section IV.A states, \"Importantly, simulated data does not include sensor noise,\" while the hardware rollouts used as the optimization target necessarily contain state-estimation noise, and Section III.A describes a Gaussian observation-noise model for training. The distributional distance therefore includes a noise-induced component that no choice of actuator parameters can physically eliminate. CMA-ES can reduce the score by shifting the simulated joint distributions to absorb this noise discrepancy, yielding friction and torque-speed values that are partly artifacts of the sensor-noise mismatch rather than physical properties of Spot. The paper reports one optimization iteration, one retraining, and a single speed comparison (3.8 to 5.2 m/s) without error bars or repeated hardware trials, so the speed improvement cannot be cleanly attributed to the optimized parameters versus reward fine-tuning, retraining stochasticity, or environmental variation. This concern does not refute the empirical demonstration that a high-speed policy was deployed, but it weakens the central methodological claim that distributional matching identifies the correct simulator parameters in a way that transfers to new policies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a workflow for sim-to-real transfer of a learned locomotion policy on a Boston Dynamics Spot robot. The authors quantify the sim-to-real gap by comparing distributions of joint positions, joint velocities, and policy actions between hardware rollouts and simulation rollouts, using Wasserstein distance and Maximum Mean Discrepancy. These measures serve as the objective for CMA-ES, which optimizes eight actuator-related parameters: hip and knee friction coefficients, torque limits, speed limits, and torque-speed intersection points. After optimization, a PPO policy is retrained in NVIDIA Isaac Lab and deployed through the Spot RL Researcher Development Kit. The paper claims the deployed policy achieves over 5.2 m/s forward speed, more than triple Spot's default controller maximum, and shows robustness to slippery surfaces and a flight phase gait. The authors state that training code is publicly available and that this is the first end-to-end RL policy deployed on Spot with open training code.","tokens_in":11450,"tokens_out":5519,"duration_ms":56620,"significance":"If the quantitative results hold, this would be a notable empirical contribution: a publicly documented end-to-end RL pipeline on a commercial quadruped, with a concrete method for tuning simulation parameters from distributional data. The idea of using Wasserstein/MMD scores as a CMA-ES objective is a reasonable and potentially reusable approach for gray-box actuator identification, and the hardware demonstrations of a flying trot at 5.2 m/s would be impressive. However, the evidence supporting the central claims is currently incomplete: the headline speed results lack trial counts and error bars, the reward weights are omitted, the code link is missing, and the scoring procedure excludes sensor noise from simulation while hardware data necessarily includes it. These gaps make the paper's central methodological and performance claims plausible but not yet fully supported.","major_comments":[{"comment":"Section IV.C reports that the policy 'reached sustained running speeds of 5.2m/s in a flying trot gait' and compares this to the baseline policy's '3.7m/s sustained maximum,' but the manuscript gives no number of hardware trials, no run-to-run variability, and no measurement protocol for either number. The headline claim of a more-than-tripled maximum speed is load-bearing and currently rests on single-point estimates. Please report repeated trials with mean and standard deviation (or all individual runs) for both the baseline and the CMA-ES-optimized policy under identical command protocols, including how the maximum speed was determined from each trial.","section":"IV.C"},{"comment":"Section IV.A states, 'Importantly, simulated data does not include sensor noise,' while the hardware rollouts used as the optimization target are necessarily corrupted by state-estimation noise. Section III.A describes a Gaussian observation-noise model, but that model is not applied to the simulated data used in scoring. The distributional distance therefore contains a noise-induced component that no actuator parameter can physically eliminate, so CMA-ES may reduce the score by shifting simulated joint distributions to match the noise spread. The optimized friction and torque-speed values in Table I may thereby partly absorb sensor-noise mismatch rather than physical actuator effects. A concrete test is to corrupt the simulated observations with the Section III.A noise model before scoring, or to denoise the hardware estimates, and to report whether the optimized parameters change.","section":"IV.A"},{"comment":"Section III.D lists reward components but gives no weights, and Sections III.C and IV.A state that 'some fine-tuning of the reward functions may be required' and that 'only minor weight value adjustments were made.' Because the speed comparison is between a baseline policy and a policy retrained after both parameter optimization and reward modifications, the reported improvement from 3.8 m/s to 5.2 m/s cannot be cleanly attributed to the CMA-ES parameter search alone; it could also come from reward fine-tuning or from PPO stochasticity across random seeds. Please report the exact reward weights before and after the optimization, run multiple training seeds, and include an ablation that keeps rewards fixed while changing only the optimized simulation parameters.","section":"III.D / IV.A"},{"comment":"Section IV.B explicitly states that 'both policies struggle to produce negative torque during swing leg recirculation... due to power distribution limits not being included in our modeling.' This is an admitted unmodeled physical effect that appears in exactly the high-performance regime where the distributional score is computed. The eight optimized parameters cannot represent this mismatch, so the optimized values and the residual distributional gap may be dominated by the missing power-limit model rather than by the identified actuator parameters. Please either include a power-distribution-limit model in the simulator, or add an experiment that isolates whether the optimized parameters capture the observed hardware-simulation difference beyond this unmodeled effect.","section":"IV.B"}],"minor_comments":[{"comment":"The abstract and conclusion promise publicly available training code, but no repository URL appears anywhere in the manuscript; add a link, ideally in a footnote in Section I or III.","section":"Abstract / Conclusion"},{"comment":"The scoring procedure is described qualitatively, but the manuscript does not state which Wasserstein variant is used (1D, sliced, or joint product), which MMD kernel and bandwidth are chosen, how the two measures are combined into the reported weighted average, or how many simulated rollouts and samples are used per CMA-ES candidate. An appendix with these details would make the method reproducible.","section":"III.B"},{"comment":"The sentence 'Delaying action application the total delay is identical to delaying both the observations and actions' is grammatically incomplete, and the choice of a 5 ms buffer is not justified against the measured delay distribution in Fig. 2.","section":"III.A"},{"comment":"The reward components are listed by name but no weights are given; please include the full reward table, since the policy and its reward tuning are central to the performance comparisons.","section":"III.D"},{"comment":"The baseline maximum speed is reported as 3.8 m/s in Section IV.A and 3.7 m/s in Section IV.C; please reconcile these numbers and use one consistent value.","section":"IV.A vs IV.C"},{"comment":"Figure 2's axis labels and tick marks are unclear (the x-axis appears to show only 2-6), and the term 'Occurences' is misspelled; Figure 6 lacks units and trial-count annotations. Please improve caption clarity.","section":"Figures 2 and 6"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the hardware demonstration is potentially exciting, but the paper currently under-supports its headline claims. The lack of trial counts, error bars, reward weights, and a code link are fixable but should be addressed before publication. The 'first end-to-end RL policy on Spot' claim should be checked against Boston Dynamics' own RL announcement (reference [6]) and any concurrent public work; the manuscript does not provide enough context for me to verify priority. I do not see a fundamental flaw in the approach, but the evidential standard for a journal paper on a claimed first deployment needs to be higher."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper is worth reading and worth a serious referee. It reports the first end-to-end RL policy deployed on Spot with the low-level API, and it backs the claim with a coherent gray-box sim-to-real tuning loop: distributional measures (Wasserstein distance and MMD) over joint positions, velocities, and actions as a fitness for CMA-ES over a small set of actuator parameters. That combination is not revolutionary—each ingredient is known—but putting them together this cleanly for a commercial platform is genuinely useful, and the authors show their work.\n\nWhat the paper does well: the modeling section explains communication delay, torque limits, and observation noise with enough detail to replicate, and the hardware demos (flight-phase gait, slippery surface robustness, 5.2 m/s on a track) are consistent with the claims. The comparison in Fig. 6 between a baseline and the optimized policy is the most convincing evidence: the optimized policy's simulated knee actions and torques match hardware substantially better. That is a real result, and it suggests the optimization is doing something beyond reward tuning.\n\nSoft spots, in order of softness. First, the headline speed comparison is a single number without error bars or trial counts. We don't know if 5.2 m/s is a one-off or a repeatable maximum. Second, the scoring function mixes apples and oranges: simulation rollouts are noise-free while hardware rollouts include state-estimation noise, so CMA-ES can reduce the distributional distance by picking parameters that absorb the noise discrepancy rather than correcting physical mismatch. The authors acknowledge this (Section IV.A says \"simulated data does not include sensor noise\"), but they don't analyze how much of the optimized friction and torque-speed values might be noise artifacts. That weakens the claim that these are the \"right\" physical parameters, though it doesn't refute the empirical improvement. Third, \"minor weight value adjustments\" to the reward after retraining confound the attribution: part of the speed gain could come from the reward change, not the parameter optimization. Fourth, the paper says training code is public but no repository link appears in the manuscript; for a reproducibility-focused contribution that needs to be fixed.\n\nOverall: the central argument holds up as an engineering result. The method is transportable, the description is honest about limitations, and the first-Spot-RL claim is a real milestone for the community. For peer review, it deserves referee time, but I'd push for repeated trials, an analysis of the noise confound, and a working code link before acceptance.\n\nYes, bring it to reading group.","headline":"A credible engineering advance—first learned end-to-end policy on Spot—with real but addressable weaknesses in the quantitative evidence and the noise handling.","tokens_in":12019,"tokens_out":3442,"would_cite":true,"duration_ms":33225,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A distributional sim-to-real gap score, optimized with CMA-ES over eight actuator parameters, lets a fully learned policy drive Spot at over 5.2 m/s with a flight-phase gallop, slippery-ground robustness, and disturbance rejection.","keywords":["reinforcement learning","sim-to-real transfer","quadruped locomotion","distributional gap measure","Wasserstein distance","maximum mean discrepancy","CMA-ES","actuator parameter optimization"],"falsifier":"Train a second policy with a substantially different gait or reward budget using the same eight optimized parameters without retuning, then deploy it; if the distributional gap score shrinks but hardware speed and robustness do not improve, the optimization was overfitting to the original policy rather than recovering the true actuator model.","tokens_in":10969,"feed_emoji":"🐕","tokens_out":9664,"duration_ms":89646,"temperature":0.7,"pith_summary":"The paper sets out to show that a gray-box modeling loop can close the sim-to-real gap for Spot enough to run a fully learned end-to-end control policy at speeds and gaits the stock controller cannot reach. The authors score the difference between hardware and simulation by comparing distributions of joint positions, joint velocities, and policy actions with Wasserstein distance and maximum mean discrepancy, then use CMA-ES to tune eight actuator parameters (joint friction and torque-speed limits) that are hard to measure directly. A single retraining pass with the optimized model yields a policy that sustains 5.2 m/s, more than triple Spot's default 1.6 m/s limit, gallops with a flight phase, and stays upright on soap-wet plexiglass. If correct, this is the first end-to-end RL policy deployed on Spot with public training code, and it offers a reusable recipe for high-performance learned control on a commercial quadruped.","feed_headline":"Learned policy runs Spot at 5.2 m/s, more than triple its stock speed","feed_subtitle":"A distributional sim-to-real gap score and CMA-ES tuning produce an end-to-end policy that gallops and rejects slips.","key_machinery":"The load-bearing mechanism is a distributional gap score built from Wasserstein distance and maximum mean discrepancy, computed over scaled joint positions, joint velocities, and policy actions from hardware and simulated rollouts of the same command sequences. The score plays the role of a GAN-style evaluation: the simulated policy is the generator and the hardware data is the target, so the two rollouts do not need to start from matched states. CMA-ES treats that score as the objective and searches an eight-parameter gray-box actuator model (two friction coefficients plus a six-parameter torque-speed curve), and the optimized values are used to retrain the policy.","core_discovery":"The paper's central claim is that the sim-to-real gap for Spot is best treated as a distribution-matching problem rather than a trajectory-matching one. Hardware rollouts are collected under several scripted command sequences, the same commands are replayed in simulation, and the gap is scored as the Wasserstein and MMD distance between the scaled joint positions, joint velocities, and policy actions of the two data sets. CMA-ES then minimizes that score over eight parameters describing joint friction and torque-speed limits, and a single retraining pass with the optimized simulator produces policies that gallop with a flight phase, reject disturbances, and reach 5.2 m/s sustained speed — more than triple the stock controller's 1.6 m/s maximum and well above the roughly 3.7 m/s baseline policy trained without the optimization.","pith_inferences":["A stress test the paper does not run: keep the eight optimized parameters fixed while training a policy with a different gait or reward budget; if speed and robustness gains persist, the parameters are true actuator properties rather than compensations for one policy's failure modes.","The scoring features could be extended beyond joint positions, velocities, and actions to exteroceptive signals such as body height or contact forces, which may expose mismatches the current joint-space score cannot see.","The optimization loop couples system identification with policy retraining; a variant that optimizes parameters against a frozen policy and only then retrains would isolate how much of the gain comes from each step."],"forward_implications":["An end-to-end learned controller can push Spot to a sustained 5.2 m/s flying-trot gallop, a capability the stock controller does not offer.","The distributional scoring procedure makes simulator tuning possible without joint torque measurements or matched-state resets, relying only on onboard state and action logs.","One iteration of optimization plus retraining is enough for a substantial transfer gain, raising sustained speed from roughly 3.7 m/s to 5.2 m/s.","The released training and deployment code provides a publicly reproducible baseline for future low-level RL control on Spot."],"supporting_citations":[{"why":"Supplies the Wasserstein distance used to score the distributional gap between hardware and simulation.","marker":"[28]"},{"why":"Supplies the maximum mean discrepancy measure used as the second distributional score.","marker":"[29]"},{"why":"Supplies the CMA-ES optimizer that searches the eight actuator parameters.","marker":"[41]"},{"why":"Provides the Spot API documentation used for the torque-position limits in the actuator model.","marker":"[5]"},{"why":"Gives Spot's stock maximum forward velocity of 1.6 m/s, the baseline for the speed comparison.","marker":"[51]"},{"why":"The simulation and training environment used to implement the Spot model and parallel policy training.","marker":"[46]"},{"why":"The reinforcement learning implementation used for policy updates during training.","marker":"[48]"}],"fun_headline_variants":["Spot gallops at 5.2 m/s via distributional sim-to-real tuning","CMA-ES closes sim-to-real gap, giving Spot a flight phase","Wasserstein-based sim tuning lets Spot triple its speed","Distribution-matching sim tweaks push Spot to 5.2 m/s","Sim-to-real fix: Wasserstein and MMD unlock Spot's gallop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that matching the distribution of joint positions, joint velocities, and actions between simulation and hardware is enough to identify the true actuator mismatches, so the eight optimized parameters are the real cause of the gap rather than an overfit to one policy and its command sequences.","fun_headline_variants_meta":{"raw":{"variants":["Spot gallops at 5.2 m/s via distributional sim-to-real tuning","CMA-ES closes sim-to-real gap, giving Spot a flight phase","Wasserstein-based sim tuning lets Spot triple its speed","Distribution-matching sim tweaks push Spot to 5.2 m/s","Sim-to-real fix: Wasserstein and MMD unlock Spot's gallop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2915,"prompt_tokens":909,"completion_tokens":2006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1908}},"tokens_in":525,"tokens_out":2006,"duration_ms":13354,"temperature":1.0,"reasoning_tokens":1908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:30:35.878067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a second policy with a substantially different gait or reward budget using the same eight optimized parameters without retuning, then deploy it; if the distributional gap score shrinks but hardware speed and robustness do not improve, the optimization was overfitting to the original policy rather than recovering the true actuator model.","supporting_citations":[{"cited_title":"The wasserstein distance and approximation the- orems,","cited_arxiv_id":null,"evidence_quote":"Supplies the Wasserstein distance used to score the distributional gap between hardware and simulation."},{"cited_title":"Tutorial cma-es: evolution strategies and covariance matrix adaptation,","cited_arxiv_id":null,"evidence_quote":"Supplies the CMA-ES optimizer that searches the eight actuator parameters."},{"cited_title":"Boston Dynamics","cited_arxiv_id":null,"evidence_quote":"Provides the Spot API documentation used for the torque-position limits in the actuator model."},{"cited_title":"Boston Dynamics","cited_arxiv_id":null,"evidence_quote":"Gives Spot's stock maximum forward velocity of 1.6 m/s, the baseline for the speed comparison."},{"cited_title":"Orbit: A unified simulation framework for interactive robot learning environments,","cited_arxiv_id":null,"evidence_quote":"The simulation and training environment used to implement the Spot model and parallel policy training."},{"cited_title":"Robotic Systems Lab","cited_arxiv_id":null,"evidence_quote":"The reinforcement learning implementation used for policy updates during training."}],"review_version":1}