{"id":"10098832-0a6a-4aac-95c6-c57d2e3286db","arxiv_id":"2412.11253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-layer MLP with recursive skip-step sub-goal planning can reach state-of-the-art offline RL scores on long-horizon D4RL tasks, challenging the need for large expressive models.","lead":"This paper proposes an offline reinforcement learning method that uses very small neural networks to plan long-horizon tasks by recursively predicting future sub-goals instead of modeling every step. The method reportedly matches or beats much larger transformer and diffusion models on several D4RL benchmarks while training in minutes and acting in under a millisecond.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closed-loop distribution shift: RSP trains dynamics on ground-truth subgoals but evaluates on predicted subgoals; the paper's own RMSE analysis assumes an oracle policy, so the claimed dynamics consistency is not validated under recursive rollouts.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing risk: the recursive planning loop is trained on ground-truth subgoals but deployed on predicted subgoals. This is not a minor implementation detail; it is the mechanism that supposedly lets shallow MLPs avoid compounding error. The paper's own Discussion and Ablation sections concede that the RMSE plots assume an oracle policy and that recursive error accumulation can occur, which flags the limitation without resolving it. If the closed-loop test shows large degradation from mode (i) to mode (iii), the headline claim that lightweight models enjoy 'accurate dynamics consistency' would be overstated, and the strong AntMaze/Kitchen results would rest on the policy's tolerance to OOD conditioning rather than on a validated dynamics model. If the test passes, the central claim is materially strengthened. I therefore keep the reader's CONDITIONAL verdict unchanged, but the closed-loop self-training comparison should be a required condition for acceptance. The paper otherwise has real strengths: the efficiency numbers are concrete, the ablation over k and N is informative, and the AntMaze-Ultra results, if reproducible, would be a notable empirical advance.","tokens_in":14846,"tokens_out":6954,"duration_ms":64942,"concrete_test":"Run AntMaze-Ultra and Kitchen-Mixed with three evaluation modes: (i) ground-truth subgoals from the dataset, (ii) one-shot open-loop predicted subgoals, and (iii) recursive self-feeding as in Eq. 6. Compare normalized scores and closed-loop RMSE at each recursion level. Then retrain the dynamics models with a DAgger-style loop that appends model rollouts to the training set; if performance in mode (iii) improves materially or approaches mode (i), the current training objective is missing the test-time distribution and the paper's mechanistic claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the recursive dynamics models learned in Eq. 5 remain accurate when their outputs are fed back through Eq. 6 at test time. During training, each level's sub-goal input is a ground-truth state from the relabeled dataset; during evaluation, the conditioning set \\hat\\kappa^{(N)} consists entirely of model-generated sub-goals. The paper never trains on predicted sub-goals or otherwise closes this loop. Section 'Discussions' explicitly admits that the RMSE analysis in Fig. 3 'assumes that an oracle policy model perfectly follow the skip-step state predictions', and the Ablation section notes that recursive error accumulation can counterbalance gains. These admissions point directly at the untested assumption: if \\hat{s}_{t+K} drifts off the training manifold, the low-level policy \\pi(a_t|\\hat\\kappa^{(N)}) receives out-of-distribution conditioning, and the long-horizon advantage would shrink. The claimed 'accurate dynamics consistency' is therefore established only for ground-truth-conditioned rollouts, not for the actual recursive operating regime.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Recursive Skip-Step Planning (RSP), a goal-conditioned supervised learning method for offline RL that uses shallow 2-layer MLPs for recursive coarse-grained sub-goal prediction and a goal-conditioned policy. The method relabels transitions with skip-step sub-goal ground truths, trains N dynamics models to predict sub-goals at increasing horizons, and at test time feeds predicted sub-goals recursively into the policy. The authors report state-of-the-art results on D4RL AntMaze and Kitchen, with training time around 180 seconds and inference under 1 ms, and claim that large expressive models are not necessary for long-horizon offline RL.","tokens_in":15027,"tokens_out":4510,"duration_ms":37817,"significance":"If the empirical claims hold, RSP is a significant result: it would show that a simple recursive coarse-grained planning scheme with MLPs can outperform Transformer- and diffusion-based sequence models on long-horizon tasks at a fraction of the compute. The paper has notable strengths: it reports 10-seed means with standard deviations, includes ablations on recursion depth and skip-step horizon, measures training/inference time in isolation, and compares against a wide set of baselines. The main weakness is that the recursive operating regime is not evaluated in closed loop, and several reproducibility details are missing or inconsistent; these issues currently prevent full confidence in the central claim.","major_comments":[{"comment":"The central technical claim requires that dynamics models trained on ground-truth relabeled sub-goals (Eq. 5) remain accurate when their outputs are fed back through Eq. 6 at test time. The paper never trains on predicted sub-goals, and the Discussions section explicitly states that the RMSE analysis in Fig. 3 'assumes that an oracle policy model perfectly follow the skip-step state predictions.' Consequently, the claimed 'accurate dynamics consistency' is validated only for ground-truth-conditioned rollouts, not for the actual recursive operating regime. This is a load-bearing gap; please add closed-loop rollout evaluation (e.g., feeding predicted sub-goals to the policy and measuring task success) and/or a DAgger-style training correction that conditions on predicted sub-goals.","section":"Recursive Skip-Step Planning (Eqs. 5-6); Discussions"},{"comment":"The paper is internally inconsistent about the main hyperparameters. Discussions and Additional Experimental Details state k=32 and N=1 are used across all tasks, but the Training Cost and Inference Latency section reports 'RSP with horizon k = 8 and recursion depth N = 1, 4' and Fig. 1 and its caption show RSP (N=1) and RSP (N=4). Please reconcile these values and report the exact configuration for each table row, since the claimed 180-second training time depends on a specific N and k.","section":"Training Cost and Inference Latency; Discussions; Additional Experimental Details"},{"comment":"Table 1 reports scores for HIQL in AntMaze and Kitchen, but HIQL is never introduced in the Baselines paragraph, no citation is given for the reproduced numbers, and it is not clear whether those numbers are from the original paper or self-reproduced. Since HIQL is a strong baseline and appears in the tasks where RSP claims large gains, the source and protocol must be specified.","section":"Baselines; Table 1"}],"minor_comments":[{"comment":"The claim of 'new SOTA performances on the D4RL benchmark' is too broad: Table 1 shows RSP is not SOTA on MuJoCo average (76.4 vs TT's 78.9) and ties TAP on Adroit average (67.0). Please restrict the SOTA claim to long-horizon AntMaze and Kitchen tasks.","section":"Abstract and Conclusion"},{"comment":"The code availability statement 'Code is available at RSP JAX' is not a usable link; please provide a repository URL or remove the claim.","section":"Abstract (code availability)"},{"comment":"Equation (4) defines κ^(n) but Eq. (5) conditions on κ^(n-1) to predict s_{K^(n-1)}; the indices are inconsistent with the textual description of the recursion and should be clarified.","section":"Recursive Skip-Step Planning, Eq. (4)-(5)"},{"comment":"Figure 4(a) is titled with four task types but has no legend distinguishing the curves; please add a legend or label the curves directly.","section":"Ablation Studies, Fig. 4(a)"},{"comment":"The footnote mentions that IQL scores on AntMaze-v2 are taken from DOGE, but no corresponding citation of DOGE appears in the reference list; please add the reference.","section":"Table 1 footnote"}],"recommendation":"major_revision","confidential_remarks":"The closed-loop evaluation gap and the hyperparameter inconsistency are the main blockers. The HIQL baseline omission may be a simple oversight but should be checked carefully. The paper's efficiency claims are interesting, but the SOTA claim needs to be scoped more carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2412.11253. The headline: RSP is a genuinely new and simple algorithm that deserves to be taken seriously, but the paper's explanation of why it works is not fully validated, and there are a few reproducibility issues that need to be fixed before I'd fully trust the numbers.\n\nWhat's actually new: the recursive skip-step sub-goal prediction scheme. Each level predicts a coarser sub-goal (skipping exponentially more steps), and the lower-level dynamics are conditioned on the higher-level predictions. The final policy is conditioned on all predicted sub-goals. That's a real algorithmic contribution, distinct from GCSL, RvS, DT, TT, TAP, and HIQL. The empirical results are strong: on AntMaze-Ultra, where most baselines score 20-50, RSP gets ~80, with 2-layer MLPs, 180s training, and sub-millisecond inference. Ten seeds with standard deviations is solid practice, and the ablations on recursion depth and horizon are informative.\n\nThe soft spots are real but not fatal. The biggest is the closed-loop distribution shift. The dynamics models are trained on ground-truth relabeled subgoals (Eq. 5) but evaluated on their own predicted subgoals (Eq. 6). The paper's RMSE analysis in Fig. 3 explicitly assumes an oracle policy, so it does not actually substantiate the 'accurate dynamics consistency' claim in the operating regime that matters. That said, the main benchmark results are themselves an end-to-end test of the closed loop, and they work; so the stress-test concern weakens the explanation, not the result. The paper should either add a direct test of recursive rollouts or temper the claim.\n\nThere are also several presentational problems. The hyperparameter reporting is inconsistent: the main text says k=8 in the training-cost section, the appendix says [32], and the discussions section says k=32. The code link is a placeholder ('RSP JAX'), not a working URL, which is a serious issue for an empirical paper. HIQL appears in Table 1 but has no reference. Baseline scores are a mix of self-reproduced, from other papers, and v0 vs v2 IQL scores; the authors disclose this, but it's a weakness. And hyperparameters were tuned on AntMaze-Ultra, the same family where the largest gains appear; they are fixed across tasks, so this is minor, but it should be acknowledged.\n\nOverall: this is a solid, novel empirical contribution that would benefit from a serious referee. The method is simple enough to be widely adopted, and the compute advantage is concrete. I'd encourage the editor to send it to review, with the requirement that the code be made available, the hyperparameter inconsistency resolved, and the closed-loop distribution shift addressed, either experimentally or by reframing the explanation. It is not a desk reject.","headline":"RSP is a genuinely new, simple algorithm with strong long-horizon D4RL results, but the paper's explanation of why it works is not fully validated and a few reproducibility issues need fixing before the numbers are fully trustworthy.","tokens_in":15607,"tokens_out":3391,"would_cite":true,"duration_ms":30313,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a 2-layer MLP recursive planner can match or beat heavy transformers and diffusion planners on long-horizon offline RL, in minutes of training.","keywords":["offline reinforcement learning","goal-conditioned supervised learning","recursive skip-step planning","subgoal prediction","compounding error","model efficiency","D4RL","long-horizon planning"],"falsifier":"On AntMaze-Ultra, run RSP once with the lowest-level subgoal taken from the ground-truth relabeled dataset and once with the recursively predicted subgoal, holding everything else fixed. If success rate rises sharply with the ground-truth subgoal, the recursive predictors are not closing the loop and the long-horizon advantage depends on policy conditioning rather than on accurate planning.","tokens_in":14626,"feed_emoji":"🤖","tokens_out":9367,"duration_ms":79538,"temperature":0.7,"pith_summary":"This paper asks whether large expressive models are necessary for offline reinforcement learning with long horizons, and answers no. It argues that a lightweight recursive planner can keep sequential modeling error low enough that shallow two-layer MLPs outperform transformer- and diffusion-based sequence models. The proposed Recursive Skip-Step Planning (RSP) predicts coarse future subgoals at exponentially shrinking horizons and conditions a goal-conditioned policy on them, entirely in a supervised-learning framework. The claim is that this combination yields state-of-the-art D4RL scores on multi-stage long-horizon tasks while training in roughly 180 seconds and acting in under a millisecond.","feed_headline":"2-layer MLPs beat transformers on long-horizon offline RL","feed_subtitle":"Recursive subgoal planning trains in minutes, reacts in milliseconds, and tops D4RL AntMaze and Kitchen.","key_machinery":"The central object is the recursive skip-step dynamics stack. For recursion depth $N$ and top-level skip $K$, the dataset is relabeled so the $n$-th model predicts the subgoal at horizon $K(n)=K/2^n$, conditioned on all higher-level subgoals $\\kappa^{(n-1)}$; the lowest-level subgoal then conditions the policy $\\pi(a_t | \\kappa^{(N)})$. At evaluation, Equation 6 runs the stack top-down, each model consuming the previous model's output. This reduces long-horizon planning to an exponentially shrinking chain of coarse-to-fine predictions, which is why a few recursive calls can replace thousands of single-step rollouts and why rollout RMSE falls as the skip step grows from 4 to 64.","core_discovery":"The central discovery is that coarse-grained recursive planning removes the need for high-capacity models in long-horizon offline RL. RSP relabels each transition with subgoal ground truths at horizons $K/2^n$, learns a stack of $N$ skip-step dynamics models and one goal-conditioned policy by maximum likelihood, and at test time predicts subgoals recursively from the highest to the lowest level before generating an action. Measured on D4RL, this scheme reaches new state-of-the-art scores on AntMaze and Kitchen, including about 80 normalized return on AntMaze-Ultra tasks where transformer and diffusion baselines mostly fail, while using only two-layer 1024-unit MLPs. The authors' interpretation is that long horizons are best handled by planning few, coarse, recursively refined subgoals instead of many fine-grained steps, so compounding error stops growing with trajectory length.","pith_inferences":["The paper's RMSE analysis suggests a testable scaling property: as the top-level skip $K$ grows, the number of recursive calls needed to cover a fixed horizon shrinks logarithmically, so the scheme may extend to horizons far beyond 3000 steps; the paper does not run that test.","A direct robustness probe left untested is to inject small noise into the lowest-level predicted subgoal and measure the success drop, which would quantify how much of RSP's edge depends on subgoal accuracy versus the policy's tolerance to inaccurate goals.","Training the recursive dynamics models on their own predictions, a closed-loop objective, would target the weakest assumption directly and could make the method robust to drift; this is an extension, not a claim of the paper.","The method is evaluated in near-deterministic simulated environments; how single-point subgoal predictions behave under stochastic transitions is an open question, since a fixed future state may not be well-defined when dynamics are random."],"forward_implications":["RSP reaches state-of-the-art normalized scores on D4RL AntMaze across all nine datasets, including about 80 on the extremely long AntMaze-Ultra tasks where most baselines score below 40 and several score zero.","Because RSP uses only two-layer MLPs, its training time is about 180 seconds and inference under 1 ms at recursion depth one, orders of magnitude faster than transformer- or diffusion-based sequence planners of comparable score.","Coarse-grained skip-step prediction lowers rollout RMSE as the skip step grows from 4 to 64, meaning cumulative error falls as the planning step becomes coarser rather than rising with sequence length.","Adding recursion depth beyond two gives little benefit on most tasks, because each extra prediction level adds its own error, so depth two is the recommended cost-performance point."],"supporting_citations":[{"why":"Supplies the D4RL benchmark datasets and the normalized-score protocol every comparison in the paper uses.","marker":"Fu et al. 2020"},{"why":"Defines the goal-conditioned supervised-learning objective that RSP extends with skip-step subgoals.","marker":"Emmons et al. 2021"},{"why":"Decision Transformer is the expressive sequence-modeling baseline whose long-horizon compounding error motivates recursive planning.","marker":"Chen et al. 2021"},{"why":"Trajectory Transformer models step-by-step sequences and is the fine-grained planning baseline RSP is contrasted against.","marker":"Janner, Li, and Levine 2021"},{"why":"Diffuser is the diffusion-based trajectory planner baseline used to represent expressive-model performance.","marker":"Janner et al. 2022"},{"why":"TAP is the latent-space planning baseline; the paper takes its AntMaze-Ultra and IQL-v2 scores from TAP's report.","marker":"Zhang et al. 2022"},{"why":"IQL is the in-sample temporal-difference baseline RSP must match on long-horizon and multi-stage tasks.","marker":"Kostrikov, Nair, and Levine 2021"},{"why":"Frames the compounding-error and distribution-shift concerns that the recursive planning scheme is designed to bypass.","marker":"Levine et al. 2020"}],"fun_headline_variants":["Recursive subgoals let 2-layer MLPs beat transformers in offline RL","Recursive subgoal planning lets 2-layer MLPs beat large models in offline RL","Skip-step planning: tiny MLPs rival expensive offline RL transformers","Coarse subgoals, not big models: tiny MLPs top offline RL benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that subgoal predictors trained on real future states stay accurate when they are fed their own predictions at test time; if the predicted subgoals drift away from the training distribution, the recursive chain and the policy that consumes it will degrade.","fun_headline_variants_meta":{"raw":{"variants":["Recursive subgoals let 2-layer MLPs beat transformers in offline RL","Recursive subgoal planning lets 2-layer MLPs beat large models in offline RL","Skip-step planning: tiny MLPs rival expensive offline RL transformers","Coarse subgoals, not big models: tiny MLPs top offline RL benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000916,"raw_usage":{"total_tokens":3936,"prompt_tokens":951,"completion_tokens":2985,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2900}},"tokens_in":567,"tokens_out":2985,"duration_ms":21362,"temperature":1.0,"reasoning_tokens":2900,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:08:23.608111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On AntMaze-Ultra, run RSP once with the lowest-level subgoal taken from the ground-truth relabeled dataset and once with the recursively predicted subgoal, holding everything else fixed. If success rate rises sharply with the ground-truth subgoal, the recursive predictors are not closing the loop and the long-horizon advantage depends on policy conditioning rather than on accurate planning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Trajectory Transformer models step-by-step sequences and is the fine-grained planning baseline RSP is contrasted against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TAP is the latent-space planning baseline; the paper takes its AntMaze-Ultra and IQL-v2 scores from TAP's report."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames the compounding-error and distribution-shift concerns that the recursive planning scheme is designed to bypass."}],"review_version":1}