{"id":"25aa588a-f049-42bc-a3b0-e725ffa9c633","arxiv_id":"2501.12668","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NBDI learns skill termination from state-action novelty (ICM prediction error) on task-agnostic demonstrations, improving downstream RL performance in maze and manipulation benchmarks.","lead":"This paper proposes NBDI, a method that uses state-action novelty to decide when a learned skill should end, turning fixed-length skills into variable-length ones. It reports faster and better learning in maze and robot manipulation tasks, including when the test environment differs from the training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains are conditional on per-environment novelty thresholds tuned with downstream performance; the post-hoc percentile rule in Appendix B.3 does not yet establish transferability.","rationale":"The paper is clearly written and the empirical story is coherent: NBDI changes the termination distribution of skills, and the ablations show that the termination distribution and state-action novelty (rather than state novelty alone) matter. The comparison to SPiRL with matched average skill length (Appendix A.4) rules out the trivial explanation that NBDI only wins by using longer skills. However, the central claim - that state-action novelty identifies decision points that improve downstream learning - is not yet cleanly separated from the per-environment threshold. Appendix A.1 is explicit that the performance gain disappears with the wrong threshold, and Appendix B.3 admits the thresholds were tuned through experiments. That makes the headline numbers conditional on a downstream-influenced choice; the 97th-percentile rule is a post-hoc rationalization, not a prescriptive procedure. I would not reject on this basis alone: the threshold is a single scalar, the percentile guidance may turn out to be sufficient, and the method has a clear mechanism. But it is the load-bearing point, and a held-out evaluation with a fixed threshold would settle it. The Section 4.2 theorem is motivational rather than a proof that ICM error identifies termination-improvement states; that gap reinforces, rather than replaces, the threshold concern.","tokens_in":23559,"tokens_out":11769,"duration_ms":109147,"concrete_test":"Hold out a new environment not used in the paper (e.g., an unseen maze layout or a new Kitchen arrangement), fix the novelty threshold a priori at the 97th percentile of the task-agnostic novelty distribution as prescribed by Appendix B.3, and run NBDI without any downstream performance feedback, comparing against SPiRL and a random-termination baseline matched for skill-length distribution. If NBDI does not reproduce its margin over SPiRL, the reported gains rest on thresholds tuned with downstream performance. Optionally, also compute the rank correlation between ICM novelty and termination-improvement frequency in the Figure 3 grid to test the proxy directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the threshold that turns ICM prediction error into termination labels. Appendix A.1 shows performance is sensitive to it: in Kitchen, NBDI-th0.1 and NBDI-th0.5 lose the clear margin over SPiRL. Appendix B.3 states the actual thresholds (kitchen 0.3, maze 50, block stacking 40) 'were tuned through experiments', i.e., using downstream task performance, and only afterward notes they lie near the 97th percentile. That percentile is a post-hoc summary of tuned values, not a pre-specified rule; Appendix I.2 separately says the top 1% prediction error percentile was used, so the labeling rule is not crisply defined. Because the threshold controls which state-action pairs train the termination distribution β, and because downstream behavior depends on β, the empirical claim 'NBDI outperforms SPiRL' is coupled to per-environment tuning. Section 4.2 does not close this gap: the termination improvement theorem is invoked, but no argument or bound shows that high ICM error identifies states where V(s) > Q(s,o); Figure 3 is qualitative and only in a small grid. For a method whose selling point is task-agnostic transfer, the core hyperparameter should not require downstream-labeled tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NBDI, a termination condition for variable-length skill extraction from task-agnostic demonstrations. State-action novelty is computed by the Intrinsic Curiosity Module (ICM) prediction error on offline demonstrations, a threshold converts novelty into Bernoulli termination labels, and a termination distribution β is trained jointly with a skill decoder and skill prior in a latent variable model. In downstream RL, a high-level policy selects skill embeddings and the low-level policy executes until β signals termination or a maximum length H=30 is reached. The method is evaluated on two maze navigation tasks, sparse block stacking, and D4RL Kitchen, showing consistent improvements over SPiRL and other baselines, alongside ablations on novelty components, thresholds, and termination distribution, and comparisons to LOVE and relative novelty.","tokens_in":23792,"tokens_out":5772,"duration_ms":54158,"significance":"If the empirical claim holds, NBDI addresses a real limitation of fixed-length skill extraction by allowing termination at decision points such as crossroads or subtask-completion states, without needing downstream rewards or task labels for the termination model. The paper's strengths include an open-source implementation, a broad set of baselines and ablations, a comparison with other variable-length skill methods, and a demonstration that NBDI improves SiMPL meta-RL. The principal weakness is that the core hyperparameter—the novelty threshold—is tuned per environment on downstream task performance, which tempers the 'task-agnostic transfer' claim. A fixed, pre-specified percentile rule that performs as well without downstream tuning would make the contribution substantially stronger.","major_comments":[{"comment":"The termination labels are produced by thresholding ICM prediction error, and the thresholds are environment-specific and were tuned using downstream task performance: Appendix B.3 states 'the thresholds we tuned through experiments (see Appendix A.1)'. Appendix A.1, Figure 10a shows that in Kitchen both NBDI-th0.1 and NBDI-th0.5 lose the clear margin over SPiRL. Because the threshold determines which state-action pairs train the termination distribution β, and β controls downstream skill switching, the headline claim of a task-agnostic termination condition is conditional on a per-environment hyperparameter fitted to the downstream task. Please either evaluate a fixed, pre-specified rule (e.g., the top 1% or top 3% of the task-agnostic novelty distribution) without downstream tuning, or reframe the contribution as requiring a tuned threshold and report the tuning cost.","section":"5.1, A.1, B.3"},{"comment":"Theorem 4.1 justifies terminating when V^μ(s) > Q^μ(s,o), but the paper does not show that high ICM prediction error identifies states satisfying this condition. Figure 3 was computed in an 8x8 grid with exact counts, not with the ICM estimator, and the connection is supported only by qualitative visualizations in Figure 2. Please provide a quantitative test on the task-agnostic datasets—for example, measuring the agreement between ICM error and termination-improvement frequency or a proxy such as action-value dispersion—or weaken the theoretical framing to a heuristic.","section":"4.2"},{"comment":"The termination distribution is defined inconsistently across the manuscript: Algorithm 1 samples β_{t+k} ∼ p(β_{t+k}|s_{t+k}, a_{t+k}); Appendix B.2, Algorithm 2 samples (a_{t+k}, β_{t+k}) ∼ p_ψ(·|z_t, s_t); and Section 6.3 describes the termination distribution as p(β|z,s). Because the method is defined by how β is learned and used in the downstream loop, this discrepancy affects both interpretation and reproduction. Please specify the exact distribution used and correct the pseudocode accordingly.","section":"5.2 (Algorithm 1), B.2 (Algorithm 2), 6.3"},{"comment":"The labeling rule is ambiguous: Appendix I.2 states that 'state-actions within the top 1% prediction error percentile serve well as critical decision points', while Appendix B.3 reports thresholds corresponding to 96–97th percentiles of the novelty distribution (i.e., the top 3–4%). These are different labeling rules, and the choice materially changes the termination labels. Please report the exact rule, including how the threshold is converted into Bernoulli labels for the decoder, and align the two appendices.","section":"I.2 vs B.3"}],"minor_comments":[{"comment":"In Sparse Block Stacking, the 95% confidence intervals for NBDI (1.12 ± 0.16) and SPiRL (0.67 ± 0.29) nearly overlap; a paired-seed comparison or a significance test would strengthen the improvement claim in this environment.","section":"Figure 5 table"},{"comment":"The three panels are small and use similar color scales, which makes cross-panel comparison difficult; consider plotting termination improvement, conditional action novelty, and state novelty on a common color scale with larger panels.","section":"Figure 3"},{"comment":"There are typos: 'Institute of Information& Coummunications Technology' in the acknowledgments and 'termianted skills' in the Figure 13 caption.","section":"Acknowledgments and Figure 13"},{"comment":"The claim that novelty estimation is robust to dataset size is based on visual inspection of box plots (Figure 8, right); report quantitative statistics, such as rank correlations between dataset usage and novelty scale, if this robustness is a stated contribution.","section":"6.6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the transferability claim is overstated given the downstream tuning of thresholds. The empirical core is solid enough for a major revision; if the authors adopt a fixed percentile rule and report results without per-environment downstream tuning, I would be inclined to accept. The inconsistency between Algorithm 1 and Algorithm 2 also suggests that the released code should be checked against the paper's description, since the termination distribution is the methodological heart of the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful new thing here is simple: use ICM prediction error on task-agnostic demonstrations to define where skills should terminate, instead of fixed-length segmentation. That is a clean, cheap extension of SPiRL-style skill extraction, and the experiments go beyond a same-layout transfer setting. The maze results with entirely new layouts and the block-stacking scale-up are genuinely more convincing than the typical skill-extraction evaluation. The kitchens, where NBDI gets all four subtasks and SPiRL plateaus at three, is a solid point in its favor. I also give credit for the ablations: the comparisons against state-novelty-only and action-novelty-only variants, and the SPiRL-with-average-length control, directly test the mechanism rather than just the headline.\n\nThe soft spot is exactly what the stress test says. The termination threshold is the load-bearing hyperparameter, and it is tuned per environment using downstream performance (Appendix B.3 admits this). Appendix A.1 shows performance is sensitive to it. The post-hoc observation that the tuned values sit near the 97th percentile is useful guidance, but it is not a pre-specified rule. Appendix I.2 even says the top 1% prediction error was used, which is not the same as the 97th percentile. That inconsistency matters because the method is marketed as task-agnostic; a practitioner cannot know the right threshold without downstream labels. The termination-improvement theorem in Section 4.2 is invoked loosely: no argument links high ICM error to V(s) > Q(s,o), and the grid-world visualization is qualitative. That said, the link is plausible and the ablation suggests conditional-action novelty does most of the work, which is consistent with the crossroads intuition.\n\nAre the results overclaimed? A bit. Some confidence intervals overlap in block stacking, and the kitchen margin is modest. But the maze gains are large and consistent, and the transfer demonstration is more than most papers in this area attempt. The paper is honest about the stochastic-data limitation (Section 6.5).\n\nWho is this for? Anyone building skill-based RL pipelines for long-horizon sparse-reward tasks, especially with offline demonstration data. It is a useful incremental contribution, not a fundamental rethinking. The central idea is likely to survive further scrutiny, but the threshold-tuning issue should be addressed with a pre-registered percentile rule or a sensitivity-aware claim. I would send this to a serious referee if it were new to me; the concerns are addressable and the empirical core is valuable. My own verdict is mixed but leans positive.\n\nFor your question: bring it to reading group? Maybe. Cite it? Probably, if you work on skill extraction. Peer review: yes, worth a serious referee.","headline":"A plausible skill-termination idea with real transfer merit, but the headline numbers lean on per-environment threshold tuning that the paper only partially owns.","tokens_in":24325,"tokens_out":1244,"would_cite":true,"duration_ms":14730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Novelty-based skill termination lifts maze success by 178%.","keywords":["state-action novelty","skill termination condition","temporal abstraction","task-agnostic demonstrations","hierarchical reinforcement learning","variable-length skills","intrinsic curiosity module","decision points"],"falsifier":"In a fresh maze, compute $V^{\\mu}(s)$ and $Q^{\\mu}(s,o)$ for the fixed-length skill policy, mark all states where the termination improvement theorem says terminating raises value, and compare those with NBDI's top-1% state-action novelty points; if the overlap is no better than chance, the claimed mechanism is falsified.","tokens_in":23353,"feed_emoji":"🎯","tokens_out":8590,"duration_ms":80302,"temperature":0.7,"pith_summary":"Skills are usually fixed-length chunks of behavior, so the agent can blow past the moment when it should change plan. NBDI ends a skill whenever the current state–action pair is novel, measured as the prediction error of a model pretrained on task-agnostic demonstrations, because those points are where multiple plausible futures branch. A Bernoulli termination distribution is trained on that novelty signal, jointly with the skill embedding, and downstream learning runs with variable-length skills. In the reported experiments NBDI beats the fixed-length skill baseline SPiRL on two maze sizes, sparse block stacking, and kitchen, including under changed environment configurations, with the largest relative gain a 177.78% success-rate improvement on the 40×40 maze. The paper's point is that when a skill stops is itself a learnable, transferable decision, and state-action novelty is a cheap signal for finding those stopping points.","feed_headline":"Novelty-based skill termination lifts maze success 178%","feed_subtitle":"Rare state-action pairs mark where a learned skill should end, beating fixed-length skills in navigation and robot manipulation.","key_machinery":"The load-bearing object is the state-action novelty module, instantiated as an Intrinsic Curiosity Module (ICM) that predicts the next state's feature encoding from $(s,a)$ and uses the squared $L^2$ prediction error as the novelty score $\\chi(s,a)$. The decomposition $\\chi(s,a) = \\chi(s)\\,\\chi(a|s)$ gives the method its semantics: state novelty flags unfamiliar regions that deserve more decisions, while conditional action novelty flags crossroads where many actions are plausible and the current skill is likely suboptimal. The same module's error ranks are converted into a Bernoulli termination distribution $p(\\beta|s,a)$ via per-environment thresholds near the 97th percentile of offline novelty values, and this termination distribution is trained jointly with the skill VAE so that the skill embedding space itself is shaped by where skills end. In downstream reinforcement learning, the termination signal turns the problem into an SMDP with discounted $Q$-updates over the variable duration $k$ of each skill, regularized toward the learned skill prior; this is the mechanism that lets the agent decide at bottlenecks rather than at fixed intervals.","core_discovery":"The central claim is that critical decision points for temporally abstracted behavior—places where a skill should hand control back to a higher-level policy—can be identified from task-agnostic offline trajectories by state-action novelty alone. The paper defines novelty as $\\chi(s,a) = \\chi(s)\\,\\chi(a|s)$, so a high value means both that the state is rare in the demonstration data and that, given that state, many different actions have been taken. An Intrinsic Curiosity Module (ICM) estimates this quantity through the squared $L^2$ prediction error of a next-state feature model. State-action pairs whose novelty exceeds roughly the 97th percentile of offline values are labeled as terminations, and a model $p(\\beta|s,a)$ is trained to predict them jointly with the skill encoder, decoder, and prior through an evidence-lower-bound objective. Downstream, the agent runs Soft Actor-Critic over the Semi-Markov Decision Process induced by variable-length skills, terminating either at a predicted $\\beta=1$ or at the maximum length $H=30$. On this machinery the paper reports that NBDI outperforms the fixed-length skill prior SPiRL by 84.62% and 177.78% success rate in the two maze navigation tasks, by 67.16% in sparse block stacking, and by 22.33% in completed kitchen subtasks, and visualizations show the termination points clustering at crossroads and subtask boundaries.","pith_inferences":["Because novelty is computed purely from offline data without rewards or task labels, the same termination criterion could apply to offline skill execution and to settings where demonstrations arrive incrementally; the paper names offline skill execution as future work, and the rest is my inference, not a paper claim.","If the observed correspondence between thresholds and the ~97th percentile holds across new environments, NBDI reduces termination-condition design to a percentile sweep; the paper reports its three thresholds match that percentile but does not claim it as a general law.","A causal test of the story would compare NBDI's termination points with an oracle built from the termination-improvement theorem: if top-novelty states do not align with states where $Q^{\\mu}(s,o) < V^{\\mu}(s)$, the reported gains would need another explanation, such as exploration rather than better decision points.","The measured sensitivity to dataset stochasticity suggests that novelty-based termination inherits data quality; for datasets dominated by near-initial-state noise, a different signal, such as epistemic uncertainty of the dynamics model, would be needed to recover meaningful decision points."],"forward_implications":["Switching from fixed-length to novelty-terminated skills improves downstream RL performance in all four tested environments, with the largest relative gain in the harder 40×40 maze (177.78% over SPiRL).","Conditional action novelty, not state novelty alone, is the main driver of the improvement; the full state-action product performs best in ablations.","The termination rule transfers to downstream tasks with substantially different environment configurations when observations are agent-centered crops.","NBDI also improves the skill-based meta-RL method SiMPL when used during skill extraction, roughly quadrupling early success rate in the maze meta-training phase relative to fixed-length skills.","The approach remains useful on mediocre stochastic behavior-cloning data, but its decision-point detection degrades as dataset stochasticity increases toward random walks."],"supporting_citations":[{"why":"Supplies the Intrinsic Curiosity Module whose next-state prediction error is the state-action novelty estimator at the core of NBDI.","marker":"Pathak et al. (2017)"},{"why":"Provides the SPiRL fixed-length skill prior and low-level skill extraction framework that NBDI builds on and compares against.","marker":"Pertsch et al. (2021a)"},{"why":"Provides the option framework and the termination improvement theorem that motivate terminating skills where better alternatives exist.","marker":"Sutton (1998)"},{"why":"LOVE is the main variable-length skill extraction baseline against which NBDI is evaluated in maze and kitchen.","marker":"Jiang et al. (2022)"},{"why":"Soft Actor-Critic is the downstream high-level RL algorithm adapted to the SMDP induced by variable-length skills.","marker":"Haarnoja et al. (2018)"},{"why":"Relative novelty is a baseline termination method and the source of the access-state intuition that links subgoals to novelty.","marker":"S ˙ims ˙ek & Barto, 2004"},{"why":"Supplies the kitchen and maze environment configurations and offline datasets used for training and evaluation.","marker":"Fu et al. (2021)"},{"why":"SiMPL is the skill-based meta-RL method into which NBDI is plugged in Section 7 to show broader applicability.","marker":"Nam et al. (2022)"}],"fun_headline_variants":["Novelty-based skill termination boosts maze success 178%","State-action novelty marks skill end points, beating baselines","Skill termination via novelty: 178% maze success gain","NBDI: rare state-actions decide when to switch skills","Off-policy novelty finds decision points for skill reuse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a state–action pair with high next-state prediction error on the task-agnostic dataset marks a state where ending a skill helps downstream learning, that per-environment novelty thresholds tuned to downstream performance keep that meaning, and that dataset stochasticity stays low enough for prediction error to separate decision points from noise.","fun_headline_variants_meta":{"raw":{"variants":["Novelty-based skill termination boosts maze success 178%","State-action novelty marks skill end points, beating baselines","Skill termination via novelty: 178% maze success gain","NBDI: rare state-actions decide when to switch skills","Off-policy novelty finds decision points for skill reuse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2384,"prompt_tokens":983,"completion_tokens":1401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1318}},"tokens_in":599,"tokens_out":1401,"duration_ms":13193,"temperature":1.0,"reasoning_tokens":1318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:55:08.918241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a fresh maze, compute $V^{\\mu}(s)$ and $Q^{\\mu}(s,o)$ for the fixed-length skill policy, mark all states where the termination improvement theorem says terminating raises value, and compare those with NBDI's top-1% state-action novelty points; if the overlap is no better than chance, the claimed mechanism is falsified.","supporting_citations":[],"review_version":1}