NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations

Byung-Jun Lee; Hayeong Lee; Junho Seo; Myunsoo Kim; Seong-Woong Shim

REVIEW 4 major objections 4 minor 24 references

NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Novelty-based skill termination lifts maze success by 178%.

desk verdict A plausible skill-termination idea with real transfer merit, but the headline numbers lean on per-environment threshold tuning that the paper only partially owns. read the letter →

arxiv 2501.12668 v3 pith:OCCDO2XT submitted 2025-01-22 cs.LG cs.AI

Myunsoo Kim , Hayeong Lee , Seong-Woong Shim , JunHo Seo , Byung-Jun Lee This is my paper · ORCID

classification cs.LGcs.AI

keywords state-actionnoveltyskillterminationconditiontemporalabstractiontask-agnosticdemonstrationshierarchicalreinforcementlearningvariable-lengthskillsintrinsiccuriositymoduledecisionpoints

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Skills are usually fixed-length chunks of behavior, so the agent can blow past the moment when it should change plan. NBDI ends a skill whenever the current state–action pair is novel, measured as the prediction error of a model pretrained on task-agnostic demonstrations, because those points are where multiple plausible futures branch. A Bernoulli termination distribution is trained on that novelty signal, jointly with the skill embedding, and downstream learning runs with variable-length skills. In the reported experiments NBDI beats the fixed-length skill baseline SPiRL on two maze sizes, sparse block stacking, and kitchen, including under changed environment configurations, with the largest relative gain a 177.78% success-rate improvement on the 40×40 maze. The paper's point is that when a skill stops is itself a learnable, transferable decision, and state-action novelty is a cheap signal for finding those stopping points.

What carries the argument

The load-bearing object is the state-action novelty module, instantiated as an Intrinsic Curiosity Module (ICM) that predicts the next state's feature encoding from $(s,a)$ and uses the squared $L^2$ prediction error as the novelty score $\chi(s,a)$. The decomposition $\chi(s,a) = \chi(s)\,\chi(a|s)$ gives the method its semantics: state novelty flags unfamiliar regions that deserve more decisions, while conditional action novelty flags crossroads where many actions are plausible and the current skill is likely suboptimal. The same module's error ranks are converted into a Bernoulli termination distribution $p(\beta|s,a)$ via per-environment thresholds near the 97th percentile of offline novelty values, and this termination distribution is trained jointly with the skill VAE so that the skill embedding space itself is shaped by where skills end. In downstream reinforcement learning, the termination signal turns the problem into an SMDP with discounted $Q$-updates over the variable duration $k$ of each skill, regularized toward the learned skill prior; this is the mechanism that lets the agent decide at bottlenecks rather than at fixed intervals.

What would settle it

In a fresh maze, compute $V^{\mu}(s)$ and $Q^{\mu}(s,o)$ for the fixed-length skill policy, mark all states where the termination improvement theorem says terminating raises value, and compare those with NBDI's top-1% state-action novelty points; if the overlap is no better than chance, the claimed mechanism is falsified.

Watch

Extended reading notes

Core claim

The central claim is that critical decision points for temporally abstracted behavior—places where a skill should hand control back to a higher-level policy—can be identified from task-agnostic offline trajectories by state-action novelty alone. The paper defines novelty as $\chi(s,a) = \chi(s)\,\chi(a|s)$, so a high value means both that the state is rare in the demonstration data and that, given that state, many different actions have been taken. An Intrinsic Curiosity Module (ICM) estimates this quantity through the squared $L^2$ prediction error of a next-state feature model. State-action pairs whose novelty exceeds roughly the 97th percentile of offline values are labeled as terminations, and a model $p(\beta|s,a)$ is trained to predict them jointly with the skill encoder, decoder, and prior through an evidence-lower-bound objective. Downstream, the agent runs Soft Actor-Critic over the Semi-Markov Decision Process induced by variable-length skills, terminating either at a predicted $\beta=1$ or at the maximum length $H=30$. On this machinery the paper reports that NBDI outperforms the fixed-length skill prior SPiRL by 84.62% and 177.78% success rate in the two maze navigation tasks, by 67.16% in sparse block stacking, and by 22.33% in completed kitchen subtasks, and visualizations show the termination points clustering at crossroads and subtask boundaries.

Load-bearing premise

The method assumes that a state–action pair with high next-state prediction error on the task-agnostic dataset marks a state where ending a skill helps downstream learning, that per-environment novelty thresholds tuned to downstream performance keep that meaning, and that dataset stochasticity stays low enough for prediction error to separate decision points from noise.

Editorial extensions

If this is right

Switching from fixed-length to novelty-terminated skills improves downstream RL performance in all four tested environments, with the largest relative gain in the harder 40×40 maze (177.78% over SPiRL).
Conditional action novelty, not state novelty alone, is the main driver of the improvement; the full state-action product performs best in ablations.
The termination rule transfers to downstream tasks with substantially different environment configurations when observations are agent-centered crops.
NBDI also improves the skill-based meta-RL method SiMPL when used during skill extraction, roughly quadrupling early success rate in the maze meta-training phase relative to fixed-length skills.
The approach remains useful on mediocre stochastic behavior-cloning data, but its decision-point detection degrades as dataset stochasticity increases toward random walks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because novelty is computed purely from offline data without rewards or task labels, the same termination criterion could apply to offline skill execution and to settings where demonstrations arrive incrementally; the paper names offline skill execution as future work, and the rest is my inference, not a paper claim.
If the observed correspondence between thresholds and the ~97th percentile holds across new environments, NBDI reduces termination-condition design to a percentile sweep; the paper reports its three thresholds match that percentile but does not claim it as a general law.
A causal test of the story would compare NBDI's termination points with an oracle built from the termination-improvement theorem: if top-novelty states do not align with states where $Q^{\mu}(s,o) < V^{\mu}(s)$, the reported gains would need another explanation, such as exploration rather than better decision points.
The measured sensitivity to dataset stochasticity suggests that novelty-based termination inherits data quality; for datasets dominated by near-initial-state noise, a different signal, such as epistemic uncertainty of the dynamics model, would be needed to recover meaningful decision points.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful new thing here is simple: use ICM prediction error on task-agnostic demonstrations to define where skills should terminate, instead of fixed-length segmentation. That is a clean, cheap extension of SPiRL-style skill extraction, and the experiments go beyond a same-layout transfer setting. The maze results with entirely new layouts and the block-stacking scale-up are genuinely more convincing than the typical skill-extraction evaluation. The kitchens, where NBDI gets all four subtasks and SPiRL plateaus at three, is a solid point in its favor. I also give credit for the ablations: the comparisons against state-novelty-only and action-novelty-only variants, and the SPiRL-with-average-length control, directly test the mechanism rather than just the headline.

The soft spot is exactly what the stress test says. The termination threshold is the load-bearing hyperparameter, and it is tuned per environment using downstream performance (Appendix B.3 admits this). Appendix A.1 shows performance is sensitive to it. The post-hoc observation that the tuned values sit near the 97th percentile is useful guidance, but it is not a pre-specified rule. Appendix I.2 even says the top 1% prediction error was used, which is not the same as the 97th percentile. That inconsistency matters because the method is marketed as task-agnostic; a practitioner cannot know the right threshold without downstream labels. The termination-improvement theorem in Section 4.2 is invoked loosely: no argument links high ICM error to V(s) > Q(s,o), and the grid-world visualization is qualitative. That said, the link is plausible and the ablation suggests conditional-action novelty does most of the work, which is consistent with the crossroads intuition.

Are the results overclaimed? A bit. Some confidence intervals overlap in block stacking, and the kitchen margin is modest. But the maze gains are large and consistent, and the transfer demonstration is more than most papers in this area attempt. The paper is honest about the stochastic-data limitation (Section 6.5).

Who is this for? Anyone building skill-based RL pipelines for long-horizon sparse-reward tasks, especially with offline demonstration data. It is a useful incremental contribution, not a fundamental rethinking. The central idea is likely to survive further scrutiny, but the threshold-tuning issue should be addressed with a pre-registered percentile rule or a sensitivity-aware claim. I would send this to a serious referee if it were new to me; the concerns are addressable and the empirical core is valuable. My own verdict is mixed but leans positive.

For your question: bring it to reading group? Maybe. Cite it? Probably, if you work on skill extraction. Peer review: yes, worth a serious referee.

Referee Report

4 major / 4 minor

Summary. The paper proposes NBDI, a termination condition for variable-length skill extraction from task-agnostic demonstrations. State-action novelty is computed by the Intrinsic Curiosity Module (ICM) prediction error on offline demonstrations, a threshold converts novelty into Bernoulli termination labels, and a termination distribution β is trained jointly with a skill decoder and skill prior in a latent variable model. In downstream RL, a high-level policy selects skill embeddings and the low-level policy executes until β signals termination or a maximum length H=30 is reached. The method is evaluated on two maze navigation tasks, sparse block stacking, and D4RL Kitchen, showing consistent improvements over SPiRL and other baselines, alongside ablations on novelty components, thresholds, and termination distribution, and comparisons to LOVE and relative novelty.

Significance. If the empirical claim holds, NBDI addresses a real limitation of fixed-length skill extraction by allowing termination at decision points such as crossroads or subtask-completion states, without needing downstream rewards or task labels for the termination model. The paper's strengths include an open-source implementation, a broad set of baselines and ablations, a comparison with other variable-length skill methods, and a demonstration that NBDI improves SiMPL meta-RL. The principal weakness is that the core hyperparameter—the novelty threshold—is tuned per environment on downstream task performance, which tempers the 'task-agnostic transfer' claim. A fixed, pre-specified percentile rule that performs as well without downstream tuning would make the contribution substantially stronger.

major comments (4)

[5.1, A.1, B.3] The termination labels are produced by thresholding ICM prediction error, and the thresholds are environment-specific and were tuned using downstream task performance: Appendix B.3 states 'the thresholds we tuned through experiments (see Appendix A.1)'. Appendix A.1, Figure 10a shows that in Kitchen both NBDI-th0.1 and NBDI-th0.5 lose the clear margin over SPiRL. Because the threshold determines which state-action pairs train the termination distribution β, and β controls downstream skill switching, the headline claim of a task-agnostic termination condition is conditional on a per-environment hyperparameter fitted to the downstream task. Please either evaluate a fixed, pre-specified rule (e.g., the top 1% or top 3% of the task-agnostic novelty distribution) without downstream tuning, or reframe the contribution as requiring a tuned threshold and report the tuning cost.
[4.2] Theorem 4.1 justifies terminating when V^μ(s) > Q^μ(s,o), but the paper does not show that high ICM prediction error identifies states satisfying this condition. Figure 3 was computed in an 8x8 grid with exact counts, not with the ICM estimator, and the connection is supported only by qualitative visualizations in Figure 2. Please provide a quantitative test on the task-agnostic datasets—for example, measuring the agreement between ICM error and termination-improvement frequency or a proxy such as action-value dispersion—or weaken the theoretical framing to a heuristic.
[5.2 (Algorithm 1), B.2 (Algorithm 2), 6.3] The termination distribution is defined inconsistently across the manuscript: Algorithm 1 samples β_{t+k} ∼ p(β_{t+k}|s_{t+k}, a_{t+k}); Appendix B.2, Algorithm 2 samples (a_{t+k}, β_{t+k}) ∼ p_ψ(·|z_t, s_t); and Section 6.3 describes the termination distribution as p(β|z,s). Because the method is defined by how β is learned and used in the downstream loop, this discrepancy affects both interpretation and reproduction. Please specify the exact distribution used and correct the pseudocode accordingly.
[I.2 vs B.3] The labeling rule is ambiguous: Appendix I.2 states that 'state-actions within the top 1% prediction error percentile serve well as critical decision points', while Appendix B.3 reports thresholds corresponding to 96–97th percentiles of the novelty distribution (i.e., the top 3–4%). These are different labeling rules, and the choice materially changes the termination labels. Please report the exact rule, including how the threshold is converted into Bernoulli labels for the decoder, and align the two appendices.

minor comments (4)

[Figure 5 table] In Sparse Block Stacking, the 95% confidence intervals for NBDI (1.12 ± 0.16) and SPiRL (0.67 ± 0.29) nearly overlap; a paired-seed comparison or a significance test would strengthen the improvement claim in this environment.
[Figure 3] The three panels are small and use similar color scales, which makes cross-panel comparison difficult; consider plotting termination improvement, conditional action novelty, and state novelty on a common color scale with larger panels.
[Acknowledgments and Figure 13] There are typos: 'Institute of Information& Coummunications Technology' in the acknowledgments and 'termianted skills' in the Figure 13 caption.
[6.6] The claim that novelty estimation is robust to dataset size is based on visual inspection of box plots (Figure 8, right); report quantitative statistics, such as rank correlations between dataset usage and novelty scale, if this robustness is a stated contribution.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline gains are conditional on per-environment novelty thresholds tuned on downstream performance; the post-hoc percentile rule does not establish a parameter-free task-agnostic termination condition.

fitted input called prediction [Appendix B.3; Appendix A.1; Section 5.1; Figure 5/table]
"In practice, the thresholds we tuned through experiments (see Appendix A.1) approximately correspond to the 97th percentile of the novelty values computed over task-agnostic demonstrations—e.g., kitchen = 0.3 (97.47th percentile), maze = 50 (96.86th percentile), and block stacking = 40 (96.12th percentile)."

The termination labels beta are constructed by thresholding the ICM novelty score (Section 5.1), and the thresholds are explicitly tuned through downstream experiments rather than derived from the task-agnostic data alone. The downstream success-rate claims in Figure 5 and its table are then reported for the same environments using these tuned thresholds; Appendix A.1 shows that alternative thresholds (th0.1, th0.5) lose the clear margin over SPiRL, so the reported improvement is conditional on the fitted threshold value. The 'top 1% prediction error percentile' statement in Appendix I.2 is a post-hoc summary rather than a pre-specified rule, and it even conflicts with the 97th-percentile description in Appendix B.3.

full rationale

The central empirical comparison is not fully circular: after the termination distribution is trained, downstream RL performance is measured on held-out tasks with new maze layouts, larger block-stacking configurations, and unseen kitchen subtask sequences, so the learned skills and the RL gains have independent content. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in solely through the authors' prior work; the termination-improvement theorem is used only as motivation, not as a proof that high ICM error marks decision points. The main circularity concern is the fitted threshold: termination labels are produced by thresholding novelty, the thresholds are tuned using downstream performance (Appendix B.3, Appendix A.1), and the reported headline margins over SPiRL are sensitive to those tuned values. This makes the 'task-agnostic' transfer claim partially fitted, but because the downstream gains themselves are evaluated through fresh RL training on held-out tasks, the circularity is partial rather than a by-construction reduction of the result to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a heuristic mapping from ICM prediction error to valuable decision points, plus two free hyperparameters (novelty threshold and max skill length). The threshold is tuned on downstream performance, which is the main fitted quantity. No new physical entities are introduced.

free parameters (2)

Novelty threshold per environment = kitchen 0.3, maze 50, block stacking 40
Tuned on downstream task performance; ablation (Figure 10a) shows performance drops when the threshold is not chosen appropriately. The 97th percentile description in Appendix B.3 is a post hoc characterization, not a principled selection rule.
Maximum skill length H = 30
Chosen by hand across all tasks; caps skill duration and interacts with the termination distribution. The paper does not analyze sensitivity to H.

assumptions (4)

standard math Termination improvement theorem (Sutton, 1998)
Invoked in Section 4.2 to motivate terminating when V(s) > Q(s,o). It is a known result and is used only as motivation, not as a derivation step.
domain assumption ICM prediction error is a valid estimate of state-action novelty chi(s,a)
Section 4.1 and Section 6.1 assume that squared L2 prediction error in the feature space approximates the inverse joint visitation count. This is a standard novelty proxy but is not proven for the specific environments.
ad hoc to paper High state-action novelty marks decision points where termination improves downstream learning
The central heuristic of the paper (Section 4.1, Figure 1, Figure 2) is that high novelty states are good termination points. Evidence is qualitative and environment-specific; no formal argument connects novelty to termination improvement.
domain assumption Agent-centered cropped images preserve decision-point structure across different environment layouts
Appendix E states that cropped images are necessary for transfer to significantly changed configurations. The assumption is that local structure (crossroads, object interactions) carries over even when the global layout changes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations." pith.science (2026). https://pith.science/paper/OCCDO2XT

@misc{pith2026250112668,
  author       = {Pith},
  title        = {Pith review of: NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCCDO2XT}},
  note         = {Machine review of arXiv:2501.12668}
}

read the original abstract

Intelligent agents are able to make decisions based on different levels of granularity and duration. Recent advances in skill learning enabled the agent to solve complex, long-horizon tasks by effectively guiding the agent in choosing appropriate skills. However, the practice of using fixed-length skills can easily result in skipping valuable decision points, which ultimately limits the potential for further exploration and faster policy learning. In this work, we propose to learn a simple and effective termination condition that identifies decision points through a state-action novelty module that leverages agent experience data. Our approach, Novelty-based Decision Point Identification (NBDI), outperforms previous baselines in complex, long-horizon tasks, and remains effective even in the presence of significant variations in the environment configurations of downstream tasks, highlighting the importance of decision point identification in skill learning.

Figures

Figures reproduced from arXiv: 2501.12668 by the authors.

**Figure 1.** Visualization of an example of critical decision points in the kitchen environment. High state-action novelty can be found in states where a subtask has been completed, and multiple subsequent subtasks are accessible. If termination occurs at a high state-action novelty point, the agent retains multiple plausible options. For example, after moving the kettle, it may choose to open the sliding cabinet or turn the ove… view at source ↗

**Figure 2.** Visualization of prediction error of ICM in maze and block stacking environment. Note the same offline data that is used to train ICM was used to compute this prediction error. (A), (B) and (C) are the state-action pairs with the highest prediction error, while (D), (E) and (F) are the ones with the lowest. Critical decision points—such as crossroads or states involving block manipulation—are typically associated wi… view at source ↗

**Figure 3.** The relative frequency of termination improvement occurrences (left), conditional action novelty (middle), and state novelty (right) in a small grid with three different goals. Higher percentile colors indicate a relatively greater number of termination improvement occurrences, higher conditional action novelty, and higher state novelty. Further details on the visualization procedure are provided in Appendix I.1.… view at source ↗

Figures from the paper (14 more)

**Figure 5.** Figure 5: Performances of our method and baselines in solving downstream tasks. The shaded region represents 95% confidence interval across five different seeds. The last column of the table below illustrates the percentage improvement of our method over SPiRL. which is an exten…

**Figure 6.** Figure 6: Performances of our method and baselines in solving downstream tasks. The shaded region and error bar represents 95% confidence interval across five different seeds. Thus, our findings validate that ICM can serve as an effective state-action novelty estimator when pre…

**Figure 7.** Figure 7: Visualization of decision points made by SPiRL and NBDI in the maze environment. We sampled 100 trajectories for each trained policy to observe the points at which they make decisions. Higher percentile colors suggest a relatively greater number of visitation frequenci…

**Figure 8.** Figure 8: Illustration of the impact of varying the width and depth configuration of hidden layer and dataset usage on state-action novelty estimation. (Left) shows how varying hidden layer configurations (depth and width) influences the estimated state-action novelty (box plot…

**Figure 9.** Figure 9: Comparison of success rates in the maze environment during the skill-based meta-training phase of SiMPL across 10 meta-training tasks (10 random goal locations). The shaded regions indicate standard errors over three random seeds. 7, SPiRL fails to reach the goal, whil…

**Figure 10.** Figure 10: Ablation in variable-length skills (a), no termination distribution (b), and criteria to determine decision points (c). The shaded region represents 95% confidence interval across five different seeds. A.1. Ablation in NBDI Figure 10a compares the performance of our m…

**Figure 11.** Figure 11: A bad example of decision point in the maze environment. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

**Figure 12.** Figure 12: Performances of our method and baselines in solving downstream tasks. The shaded region represents 95% confidence interval across five different seeds. A.4. Comparison to SPiRL with Fixed Average Skill Length of NBDI [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

**Figure 13.** Figure 13: Novelty-based Decision Point Identification (NBDI), has two main procedures: (i) novelty learning and skill extraction: training the state-action novelty model and learning the skill prior, skill embedding space and termination distribution with the pre-trained novelt…

**Figure 14.** Figure 14: Visualization of visitations and decision points made by SPiRL and NBDI in the maze environment (top: trained with stochastic BC (σ = 0.5) dataset, bottom: trained with stochastic BC (σ = 0.75) dataset). We sampled 100 trajectories for each trained policy to observe t…

**Figure 15.** Figure 15: Visualization of prediction error of ICM in maze environment (top: trained with stochastic BC (σ = 0.5) dataset, middle: trained with stochastic BC (σ = 0.75) dataset, bottom: trained with random walk dataset). Note the same offline data that is used to train ICM was …

**Figure 16.** Figure 16: Visualization of critical decision points in MuJoCo (Todorov et al., 2012) environment (top-left: halfcheetah-medium-expert, top-right: halfcheetah-medium-replay, bottom-left: ant-medium-expert, bottom-right: ant-medium-replay) We investigated whether meaningful decis…

**Figure 17.** Figure 17: Examples of maze layouts used for training (left) and downstream task (right). The agent starts at the white point and aims to reach the red point. F. Task-Agnostic Dataset Size and Coverage The number of task-agnostic trajectories collected for each environment were …

**Figure 18.** Figure 18: Visualization of skill termination points in a grid-based maze environment using NBDI (Conditional action novelty, State novelty), Relative Novelty, and LOVE (Minimum Description Length). Darker shades represent states with a relatively higher frequency of termination…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 canonical work pages

[1]

Variational option discovery algorithms

Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299,

arXiv
[4]

Reinforcement Learning with NBDI In downstream learning, our objective is to learn a skill policy πθ(z|st) that maximizes the expected sum of discounted rewards, parameterized by θ

CHANNELS 8, 16, 32 # CONVOLUTION LAYERS 3 SKILL DECODER HIDDEN DIM 128 # HIDDEN LAYERS 6 15 A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations B.2. Reinforcement Learning with NBDI In downstream learning, our objective is to learn a skill policy πθ(z|st) that maximizes the expected sum of discounted rewards...

work page 2018
[6]

Hierarchical few-shot imitation with skill tran- sition models

Hakhamaneshi, K., Zhao, R., Zhan, A., Abbeel, P., and Laskin, M. Hierarchical few-shot imitation with skill tran- sition models. arXiv preprint arXiv:2107.08981,

arXiv
[8]

D., Saeedi, A., Gautam, S., and Gershman, S

Kulkarni, T. D., Saeedi, A., Gautam, S., and Gershman, S. J. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396,

arXiv
[16]

Parrot: Data-driven behavioral priors for re- inforcement learning

Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and Levine, S. Parrot: Data-driven behavioral priors for re- inforcement learning. arXiv preprint arXiv:2011.10024,

arXiv 2011
[17]

The agent starts at the white point and aims to reach the red point

Examples of maze layouts used for training (left) and downstream task (right). The agent starts at the white point and aims to reach the red point. F. Task-Agnostic Dataset Size and Coverage The number of task-agnostic trajectories collected for each environment were as follows: maze (85,000 trajectories), block stacking (37,000 trajectories), and kitchen...

work page 2004
[23]

Note that task and environment setup differ between the training data and the downstream task, demonstrating the model’s capacity to handle unseen downstream tasks

(Kitchen) (CC BY 4.0). Note that task and environment setup differ between the training data and the downstream task, demonstrating the model’s capacity to handle unseen downstream tasks. Kitchen Environment The kitchen environment is provided by the D4RL benchmark (Fu et al., 2021), featuring seven manipulable objects. The training trajectories consist o...

work page 2021
[24]

In the maze and sparse block stacking environment, a single fully-connected layer with hidden dimension 52 and 70 have been used, respectively

In the kitchen environment, the structure of the dynamic model is the same as its feature encoder. In the maze and sparse block stacking environment, a single fully-connected layer with hidden dimension 52 and 70 have been used, respectively. The state-action novelty χ(s, a) is computed as the squared L2 distance between ˆϕ(ϕ(st), at) and ϕ(st+1), represe...

work page 2021

Show all 24 references

[1993]

Offline reinforce- ment learning with implicit q-learning

Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2110.06169,

arXiv
[1994]

Ex- ploration by random network distillation

Burda, Y ., Edwards, H., Storkey, A., and Klimov, O. Ex- ploration by random network distillation. arXiv preprint arXiv:1810.12894,

arXiv
[1997]

encoder qϕ(z|τ, β) and a decoder pψ(at, βt|z, st). To learn model parameters ϕ and ψ, the latent variable model receives a randomly sampled experience τ from the training dataset D along with a termination condition vector β from the state-action novelty module, and tries to r...

2016
[1998]

Mujoco: A physics engine for model-based control

Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp. 5026–5033. IEEE,

2012
[2001]

Q-cut—dynamic discovery of sub-goals in reinforcement learning

Menache, I., Mannor, S., and Shimkin, N. Q-cut—dynamic discovery of sub-goals in reinforcement learning. In Ma- chine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13, pp. 295–306. Springer,

2002
[2002]

J., and Lim, J

Nam, T., Sun, S.-H., Pertsch, K., Hwang, S. J., and Lim, J. J. Skill-based meta-reinforcement learning. arXiv preprint arXiv:2204.11828,

arXiv
[2003]

J., and Wierstra, D

Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control. arXiv preprint arXiv:1611.07507,

arXiv
[2004]

P., and Barto, A

Simsek, ¨O., Wolfe, A. P., and Barto, A. G. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Raedt, L. D. and Wrobel, S. (eds.), Ma- chine Learning, Proceedings of the Twenty-Second In- ternational Conference (ICML 2005), Bonn, Germany, Aug...

2005
[2005]

URL https://doi

doi: 10.1145/1102351.1102454. URL https://doi. org/10.1145/1102351.1102454. S ¸ims ¸ek, ¨O., Wolfe, A. P., and Barto, A. G. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the 22nd international conference on Machine learnin...
[2012]

Trail: Near-optimal imitation learning with suboptimal data

Yang, M., Levine, S., and Nachum, O. Trail: Near-optimal imitation learning with suboptimal data. arXiv preprint arXiv:2110.14770,

arXiv
[2016]

C., Bellemare, M

Machado, M. C., Bellemare, M. G., and Bowling, M. A laplacian framework for option discovery in reinforce- ment learning. In International Conference on Machine Learning, pp. 2295–2304. PMLR, 2017a. Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., and Campbell, M....

arXiv
[2018]

Opal: Offline primitive discovery for acceler- ating offline reinforcement learning

Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for acceler- ating offline reinforcement learning. arXiv preprint arXiv:2010.13611,

2010 arXiv
[2019]

Accelerating reinforcement learning with learned skill priors

Pertsch, K., Lee, Y ., and Lim, J. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188–204. PMLR, 2021a. Pertsch, K., Lee, Y ., Wu, Y ., and Lim, J. J. Guided rein- forcement learning with learned skills. arXiv preprint arXiv...

arXiv
[2020]

Option discovery in hierarchical reinforcement learning using spatio-temporal clustering

Srinivas, A., Krishnamurthy, R., Kumar, P., and Ravin- dran, B. Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv:1605.05359,

arXiv
[2021]

Investigating the effectiveness of bpe: The power of shorter sequences

Gall´e, M. Investigating the effectiveness of bpe: The power of shorter sequences. In Proceedings of the 2019 confer- ence on empirical methods in natural language process- ing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 1375–1381,

2019
[2022]

Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D

URL https://arxiv.org/ abs/2209.01947. Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583–8592. PMLR,

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Variational option discovery algorithms

Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299,

arXiv

[2] [4]

Reinforcement Learning with NBDI In downstream learning, our objective is to learn a skill policy πθ(z|st) that maximizes the expected sum of discounted rewards, parameterized by θ

CHANNELS 8, 16, 32 # CONVOLUTION LAYERS 3 SKILL DECODER HIDDEN DIM 128 # HIDDEN LAYERS 6 15 A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations B.2. Reinforcement Learning with NBDI In downstream learning, our objective is to learn a skill policy πθ(z|st) that maximizes the expected sum of discounted rewards...

work page 2018

[3] [6]

Hierarchical few-shot imitation with skill tran- sition models

Hakhamaneshi, K., Zhao, R., Zhan, A., Abbeel, P., and Laskin, M. Hierarchical few-shot imitation with skill tran- sition models. arXiv preprint arXiv:2107.08981,

arXiv

[4] [8]

D., Saeedi, A., Gautam, S., and Gershman, S

Kulkarni, T. D., Saeedi, A., Gautam, S., and Gershman, S. J. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396,

arXiv

[5] [16]

Parrot: Data-driven behavioral priors for re- inforcement learning

Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and Levine, S. Parrot: Data-driven behavioral priors for re- inforcement learning. arXiv preprint arXiv:2011.10024,

arXiv 2011

[6] [17]

The agent starts at the white point and aims to reach the red point

Examples of maze layouts used for training (left) and downstream task (right). The agent starts at the white point and aims to reach the red point. F. Task-Agnostic Dataset Size and Coverage The number of task-agnostic trajectories collected for each environment were as follows: maze (85,000 trajectories), block stacking (37,000 trajectories), and kitchen...

work page 2004

[7] [23]

Note that task and environment setup differ between the training data and the downstream task, demonstrating the model’s capacity to handle unseen downstream tasks

(Kitchen) (CC BY 4.0). Note that task and environment setup differ between the training data and the downstream task, demonstrating the model’s capacity to handle unseen downstream tasks. Kitchen Environment The kitchen environment is provided by the D4RL benchmark (Fu et al., 2021), featuring seven manipulable objects. The training trajectories consist o...

work page 2021

[8] [24]

In the maze and sparse block stacking environment, a single fully-connected layer with hidden dimension 52 and 70 have been used, respectively

In the kitchen environment, the structure of the dynamic model is the same as its feature encoder. In the maze and sparse block stacking environment, a single fully-connected layer with hidden dimension 52 and 70 have been used, respectively. The state-action novelty χ(s, a) is computed as the squared L2 distance between ˆϕ(ϕ(st), at) and ϕ(st+1), represe...

work page 2021