Pith. sign in

REVIEW 3 major objections 4 minor 69 references

MaestroMotif: Skill Design from Artificial Intelligence Feedback

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MaestroMotif claims that describing skills in natural language is enough to train an agent that solves hard NetHack tasks zero-shot, outperforming task-specific RL baselines.

desk verdict A credible, useful integration of LLM preference rewards with code-policy composition that reports large NLE margins, but the central attribution would be stronger with a control using untrained skills. read the letter →

arxiv 2412.08542 v1 pith:YO3A3KY6 submitted 2024-12-11 cs.AI cs.CLcs.LG

Martin Klissarov , Mikael Henaff , Roberta Raileanu , Shagun Sodhani , Pascal Vincent , Amy Zhang , Pierre-Luc Bacon , Doina Precup
show 2 more authors
Marlos C. Machado Pierluca D'Oro
This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords AI-assistedskilldesignreinforcementlearningfromAIfeedbackLLMcodegenerationNetHackEnvironmentoptionsframeworkzero-shotcompositionpreference-basedrewardhierarchical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MaestroMotif aims to show that the skill design problem, converting a high-level description of a behavior into a usable low-level policy, can be automated almost entirely by a large language model, with a human providing only natural-language descriptions. The paper claims that LLM preferences over short state-difference summaries are enough to train reward functions for individual skills, and that LLM-generated Python code can specify when skills start and stop and how they are sequenced, both during training and at test time. On the NetHack Learning Environment, this combination solves navigation, interaction, and composite tasks zero-shot at rates (46% on Gnomish Mines, 24.8% on Golden Exit) that task-specific RL baselines and prompt-only LLM policies essentially fail to reach. If correct, this means a natural-language description plus unannotated interaction data is sufficient to build a reusable skill hierarchy without hand-designed rewards or task-specific training.

What carries the argument

The load-bearing object is the option triple (initiation function, intra-option policy, termination function) in the call-and-return style of the options framework, together with a code-generated policy over skills. MaestroMotif obtains each component automatically: rewards from a Bradley-Terry model fitted to LLM preferences (the Motif method), initiation and termination functions and the training-time sequencing policy as LLM-written Python, and a single neural network conditioned on a one-hot skill vector as the shared intra-option policy. The design point is that the LLM never chooses low-level actions; it only supplies rewards and code for when to enter, exit, and switch skills. A notable secondary mechanism is the use of 100-step state differences as a compressed, non-Markovian representation for preference annotation, which the ablation study shows is necessary for the skill rewards to capture the intended behaviors.

What would settle it

Run the full MaestroMotif pipeline with the same code policies and skill descriptions, but with the LLM preference labels randomly flipped with probability 0.5 during reward training; if task success rates on Gnomish Mines and Golden Exit remain around 46% and 24.8%, the claim that LLM feedback is what carries skill learning would be falsified.

Watch

Extended reading notes

Core claim

MaestroMotif claims to be the first method that, using only natural-language skill descriptions and unannotated interaction data, produces an agent that solves end-to-end tasks in the NetHack Learning Environment. The pipeline distills LLM pairwise preferences over 100-step state differences into a per-skill reward, uses LLM-written code for initiation and termination functions and for a training-time sequencing policy, trains a shared skill-conditioned network with PPO, and then at deployment lets an LLM write a new code policy that recombines the learned skills without further training. The reported results are success rates of 46% on Gnomish Mines, 29% on Delphi, 7.2% on Minetown, and 24.8% on the Golden Exit composite task, with interaction tasks yielding 0.66 transactions, 0.47 price identifications, and 1.60 bless/cursed identifications per episode, numbers that task-specific RL baselines, prompt-based LLM policies, and even an RL agent trained with privileged reward information do not reach. The authors interpret this as evidence that the binding constraint on such tasks is decomposition into skills, and that LLM-generated code supplies that decomposition.

Load-bearing premise

The load-bearing premise is that the pairwise preferences an LLM expresses over compressed 100-step state differences are reliable enough to train reward functions that induce the intended skill behaviors, and if those preferences are misaligned or the compression discards essential non-Markovian state such as hunger, every downstream skill and composition fails no matter how good the code policy is.

Editorial extensions

If this is right

  • Language-only skill design suffices: with MaestroMotif, a human provides skill descriptions and the pipeline produces policies for navigation, interaction, and composite NetHack tasks without task-specific reward engineering.
  • Deployment-time adaptation becomes a code-writing problem: new task specifications are turned into a policy over skills at inference time, so no additional gradient training is needed to handle a new task from the benchmark.
  • Simultaneous training under a code-defined sequencing policy induces an emergent curriculum, and ablations show both the skill-conditioned architecture and this curriculum are required for the reported success rates.
  • The reported scaling trend for the code-generating LLM implies that improvements in code generation quality directly improve zero-shot task performance.
  • Score-maximizing agents, including behavioral cloning baselines, fail the benchmark tasks almost entirely, indicating that game score is not a proxy for the landmark behaviors MaestroMotif targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same recipe, LLM preference rewards plus code-defined skill sequencing, should transfer to other partially observable environments with rich text-like observations, because the protocol only needs a serializable state difference and a reliable annotator.
  • Editorial inference: if the learned skill rewards are behaviorally aligned with their descriptions, the skill policies become reusable primitives for a learned high-level planner, a path the paper only hints at in its discussion of online adaptation.
  • Editorial inference: the method's ceiling is set by LLM quality; a testable extension would be to swap the annotator for a smaller or open-weight model and measure the preference-reliability threshold at which the skill hierarchy loses its edge over per-task RL.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MaestroMotif, a pipeline for AI-assisted skill design in the NetHack Learning Environment (NLE). The method takes natural-language skill descriptions, uses an LLM to produce pairwise preferences over unannotated interaction data, distills those preferences into per-skill reward functions (after Motif), uses an LLM coder to generate initiation/termination functions and a training-time policy over skills, and trains a shared skill-conditioned policy with PPO. At deployment, another LLM-generated code policy recombines the trained skills zero-shot. The authors introduce an evaluation suite of navigation, interaction, and composite tasks and report large margins over existing methods, including task-specific RL baselines. They also present ablations of the skill architecture, synchronous versus isolated skill learning, code-refinement, the scale of the code-generating LLM, and the preference representation.

Significance. If the central claim holds, MaestroMotif is a significant step: it suggests that natural-language descriptions plus unannotated data and LLM feedback can replace both hand-engineered reward design and manual low-level controller programming in a complex, partially observable domain such as NetHack. The paper is honestly written and contains useful ablations, an attempt at equivalent-prompting baselines, and detailed appendix prompts and code examples. In particular, the result that giving all the MaestroMotif prompts to the LLM-Policy baseline does not close the gap (Appendix A.11, Table 3) is valuable evidence for the hierarchical structure. However, the headline results depend on an attribution that the current experiments do not fully establish: the contribution of the trained skill policies is never isolated from the LLM-generated code policy and the hand-specified environment priors embedded in the deployment prompts. I do not see a circularity problem, because the success metrics are external NetHack landmarks and not quantities fitted by the method, but the missing control is load-bearing for the central 'skill design from AI feedback' claim.

major comments (3)
  1. [Section 4.3 and Appendix A.11] The ablations in Section 4.3 and Appendix A.11 vary the skill architecture (goal-conditioned versus multi-head), the training strategy (simultaneous versus in isolation), and the amount of prompting information, but every MaestroMotif configuration retains the five RL-trained skill policies. The deployment policy shown in Prompt 7 and Output 4 contains substantial hand-specified environment knowledge: the branch_depth parameter, the fact that the Gnomish Mines branch appears at branch_depth+1, and the purpose-built helper methods reach_gnomish_mines and reach_dungeons_of_doom. A policy-over-skills state machine of this kind, operating over even a weak low-level movement/attack controller, could in principle produce much of the navigation and composite-task success on its own. To support the claim that reward-from-LLM-feedback skill training is the driver of the results in Tables 1 and 2, the paper needs a control in which the same generated code policy is paired with randomly initialized or untrained skill policies (or with a simple hand-coded low-level controller). Without this control, the improvement over the Motif and LLM-Policy baselines could be attributable to the code policy and the environment priors rather than to the learned skills.
  2. [Appendix A.5] The environment is modified so that the agent can eat and quaff any item in its inventory by pressing the 'y' key, and this modification is directly load-bearing for the composite task Discovery Hunger, whose specification in Table 2 requires the agent to eat food after becoming hungry. The paper states that all trained agents are evaluated under these same conditions, but it does not report whether the modification changes the difficulty of the tasks or whether the baselines can effectively exploit the new action. Since the paper claims to evaluate on the NetHack Learning Environment, the main text should clearly state that a modified environment is used, and the paper should ideally report results on both the original and modified environments, or at least quantify the effect of this modification on the reported success rates.
  3. [Section 3.3 and Equation (2)] The LLM's pairwise preferences over 100-step state differences are treated as ground-truth labels for reward training, but no validation is provided that the labels agree with human judgments or with the intended semantics of the skill descriptions. The ablations in Figure 11 show sensitivity to the representation of the state difference (diffStats, player statistics, human gameplay data), but they do not test the correctness of the preference labels themselves. Because every skill policy is optimized against rewards distilled from these labels, a mismatch between the LLM's preferences and the intended skills would invalidate the 'AI-assisted skill design' claim even if the code policy performed well. At a minimum, the paper should report agreement statistics between the LLM annotator and human annotators on a sample of preference pairs, or ablate the reward source by comparing against a reward trained on a different LLM or on human preferences.
minor comments (4)
  1. [Table 1] The column 'RL w/ task reward + score' mixes units: for the navigation tasks it reports percentages, while for Transactions, Price Identified, and BUC Identified it appears to report item counts, but one entry reads '0.01%' even though these are count-based metrics; the table should separate units or use a per-row notation.
  2. [Figure 8] The legend entries 'Goal-Cond. & Simultaneously-Learned' and related abbreviations are not expanded in the caption; the caption should state that 'Goal-Cond.' means goal-conditioned policy and define 'Simultaneously-Learned' versus 'Learned In Isolation'.
  3. [Output 4] In the illustrative code for Discovery Hunger, the perform_task method sets self.eaten_food = True immediately upon entering the Gnomish Mines, without checking that the agent has actually eaten food or that it is hungry; if this example is representative, the deployment code may not faithfully implement the task specification, and the final code after refinement should be shown or verified against the external success metric.
  4. [Section A.5] The sentence 'letting the agent eat and quaff any of its items, at random, by performing a particular command' is slightly ambiguous: it should clarify whether the item is chosen randomly by the environment or by the agent, since this affects the controllability of the eat/quaff action.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: learned rewards are fit to external LLM preference data, and the headline numbers are measured against independent NetHack landmarks.

full rationale

MaestroMotif's derivation chain is: (1) elicit LLM preferences over pairs of unannotated observations using skill-specific prompts; (2) distill those preferences into per-skill reward functions via Eq. 2; (3) train skill policies by RL to maximize those rewards; (4) generate a code policy over skills for each deployment task; (5) measure success on fixed NetHack landmarks such as Gnomish Mines, Delphi, Minetown, transactions, BUC identification, and composite tasks. None of the evaluation quantities is a parameter fitted by the method. The reward functions are learned from preference annotations, not from the task success labels, and the tasks are defined from the NLE paper and the NetHack community, as described in Appendix A.6. The relation between reward and evaluation is a competence test of a specified skill, not an equation-level equivalence. Self-citations to Motif (Klissarov et al., 2024) are used to justify prompts and reward transformations, but the paper re-implements the protocol, reports learning curves in Figure 8, and evaluates Motif as a separate baseline, so the central zero-shot claim does not reduce to the citation. Appendix A.11 additionally controls for information leakage by giving all MaestroMotif prompts and code to flat baselines, with no performance gain. The main residual concern is attribution: no control removes the LLM-generated code policy to isolate the contribution of the trained skill policies, and Appendix A.5's environment modification enables the eat and quaff subtasks. That is a missing-control and correctness issue, not a circularity under the definitional or fitted-input criteria.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the reliability of LLM preference labels, the correctness of LLM-generated code, the options framework, and a set of hand-chosen hyperparameters. No new physical entities are introduced.

free parameters (5)
  • Per-skill reward threshold quantile = 85th percentile; 95th percentile for Discoverer
    In the reward transformation (Eq. 3), rewards below a threshold are zeroed; the threshold is chosen per skill from the empirical reward distribution, altering which states produce learning signal.
  • Episodic count exponent beta = 3
    Eq. 3 divides reward by N(observation)^beta; beta=3 is manually selected and changes reward shaping and exploration pressure.
  • Reward scale = 0.1
    Listed as a PPO hyperparameter; scales the learned reward and affects policy optimization.
  • State difference history length = 100 time steps
    LLM annotator is shown the difference between states 100 steps apart; this choice defines the non-Markovian context for preference labels.
  • RL training hyperparameters (PPO clip, batch size, workers) = see Appendix A.5 Table 2
    Standard but hand chosen; not unique to the method, but needed to reproduce results.
assumptions (6)
  • domain assumption Llama 3.1 70B preference annotations are treated as ground truth for reward learning.
    Eq. 2 optimizes a Bradley-Terry model against LLM-produced labels; when the LLM is wrong, the reward functions encode the wrong behavior.
  • domain assumption The 100-step state differences plus player statistics provide sufficient information for the LLM to judge skill-relevant preferences.
    Section 3.3 uses diff histories to compress observations; assumes no critical information is lost for reward design.
  • domain assumption The LLM-generated Python code for skill initiation, termination, and selection is correct and safe to execute.
    Zero-shot deployment runs this code directly; unit-test refinement reduces but does not eliminate bugs.
  • standard math Options framework and call-and-return semantics are valid for composing the learned skills.
    Section 2 defines skills as options; assumes convergence and correct composition under call-and-return.
  • domain assumption The manually selected skill set is sufficient for the evaluation suite.
    The five skills define the expressible behavior space; the paper acknowledges this limitation in Section 6.
  • ad hoc to paper The modified NetHack environment (random eat/quaff on 'y') preserves comparability with prior work.
    Appendix A.5 changes the action set for all compared agents; this is not standard NLE and may affect baseline difficulty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MaestroMotif: Skill Design from Artificial Intelligence Feedback." pith.science (2026). https://pith.science/paper/YO3A3KY6

@misc{pith2026241208542,
  author       = {Pith},
  title        = {Pith review of: MaestroMotif: Skill Design from Artificial Intelligence Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YO3A3KY6}},
  note         = {Machine review of arXiv:2412.08542}
}
read the original abstract

Describing skills in natural language has the potential to provide an accessible way to inject human knowledge about decision-making into an AI system. We present MaestroMotif, a method for AI-assisted skill design, which yields high-performing and adaptable agents. MaestroMotif leverages the capabilities of Large Language Models (LLMs) to effectively create and reuse skills. It first uses an LLM's feedback to automatically design rewards corresponding to each skill, starting from their natural language description. Then, it employs an LLM's code generation abilities, together with reinforcement learning, for training the skills and combining them to implement complex behaviors specified in language. We evaluate MaestroMotif using a suite of complex tasks in the NetHack Learning Environment (NLE), demonstrating that it surpasses existing approaches in both performance and usability.

Figures

Figures reproduced from arXiv: 2412.08542 by the authors.

Figure 1
Figure 1. Performance across NLE task categories. MaestroMotif largely outper￾forms existing methods zero-shot, includ￾ing the ones trained on each task. Alice is framing her language description of how to be a versatile researcher as the description of a set of skills. This often happens among people, since this type of description is a convenient way to exchange information on how to become proficient in a given domain. Ali… view at source ↗
Figure 2
Figure 2. AI-assisted Skill Design with MaestroMotif. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Generation of policy over skills during deployment. The LLM takes a task description and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Simplified depiction of the early NetHack game where [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance of MaestroMotif and score-maximizing baselines in interaction tasks (first row) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Illustration of MaestroMotif on the composite task [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: (a) Goal-conditioning as an architecture for skill selection and synchronous alternation of the skills using an exploration policy is essential for obtaining good performance. (b) When learning skills asynchronously (alternating them in different episodes), some import…
Figure 7
Figure 7. Figure 7: Impact of scaling for the LLM code generator on final per￾formance across tasks. Hierarchical architecture As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 9
Figure 9. Figure 9: Refining the LLM Coder outputs through a self-generated unit tests yields significant [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Neural network architectures. The architecture on left, used throughout the paper, was key [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Ablation studies on MaestroMotif’s design choices. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 43 canonical work pages

  1. [1]

    Do as i can, not as i say: Grounding language in robotic affordances, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, K...

    arXiv 2022

  2. [2]

    Modular multitask reinforcement learning with policy sketches

    Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research, pp.\ 166--175. PMLR , 2017

    work page 2017

  3. [3]

    Senthil, and George Dimitri Konidaris

    Akhil Bagaria, Jason K. Senthil, and George Dimitri Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, 2021. URL https://api.semanticscholar.org/CorpusID:221094080

    work page 2021

  4. [4]

    Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and R \'e mi Munos

    Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and R \'e mi Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Processing Systems, 2016. URL https://api.semanticscholar.org/CorpusID:8310565

    work page 2016

  5. [5]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324, 1952. URL https://api.semanticscholar.org/CorpusID:125209808

    1952

  6. [6]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL https://api.semanticscholar.org/CorpusID:4787508

    arXiv 2017

  7. [7]

    \" O zg\" u r S im s ek and Andrew G. Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning, ICML '04, pp.\ 95, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi:10.1145/1015330.1015353. URL https://d...

    arXiv 2004

  8. [8]

    Active reward learning

    Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Robotics: Science and Systems, 2014. URL https://api.semanticscholar.org/CorpusID:16043466

    work page 2014

Show all 69 references
  1. [9]

    Feudal reinforcement learning

    Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pp.\ 271--278, 1993

    1993

  2. [10]

    Drescher

    Gary L. Drescher. Made-up minds - a constructivist approach to artificial intelligence. 1991. URL https://api.semanticscholar.org/CorpusID:3099707

    1991

  3. [11]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint...

    2023 arXiv

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. URL https://api.semanticscholar.org/CorpusID:271571434

    2024 arXiv

  5. [13]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Sy...

    2022

  6. [14]

    Hart, and Nils J

    Richard Fikes, Peter E. Hart, and Nils J. Nilsson. Learning and executing generalized robot plans. Artif. Intell., 3: 0 251--288, 1993. URL https://api.semanticscholar.org/CorpusID:17260619

    1993

  7. [15]

    Variational intrinsic control

    Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/foru...

    2017

  8. [16]

    u ttler, Taehwon Kwon, Donghoon Lee, Vegard Mella, Nantas Nardelli, Ivan Nazarov, Nikita Ovsov, Jack Holder, Roberta Raileanu, Karolis Ramanauskas, Tim Rockt \

    Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Minqi Jiang, Jo Daejin, Anssi Kanervisto, Jongmin Kim, Sungwoong Kim, Robert Kirk, Vitaly Kurin, Heinrich K \"u ttler, Taehwon Kwon, Donghoon Lee, Vegard Mella, Nantas Nardelli,...

    2021

  9. [17]

    Dungeons and data: A large-scale nethack dataset

    Eric Hambro, Roberta Raileanu, Danielle Rothermel, Vegard Mella, Tim Rockt \"a schel, Heinrich Kuttler, and Naila Murray. Dungeons and data: A large-scale nethack dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022 b...

    2022

  10. [18]

    When waiting is not an option : Learning options with a deliberation cost

    Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option : Learning options with a deliberation cost. ArXiv, abs/1709.04571, 2017. URL https://api.semanticscholar.org/CorpusID:19247295

    2017 arXiv

  11. [19]

    Exploration via elliptical episodic bonuses

    Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rockt \" a schel. Exploration via elliptical episodic bonuses. In NeurIPS, 2022. URL http://papers.nips.cc/paper\_files/paper/2022/hash/f4f79698d48bdc1a6dec20583724182b-Abstract-Conference.html

    2022

  12. [20]

    C. A. R. Hoare. An axiomatic basis for computer programming. Commun. ACM, 26: 0 53--56, 1969. URL https://api.semanticscholar.org/CorpusID:6059550

    1969

  13. [21]

    Open-endedness is essential for artificial superhuman intelligence, 06 2024

    Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. Open-endedness is essential for artificial superhuman intelligence, 06 2024

    2024

  14. [22]

    Glenn A. Iba. A heuristic approach to the discovery of macro-operators. Machine Learning, 3: 0 285--317, 1989. URL https://api.semanticscholar.org/CorpusID:13649095

    1989

  15. [23]

    Options of interest: Temporal abstraction with interest functions

    Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert, Pierre-Luc Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 4444--4451, 2020

    2020

  16. [24]

    Martin Klissarov and Marlos C. Machado. Deep laplacian-based options for temporally-extended exploration. In International Conference on Machine Learning, 2023. URL https://api.semanticscholar.org/CorpusID:256274634

    2023

  17. [25]

    Motif: Intrinsic motivation from artificial intelligence feedback

    Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. In The Twelfth International Conference on Learning Representations, 2024. URL...

    2024

  18. [26]

    Keep your options open: An information-based driving principle for sensorimotor systems

    Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Keep your options open: An information-based driving principle for sensorimotor systems. PloS one, 3 0 (12): 0 e4018, 2008

    2008

  19. [27]

    Interactively shaping agents via human reinforcement: The tamer framework

    W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pp.\ 9--16, 2009

    2009

  20. [28]

    From skills to symbols: Learning symbolic representations for abstract high-level planning

    George Dimitri Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. J. Artif. Intell. Res., 61: 0 215--289, 2018. URL https://api.semanticscholar.org/CorpusID:31918172

    2018

  21. [29]

    Practice makes perfect: Planning to learn skill parameter policies

    Nishanth Kumar, Tom Silver, Willie McClinton, Linfeng Zhao, Stephen Proulx, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Jennifer Barry. Practice makes perfect: Planning to learn skill parameter policies. In Robotics: Science and Systems (RSS), 2024

    2024

  22. [30]

    u ttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rockt \

    Heinrich K \" u ttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rockt \" a schel. The NetHack Learning Environment . In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020

    2020

  23. [31]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  24. [32]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023 , ...

    2023

  25. [33]

    Liu, Yuqian Jiang, Xiaohan Zhang, Qian Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone

    B. Liu, Yuqian Jiang, Xiaohan Zhang, Qian Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+p: Empowering large language models with optimal planning proficiency. ArXiv, abs/2304.11477, 2023. URL https://api.semanticscholar.org/CorpusID:258298051

    2023 arXiv

  26. [34]

    Goal-conditioned reinforcement learning: Problems and solutions

    Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In Luc De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pp.\ 5...

    2022

  27. [35]

    A laplacian framework for option discovery in reinforcement learning

    Marlos C Machado, Marc G Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, pp.\ 2295--2304. PMLR, 2017

    2017

  28. [36]

    Machado, Andr \'e Barreto, and Doina Precup

    Marlos C. Machado, Andr \'e Barreto, and Doina Precup. Temporal abstraction in reinforcement learning with the successor representation. J. Mach. Learn. Res., 24: 0 80:1--80:69, 2023. URL https://api.semanticscholar.org/CorpusID:238634579

    2023

  29. [37]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with s...

    2023 arXiv

  30. [38]

    Concurrent hierarchical reinforcement learning

    Bhaskara Marthi, Stuart Russell, David Latham, and Carlos Guestrin. Concurrent hierarchical reinforcement learning. In Leslie Pack Kaelbling and Alessandro Saffiotti (eds.), IJCAI-05, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edin...

    2005

  31. [39]

    Howe, Craig A

    Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. URL https://api.semanticscholar.org/CorpusID:59656859

    1998

  32. [40]

    Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In International Conference on Machine Learning, 2001. URL https://api.semanticscholar.org/CorpusID:1223826

    2001

  33. [41]

    Q-cut - dynamic discovery of sub-goals in reinforcement learning

    Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut - dynamic discovery of sub-goals in reinforcement learning. In European Conference on Machine Learning, 2002. URL https://api.semanticscholar.org/CorpusID:7830103

    2002

  34. [42]

    Embodied lifelong learning for task and motion planning

    Jorge Mendez-Mendez, Leslie Pack Kaelbling, and Tom \'a s Lozano-P \'e rez. Embodied lifelong learning for task and motion planning. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=ZFjgfJb_5c

    2023

  35. [43]

    nle-sample-factory-baseline, 2022

    Miffyli. nle-sample-factory-baseline, 2022. URL https://github.com/Miffyli/nle-sample-factory-baseline. GitHub repository

    2022

  36. [44]

    Nethack: an illustrated guide to the mazes of menace, dec 2022

    Dion Moult. Nethack: an illustrated guide to the mazes of menace, dec 2022. URL https://thinkmoult.com/nethack-illustrated-guide-mazes-of-menace.html. Accessed: 2024-09-02

    2022

  37. [45]

    Courville

    Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 2017. URL https://api.semanticscholar.org/CorpusID:19119291

    2017

  38. [46]

    Sukhatme, and Vladlen Koltun

    Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 1...

    2020

  39. [47]

    diff history for neural language agents, 2023 a

    Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. diff history for neural language agents, 2023 a

    2023

  40. [48]

    Nethack is hard to hack, 2023 b

    Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. Nethack is hard to hack, 2023 b

    2023

  41. [49]

    Temporal Abstraction in Reinforcement Learning

    Doina Precup. Temporal Abstraction in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 2000

    2000

  42. [50]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  43. [51]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Neural Information Processing Systems, 2023. URL https://api.semanticscholar.org/CorpusID:258833055

    2023

  44. [52]

    Tenenbaum, Leslie Pack Kaelbling, and Michael Katz

    Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In AAAI Conference on Artificial Intelligence, 2023. URL https://api.semanticscholar.org/CorpusID:...

    2023

  45. [53]

    Learning options in reinforcement learning

    Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Symposium on Abstraction, Reformulation and Approximation, 2002. URL https://api.semanticscholar.org/CorpusID:16398811

    2002

  46. [54]

    Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BkxUvnEYDH

    2020

  47. [55]

    Sutton, Doina Precup, and Satinder Singh

    Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. 1999. URL https://api.semanticscholar.org/CorpusID:259159153

    1999

  48. [56]

    Autoascend -- 1st place nethack agent for the nethack challenge at neurips 2021

    Maciej Sypetkowski and Michał Sypetkowski. Autoascend -- 1st place nethack agent for the nethack challenge at neurips 2021. https://github.com/maciej-sypetkowski/autoascend, 2021. GitHub repository

    2021

  49. [57]

    Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance

    Andrea Lockerd Thomaz, Cynthia Breazeal, et al. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Aaai, volume 6, pp.\ 1000--1005. Boston, MA, 2006

    2006

  50. [58]

    Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2023

    Ahmed Touati, J \'e r \'e my Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=MYEap_OcQI

    2023

  51. [59]

    Python tutorial, volume 620

    Guido Van Rossum and Fred L Drake Jr. Python tutorial, volume 620. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  52. [60]

    Feudal networks for hierarchical reinforcement learning

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp.\ 3540--3549, 2017

    2017

  53. [61]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id...

    2024

  54. [62]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 0 24824--24837, 2022

    2022

  55. [63]

    Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem

    Maciej Wolczyk, Bartłomiej Cupiał, Mateusz Ostaszewski, Michal Bortkiewicz, Michal Zajkac, Razvan Pascanu, Lukasz Kuci'nski, and Piotr Milo's. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. ArXiv, abs/2402.02868, 2024. URL https://api.se...

    2024 arXiv

  56. [64]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022. URL https://api.semanticscholar.org/CorpusID:252762395

    2022 arXiv

  57. [65]

    Noveld: A simple yet effective exploration criterion

    Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34, 2021

    2021

  58. [66]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  59. [67]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  60. [68]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  61. [69]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.