{"id":"347db7c9-d3ca-485d-a084-28080844e4f2","arxiv_id":"2412.08542","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MaestroMotif uses LLM-generated rewards and code policies to train and combine five NetHack skills, solving complex text-specified tasks without task-specific RL training.","lead":"MaestroMotif turns natural-language descriptions of skills into trained game-playing behaviors by using an LLM to design rewards and code that switches between skills. On a suite of NetHack tasks, it beats baselines that are trained directly on each task, composing skills zero-shot from a text command.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Results do not isolate trained skill policies from the LLM-generated code policy; a missing control leaves the central 'skill design from AI feedback' claim unproven.","rationale":"The reader's weakest assumption concerns the reliability of LLM preference-based rewards (Section 3.3, Eq. 2). That is a real risk, and the paper's ablations (Figure 11) partially address it by showing diffStats, player statistics, and human gameplay each matter. I go one step further and point to a missing control that is more directly tied to the headline numbers: no comparison isolates the trained skill policies from the LLM-generated code policy. If the code policy alone (with untrained skills) reproduces the results, then even perfect reward reliability would not save the central claim that AI feedback produces the skills. This is an internal gap in the evidence rather than a disagreement with consensus; it can be settled by one ablation. The verdict remains CONDITIONAL as the reader recommended: the numbers are internally consistent and well-ablated within their chosen design, but the causal role of the trained skills is not yet established, and the paper's 'first method' and 'zero-shot' claims outrun the released evidence (no code or data). I do not see grounds to reject the empirical results outright, so UNCHANGED is the appropriate final verdict.","tokens_in":31331,"tokens_out":8154,"duration_ms":92587,"concrete_test":"Run the full MaestroMotif evaluation with the five trained skill-conditioned policies replaced by untrained, randomly initialized copies of the same network, holding fixed the LLM-generated policy over skills, initiation and termination functions, environment modification, and evaluation protocol. Use the same nine-seed aggregation on Gnomish Mines, Delphi, Minetown, and Golden Exit (Tables 1-2). If success collapses to near the LLM Policy baseline (~0%), the trained skills are load-bearing and the central claim is supported. If success remains materially above Motif and task-specific baselines, the headline numbers are driven by the code policy and hand-specified environment knowledge, not by skill design from AI feedback, and the paper's central claim should be substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM preference-based reward training produces reusable skills that, when composed by an LLM-written code policy, solve complex NLE tasks zero-shot. The paper never tests whether the trained skill policies are actually responsible for the reported success. Every MaestroMotif result uses the same five RL-trained skill policies, and the ablations in Section 4.3 vary architecture and training strategy but keep those trained policies in place. The deployment code policy (Prompt 7, Output 4) contains substantial hand-specified environment knowledge: branch numbers, branch depth, and purpose-built methods 'reach_gnomish_mines' and 'reach_dungeons_of_doom'. A state machine of this kind, operating over even weak low-level movement, could in principle produce much of the navigation and composite-task success on its own. If an untrained or randomly initialized policy under the same code policy achieved similar success rates, the improvement over Motif and LLM Policy would be attributable to LLM code generation and environment priors, not to the reward-from-AI-feedback skill design that the paper claims as its contribution. Appendix A.5 also modifies the environment to allow eating and quaffing any item, which directly enables the 'eat food' composite subtask and is a further uncontrolled component. This missing control is the load-bearing gap: it sits between the method's two pillars (learned rewards and skills, and generated code) and the headline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MaestroMotif, a pipeline for AI-assisted skill design in the NetHack Learning Environment (NLE). The method takes natural-language skill descriptions, uses an LLM to produce pairwise preferences over unannotated interaction data, distills those preferences into per-skill reward functions (after Motif), uses an LLM coder to generate initiation/termination functions and a training-time policy over skills, and trains a shared skill-conditioned policy with PPO. At deployment, another LLM-generated code policy recombines the trained skills zero-shot. The authors introduce an evaluation suite of navigation, interaction, and composite tasks and report large margins over existing methods, including task-specific RL baselines. They also present ablations of the skill architecture, synchronous versus isolated skill learning, code-refinement, the scale of the code-generating LLM, and the preference representation.","tokens_in":31610,"tokens_out":5444,"duration_ms":61704,"significance":"If the central claim holds, MaestroMotif is a significant step: it suggests that natural-language descriptions plus unannotated data and LLM feedback can replace both hand-engineered reward design and manual low-level controller programming in a complex, partially observable domain such as NetHack. The paper is honestly written and contains useful ablations, an attempt at equivalent-prompting baselines, and detailed appendix prompts and code examples. In particular, the result that giving all the MaestroMotif prompts to the LLM-Policy baseline does not close the gap (Appendix A.11, Table 3) is valuable evidence for the hierarchical structure. However, the headline results depend on an attribution that the current experiments do not fully establish: the contribution of the trained skill policies is never isolated from the LLM-generated code policy and the hand-specified environment priors embedded in the deployment prompts. I do not see a circularity problem, because the success metrics are external NetHack landmarks and not quantities fitted by the method, but the missing control is load-bearing for the central 'skill design from AI feedback' claim.","major_comments":[{"comment":"The ablations in Section 4.3 and Appendix A.11 vary the skill architecture (goal-conditioned versus multi-head), the training strategy (simultaneous versus in isolation), and the amount of prompting information, but every MaestroMotif configuration retains the five RL-trained skill policies. The deployment policy shown in Prompt 7 and Output 4 contains substantial hand-specified environment knowledge: the branch_depth parameter, the fact that the Gnomish Mines branch appears at branch_depth+1, and the purpose-built helper methods reach_gnomish_mines and reach_dungeons_of_doom. A policy-over-skills state machine of this kind, operating over even a weak low-level movement/attack controller, could in principle produce much of the navigation and composite-task success on its own. To support the claim that reward-from-LLM-feedback skill training is the driver of the results in Tables 1 and 2, the paper needs a control in which the same generated code policy is paired with randomly initialized or untrained skill policies (or with a simple hand-coded low-level controller). Without this control, the improvement over the Motif and LLM-Policy baselines could be attributable to the code policy and the environment priors rather than to the learned skills.","section":"Section 4.3 and Appendix A.11"},{"comment":"The environment is modified so that the agent can eat and quaff any item in its inventory by pressing the 'y' key, and this modification is directly load-bearing for the composite task Discovery Hunger, whose specification in Table 2 requires the agent to eat food after becoming hungry. The paper states that all trained agents are evaluated under these same conditions, but it does not report whether the modification changes the difficulty of the tasks or whether the baselines can effectively exploit the new action. Since the paper claims to evaluate on the NetHack Learning Environment, the main text should clearly state that a modified environment is used, and the paper should ideally report results on both the original and modified environments, or at least quantify the effect of this modification on the reported success rates.","section":"Appendix A.5"},{"comment":"The LLM's pairwise preferences over 100-step state differences are treated as ground-truth labels for reward training, but no validation is provided that the labels agree with human judgments or with the intended semantics of the skill descriptions. The ablations in Figure 11 show sensitivity to the representation of the state difference (diffStats, player statistics, human gameplay data), but they do not test the correctness of the preference labels themselves. Because every skill policy is optimized against rewards distilled from these labels, a mismatch between the LLM's preferences and the intended skills would invalidate the 'AI-assisted skill design' claim even if the code policy performed well. At a minimum, the paper should report agreement statistics between the LLM annotator and human annotators on a sample of preference pairs, or ablate the reward source by comparing against a reward trained on a different LLM or on human preferences.","section":"Section 3.3 and Equation (2)"}],"minor_comments":[{"comment":"The column 'RL w/ task reward + score' mixes units: for the navigation tasks it reports percentages, while for Transactions, Price Identified, and BUC Identified it appears to report item counts, but one entry reads '0.01%' even though these are count-based metrics; the table should separate units or use a per-row notation.","section":"Table 1"},{"comment":"The legend entries 'Goal-Cond. & Simultaneously-Learned' and related abbreviations are not expanded in the caption; the caption should state that 'Goal-Cond.' means goal-conditioned policy and define 'Simultaneously-Learned' versus 'Learned In Isolation'.","section":"Figure 8"},{"comment":"In the illustrative code for Discovery Hunger, the perform_task method sets self.eaten_food = True immediately upon entering the Gnomish Mines, without checking that the agent has actually eaten food or that it is hungry; if this example is representative, the deployment code may not faithfully implement the task specification, and the final code after refinement should be shown or verified against the external success metric.","section":"Output 4"},{"comment":"The sentence 'letting the agent eat and quaff any of its items, at random, by performing a particular command' is slightly ambiguous: it should clarify whether the item is chosen randomly by the environment or by the agent, since this affects the controllability of the eat/quaff action.","section":"Section A.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the idea is interesting, but the missing isolated control for the trained skill policies is decisive. The authors can likely address this with a relatively contained set of additional experiments, so I do not recommend rejection. I also noted that the paper self-cites the Motif work heavily, which is understandable given the direct extension, but the presentation would be stronger with an explicit statement of what is newly contributed beyond Motif's protocol beyond the composition and code-policy aspects."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a solid, honest integration paper. The new piece is training a single skill-conditioned network with rewards elicited as LLM preferences (Motif), plus LLM-generated initiation/termination functions and code policies for composition, so skills recombine zero-shot at test time. Results are much better than the LLM-policy and Motif baselines, and the ablations (single network vs multi-head, simultaneous vs isolated training, diffStats, human gameplay, LLM scale) are thorough and real.\n\nThe main soft spot is attribution. The paper never runs the deployment code policy with untrained or random skill policies. The code policy in the appendix carries real environment knowledge: branch numbers, branch depth, and purpose-built helper methods for moving between branches. If those helpers plus a weak low-level policy were enough to produce a decent fraction of the reported success, the \"skill design from AI feedback\" claim would be overstated. I suspect the learned skills matter—random low-level control almost certainly cannot find staircases, reach Minetown, or do the interactions at the reported rates—but the experiment should be in the paper because the headline is about the learned rewards, not code generation. The A.5 environment modification (allowing the agent to eat/quaff any item via key 'y') directly enables the eat-food subgoal of the showcase composite task; it is disclosed and applied to all baselines, so not fatal, but it should be flagged more prominently.\n\nA lesser concern: preference labels come from Llama 3.1 over a compressed 100-step state difference, and the paper reports response rates but no annotation agreement or human spot-check. If preferences are noisy, these coarse rewards may still work, but reliability evidence would strengthen the claim. The benchmark is self-designed around the chosen skills, so the \"zero-shot\" phrasing should be tempered—the skill vocabulary was selected with these tasks in mind.\n\nOverall, the central machinery holds up better than the stress-test note suggests. The missing control is worth requesting, not rejecting over. This paper deserves a serious referee and would benefit from a revision that isolates the skill-reward contribution and analyzes preference quality. I'd send it to review.","headline":"A credible, useful integration of LLM preference rewards with code-policy composition that reports large NLE margins, but the central attribution would be stronger with a control using untrained skills.","tokens_in":32153,"tokens_out":2636,"would_cite":true,"duration_ms":31436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MaestroMotif claims that describing skills in natural language is enough to train an agent that solves hard NetHack tasks zero-shot, outperforming task-specific RL baselines.","keywords":["AI-assisted skill design","reinforcement learning from AI feedback","LLM code generation","NetHack Learning Environment","options framework","zero-shot skill composition","preference-based reward learning","hierarchical reinforcement learning"],"falsifier":"Run the full MaestroMotif pipeline with the same code policies and skill descriptions, but with the LLM preference labels randomly flipped with probability 0.5 during reward training; if task success rates on Gnomish Mines and Golden Exit remain around 46% and 24.8%, the claim that LLM feedback is what carries skill learning would be falsified.","tokens_in":31118,"feed_emoji":"🧩","tokens_out":6949,"duration_ms":65218,"temperature":0.7,"pith_summary":"MaestroMotif aims to show that the skill design problem, converting a high-level description of a behavior into a usable low-level policy, can be automated almost entirely by a large language model, with a human providing only natural-language descriptions. The paper claims that LLM preferences over short state-difference summaries are enough to train reward functions for individual skills, and that LLM-generated Python code can specify when skills start and stop and how they are sequenced, both during training and at test time. On the NetHack Learning Environment, this combination solves navigation, interaction, and composite tasks zero-shot at rates (46% on Gnomish Mines, 24.8% on Golden Exit) that task-specific RL baselines and prompt-only LLM policies essentially fail to reach. If correct, this means a natural-language description plus unannotated interaction data is sufficient to build a reusable skill hierarchy without hand-designed rewards or task-specific training.","feed_headline":"Skill descriptions can train a NetHack agent that beats per-task RL","feed_subtitle":"A pipeline of LLM preference rewards plus LLM-written code beats task-specific training on NetHack.","key_machinery":"The load-bearing object is the option triple (initiation function, intra-option policy, termination function) in the call-and-return style of the options framework, together with a code-generated policy over skills. MaestroMotif obtains each component automatically: rewards from a Bradley-Terry model fitted to LLM preferences (the Motif method), initiation and termination functions and the training-time sequencing policy as LLM-written Python, and a single neural network conditioned on a one-hot skill vector as the shared intra-option policy. The design point is that the LLM never chooses low-level actions; it only supplies rewards and code for when to enter, exit, and switch skills. A notable secondary mechanism is the use of 100-step state differences as a compressed, non-Markovian representation for preference annotation, which the ablation study shows is necessary for the skill rewards to capture the intended behaviors.","core_discovery":"MaestroMotif claims to be the first method that, using only natural-language skill descriptions and unannotated interaction data, produces an agent that solves end-to-end tasks in the NetHack Learning Environment. The pipeline distills LLM pairwise preferences over 100-step state differences into a per-skill reward, uses LLM-written code for initiation and termination functions and for a training-time sequencing policy, trains a shared skill-conditioned network with PPO, and then at deployment lets an LLM write a new code policy that recombines the learned skills without further training. The reported results are success rates of 46% on Gnomish Mines, 29% on Delphi, 7.2% on Minetown, and 24.8% on the Golden Exit composite task, with interaction tasks yielding 0.66 transactions, 0.47 price identifications, and 1.60 bless/cursed identifications per episode, numbers that task-specific RL baselines, prompt-based LLM policies, and even an RL agent trained with privileged reward information do not reach. The authors interpret this as evidence that the binding constraint on such tasks is decomposition into skills, and that LLM-generated code supplies that decomposition.","pith_inferences":["Editorial inference: the same recipe, LLM preference rewards plus code-defined skill sequencing, should transfer to other partially observable environments with rich text-like observations, because the protocol only needs a serializable state difference and a reliable annotator.","Editorial inference: if the learned skill rewards are behaviorally aligned with their descriptions, the skill policies become reusable primitives for a learned high-level planner, a path the paper only hints at in its discussion of online adaptation.","Editorial inference: the method's ceiling is set by LLM quality; a testable extension would be to swap the annotator for a smaller or open-weight model and measure the preference-reliability threshold at which the skill hierarchy loses its edge over per-task RL."],"forward_implications":["Language-only skill design suffices: with MaestroMotif, a human provides skill descriptions and the pipeline produces policies for navigation, interaction, and composite NetHack tasks without task-specific reward engineering.","Deployment-time adaptation becomes a code-writing problem: new task specifications are turned into a policy over skills at inference time, so no additional gradient training is needed to handle a new task from the benchmark.","Simultaneous training under a code-defined sequencing policy induces an emergent curriculum, and ablations show both the skill-conditioned architecture and this curriculum are required for the reported success rates.","The reported scaling trend for the code-generating LLM implies that improvements in code generation quality directly improve zero-shot task performance.","Score-maximizing agents, including behavioral cloning baselines, fail the benchmark tasks almost entirely, indicating that game score is not a proxy for the landmark behaviors MaestroMotif targets."],"supporting_citations":[{"why":"Supplies the Motif preference-distillation method that converts LLM pairwise preferences into per-skill reward functions.","marker":"Klissarov et al. (2024)"},{"why":"Defines the options formalism (initiation, intra-option policy, termination) that structures MaestroMotif's skills.","marker":"Sutton et al. (1999)"},{"why":"Provides the NetHack Learning Environment and the baseline task suite used for evaluation.","marker":"Küttler et al. (2020)"},{"why":"The Llama 3.1 models serve as both the preference annotator and the code generator; the scaling results depend on them.","marker":"Dubey et al. (2024)"},{"why":"Dungeons and Data is the unannotated human gameplay dataset that provides broad coverage for preference elicitation.","marker":"Hambro et al. (2022b)"},{"why":"Contributes the 100-step state-difference representation that compresses history for the LLM annotator.","marker":"Piterbarg et al. (2023a)"},{"why":"Supplies the paired-comparison model used to fit a reward from the LLM's preference labels.","marker":"Bradley & Terry (1952)"},{"why":"PPO is the RL algorithm used to train the skill policies from the LLM-derived rewards.","marker":"Schulman et al. (2017)"},{"why":"Sample Factory is the asynchronous RL implementation that makes large-scale skill training practical.","marker":"Petrenko et al. (2020)"}],"fun_headline_variants":["LLM-written skills and rewards beat per-task RL in NetHack","MaestroMotif: LLM feedback designs skills that outperform RL baselines","Language-described skills plus LLM code outdo task-specific RL","Skill design via LLM feedback tops NetHack RL baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pairwise preferences an LLM expresses over compressed 100-step state differences are reliable enough to train reward functions that induce the intended skill behaviors, and if those preferences are misaligned or the compression discards essential non-Markovian state such as hunger, every downstream skill and composition fails no matter how good the code policy is.","fun_headline_variants_meta":{"raw":{"variants":["LLM-written skills and rewards beat per-task RL in NetHack","MaestroMotif: LLM feedback designs skills that outperform RL baselines","Language-described skills plus LLM code outdo task-specific RL","Skill design via LLM feedback tops NetHack RL baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1594,"prompt_tokens":922,"completion_tokens":672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":595}},"tokens_in":538,"tokens_out":672,"duration_ms":7212,"temperature":1.0,"reasoning_tokens":595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:23.234758+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full MaestroMotif pipeline with the same code policies and skill descriptions, but with the LLM preference labels randomly flipped with probability 0.5 during reward training; if task success rates on Gnomish Mines and Golden Exit remain around 46% and 24.8%, the claim that LLM feedback is what carries skill learning would be falsified.","supporting_citations":[],"review_version":1}