Pith. sign in

REVIEW 2 major objections 5 minor 56 references

Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EXIF: language agents acquire skills by exploring, relabeling, and iterating on feedback.

desk verdict A clean and promising closed-loop skill-discovery method, with a load-bearing instruction-fidelity check left unvalidated and some numbers that need cleaning up before the magnitude claims can be trusted. read the letter →

arxiv 2506.04287 v2 pith:RDTUT6HY submitted 2025-06-04 cs.AI cs.LG

Yongjin Yang , Sinjae Kang , Juyong Lee , Dongjun Lee , Se-Young Yun , Kimin Lee This is my paper
classification cs.AIcs.LG
keywords LLMagentsskilldiscoveryexploration-firstiterativefeedbackself-improvementsyntheticdataWebshopCrafter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes EXIF, a fully automatic framework for discovering skills in language-model agents. An exploration agent, Alice, first wanders an environment without a goal, then retroactively labels each collected trajectory with a natural-language instruction, forming a skill dataset. A second agent, Bob, is fine-tuned on that dataset, Alice evaluates Bob's failures, and her feedback steers the next round of exploration. The authors claim this loop yields consistent, substantial gains without any human demonstrations or manual task proposals, and that even using the same model for Alice and Bob improves performance. The work matters because it points toward agents that can grow their own skills and become self-evolving in unfamiliar environments.

What carries the argument

The mechanism is a two-agent closed loop. Alice, parameterized by φ, explores the environment without a fixed goal, generating exploratory trajectories; she then analyzes each trajectory and writes a natural-language instruction I(j) describing the demonstrated behavior, producing a skill dataset Dskill = {(I(j), τ(j))}. Bob, parameterized by θ, is trained by supervised fine-tuning to maximize the log-likelihood of each trajectory's actions conditioned on the instruction and history. After evaluation, Alice examines Bob's successes and failures and emits concise textual feedback F(k), which conditions her next exploration so the new data targets Bob's known weaknesses. The loop repeats, with data accumulating across iterations.

What would settle it

Run EXIF with the instruction-to-trajectory pairings randomly permuted before training Bob while keeping all other details identical. If Bob's final performance remains close to the normal EXIF result, then instruction fidelity is not what drives the improvements and the skill-labeling mechanism is not load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that an exploration-first, feedback-driven data-generation loop lets a small open-weight language agent surpass a much larger proprietary model on a held-out web shopping benchmark and match it in an open-world game. In Webshop, Llama3.1-8B trained with EXIF improves its reward from 2.0 to 53.7 over four iterations, compared with GPT-4o's 16.5, while in Crafter its learned skills rise from 7 to 14 out of 22 and average progress from 11.4% to 31.9%. The authors further show that replacing the proprietary exploration agent with the same small model as Bob still produces large gains, which they take as evidence for a self-evolving system.

Load-bearing premise

The framework assumes Alice's retrospective relabeling is faithful: each generated instruction genuinely describes what the trajectory accomplishes, and Bob imitates those instructions as if they were correct skills, with no accuracy check on the generated labels.

Editorial extensions

If this is right

  • If the central claim holds, language agents can be trained to master new environments without any human-written demonstrations or task proposals, removing a major bottleneck in open-ended agent learning.
  • Exploration-first data generation is likely to be broadly preferable to proposal-first task generation, since the resulting instructions are guaranteed to match actually achievable behavior.
  • Feedback-driven exploration is the component that sustains improvement across iterations; simply adding more exploration data without feedback plateaus quickly.
  • The same-model result implies that a single model can serve as both teacher and student, opening a route to autonomous self-improvement without proprietary teacher models.
  • The improvement transfers to both an e-commerce navigation task and a long-horizon survival game, suggesting the method is not tied to one environment type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to measure the fidelity of Alice's relabeled instructions against a verifier or against human labels; if instruction accuracy is low yet training still helps, the learning signal may come more from action diversity than from instruction grounding.
  • The method could be applied to other interactive environments such as GUI control or tool use, but its reliance on natural-language feedback may be a bottleneck in environments where failures are hard to articulate.
  • Because feedback in Webshop is computed from validation tasks 501-550 while evaluation uses tasks 0-499, an implicit assumption is that weaknesses observed in the validation subset transfer to the test distribution; this could be checked by varying the validation subset.
  • One risk left implicit in the paper is that Alice's exploration itself may drift toward behaviors that are easy to relabel rather than genuinely useful, so in richer environments a curriculum or intrinsic motivation signal might be needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes EXIF, an automatic skill discovery framework for LLM agents. A teacher agent (Alice) first explores an environment without explicit goals, then retrospectively relabels the collected trajectories into natural-language instruction–trajectory pairs (the 'skill dataset'). A student agent (Bob) is fine-tuned on this dataset via supervised learning. In subsequent iterations, Alice evaluates Bob on validation tasks, produces natural-language feedback about Bob's weaknesses, and uses that feedback to condition its next exploration round, yielding a closed loop. Experiments on Webshop and Crafter compare EXIF against the base models, a proposal-first baseline (PF), and an exploration-first baseline without feedback (EF), using GPT-4o as Alice and Qwen2.5-7B or Llama3.1-8B as Bob. The main claims are that EXIF consistently improves Bob over iterations, that exploration-first data is more valid than proposal-first data, that feedback is responsible for sustained improvement beyond EF, and that using the same model for Alice and Bob still yields substantial gains. The paper includes detailed prompts, pseudocode, and implementation details in the supplementary material.

Significance. If the core claims hold, EXIF would be a practical step toward automatic, human-demonstration-free skill acquisition for LLM agents, with an interesting self-evolving variant. The paper's strengths include evaluation on two external benchmarks with standard metrics, ablations isolating the feedback contribution, and unusually complete disclosure of prompts and training details in the appendix. The headline qualitative result—EXIF beats base models and its own ablations—is supported by Table 1 and Figure 2. However, the central premise that Alice's retrospective relabeling is faithful is not directly validated, and the only reported validity metric is internally inconsistent (Section 3.3 vs. Figure 4a). Because Equation (1) has no other supervision signal, the relabeling quality is load-bearing for every downstream claim, including the same-model self-evolving result.

major comments (2)
  1. [Section 2.1, Eq. (1)] The entire training signal for Bob is the instruction–trajectory pair produced by Alice's retrospective relabeling. The paper provides no direct accuracy check of the generated instructions, and the only evidence offered (the valid-task-ratio analysis in Section 3.3) conflates feasibility with instruction–trajectory alignment and is internally inconsistent (see next comment). This matters because Eq. (1) trains Bob to map each generated instruction to that trajectory's actions: any systematic relabeling error—misleading goals, omitted preconditions, or instructions that capture only the final four steps of a Crafter segment—will be imitated as if it were a skill. Please add a direct fidelity evaluation on a sample of generated instructions (e.g., human or independent LLM judgment of whether the instruction describes the demonstrated behavior and whether the trajectory actually achieves it), reported per iteration and per environment.
  2. [Section 3.3, Figure 4a] The prose states that exploration-first methods yield '85% and 70%' valid skill datasets in Webshop and Crafter, respectively, but Figure 4a displays 88.0% and 93.0%. These numbers must be reconciled. Furthermore, the validity criterion is not operationalized: Appendix E does not specify who makes the validity judgment (human annotator, the same Alice model, or another LLM) or with what prompt/rubric. Since this ratio is the only quantitative support for the exploration-first premise, the discrepancy and missing methodology undermine the claim that exploration-first relabeling produces mostly faithful skill data.
minor comments (5)
  1. [Section 3.2] The text says that adding feedback to EF 'boosts performance by nearly 50%', but Table 1 shows an improvement of about 24.9% for Qwen2.5-7B (42.1 to 52.6) and 40.9% for Llama3.1-8B (38.1 to 53.7) in Webshop reward; please correct the magnitude claim.
  2. [Introduction] The abstract states that the reward improves 'from 2.0 to 52.0', but Table 1 reports final rewards of 52.6 (Qwen) and 53.7 (Llama); the value 52.0 does not match either result and should be updated.
  3. [Figure 4b] The caption says repeated actions, attribute clicks, and search keywords are 'normalized by 20', but the y-axis labels and scale are unclear; please specify the normalization procedure and label the units explicitly.
  4. [Section 3.1 / Appendix B.1] The Webshop success rate (SR) is used as a headline metric, but the main text never defines its threshold; Appendix B.1 defines success as reward 1.0. Consider stating this definition in the main text.
  5. [Table 1 caption] The caption contains a capitalization typo ('We report' with capital W mid-sentence) and the format 'mean±standard error' lacks a space; minor editorial cleanup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main performance claims are evaluated against external benchmarks and held-out tasks, and the relabeling/feedback loop is not forced by construction.

full rationale

The paper's central claim—that EXIF improves Bob's task performance—is measured on external evaluation tasks (Webshop test tasks 0–499; Crafter NS with explicit instructions and AP over fixed seeds), which are distinct from the data-generation pipeline. The skill dataset is produced by Alice's retrospective relabeling of exploration trajectories, and Eq. (1) trains Bob to imitate those instruction-trajectory pairs; the pair is consistent by construction, but whether this transfers to unseen tasks is an empirical question answered by held-out evaluation, not by the training objective itself. The feedback loop in Webshop uses validation tasks 501–550 to guide Alice's next exploration, while evaluation is on test tasks 0–499; in Crafter, feedback comes from open-ended survival rollouts, and NS/AP are evaluated on predefined benchmark tasks with fixed seeds. The EF ablation shows that scaling data without feedback plateaus, so the improvement from feedback is not tautological. No load-bearing self-citations are used; the authors' own prior works appear only in related-work context. The 'valid task ratio' analysis in Sec. 3.3 is an inspection-based sanity check rather than the basis of the performance claim, and the prose/figure discrepancy is an internal inconsistency, not circularity. Concerns about unfaithful instruction relabeling are validity/correctness risks, not circular reductions: the paper does not define the target result in terms of the relabeling, nor does it fit a parameter that is then reported as a prediction. Hence the derivation is self-contained with respect to circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests less on mathematical derivation than on a chain of hand-chosen data-construction choices: which exploration trajectories are kept (Webshop: only episodes ending in 'buy now' within 10 steps; Crafter: last four steps of segments flagged by an unspecified rule-based classifier), which instructions Alice writes (prompt-guided phrasing), which episodes Alice sees for feedback (Webshop: 2 successful + 4 failed at reward threshold 0.5), whether data accumulates across iterations (non-cumulative in Webshop, cumulative in Crafter), and how many iterations are run (4 vs 3). Each of these choices can shape the measured improvement. The method also assumes the teacher model is a capable explorer and diagnostician; the authors concede this in Appendix A.

free parameters (5)
  • Feedback sampling reward threshold (Webshop) = 0.5
    Section 3.1: trajectories are sampled for feedback based on a reward threshold of 0.5; this hand-chosen value determines which failures Alice analyzes and thus shapes all subsequent data generation.
  • Number of EXIF iterations per benchmark = 4 (Webshop), 3 (Crafter)
    Table 1 stops at iteration counts chosen per benchmark; the reported gains depend on where the loop is stopped.
  • Data accumulation scheme = non-cumulative (Webshop), cumulative (Crafter)
    Appendix E and Figure 5b select a different scheme per environment after observing which performs better, a post-hoc experimental choice that affects the measured improvement.
  • Crafter skill-trajectory window = last 4 observation-action pairs per segment
    Appendix E.2 retains only the last four steps of each segment; the window length is hand-chosen and determines what Alice relabels as a skill.
  • Webshop exploration retention rule = trajectories ending in buy now within 10 steps only
    Appendix E.1 keeps only successful purchase episodes; this filter defines the instruction distribution Bob is trained on and was chosen to ensure validity.
assumptions (4)
  • standard math Supervised fine-tuning (Eq. 1) teaches Bob by maximizing the likelihood of Alice's actions given the generated instruction; imitation of the trajectory is assumed to transfer to the evaluation tasks.
    Section 2.2 defines the SFT objective used to train Bob; it presupposes that imitating the exploration trajectories yields policies that generalize to held-out tasks of the same benchmark.
  • domain assumption Alice (GPT-4o) is a reliable exploration policy, relabeler, and diagnostician of Bob's weaknesses.
    Every data-generation step depends on GPT-4o's judgments; the authors concede in Appendix A that the natural-language feedback 'may struggle in more complex environments'.
  • domain assumption Webshop validation tasks 501-550 used for feedback are representative of test tasks 0-499.
    Section 3.1 describes sampling successful and failed trajectories from the validation set for feedback; if the validation and test distributions differ, the feedback-conditioned data could overfit to the validation style.
  • ad hoc to paper The rule-based trajectory segmenter detects skill-relevant changes using only agent-observable information, and the last-four-steps window captures each skill.
    Appendix E.2 introduces the segmenter and the four-step retention rule without specifying the classifier's thresholds; the resulting segments are the units that Alice relabels into skills.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback." pith.science (2026). https://pith.science/paper/RDTUT6HY

@misc{pith2026250604287,
  author       = {Pith},
  title        = {Pith review of: Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDTUT6HY}},
  note         = {Machine review of arXiv:2506.04287}
}
read the original abstract

Training large language model (LLM) agents to acquire necessary skills and perform diverse tasks within an environment is gaining interest as a means to enable open-endedness. However, creating the training dataset for their skill acquisition faces several challenges. Manual trajectory collection requires significant human effort. Another approach, where LLMs directly propose tasks to learn, is often invalid, as the LLMs lack knowledge of which tasks are actually feasible. Moreover, the generated data may not provide a meaningful learning signal, as agents often already perform well on the proposed tasks. To address this, we propose a novel automatic skill discovery framework EXIF for LLM-powered agents, designed to improve the feasibility of generated target behaviors while accounting for the agents' capabilities. Our method adopts an exploration-first strategy by employing an exploration agent (Alice) to train the target agent (Bob) to learn essential skills in the environment. Specifically, Alice first interacts with the environment to retrospectively generate a feasible, environment-grounded skill dataset, which is then used to train Bob. Crucially, we incorporate an iterative feedback loop, where Alice evaluates Bob's performance to identify areas for improvement. This feedback then guides Alice's next round of exploration, forming a closed-loop data generation process. Experiments on Webshop and Crafter demonstrate EXIF's ability to effectively discover meaningful skills and iteratively expand the capabilities of the trained agent without any human intervention, achieving substantial performance improvements. Interestingly, we observe that setting Alice to the same model as Bob also notably improves performance, demonstrating EXIF's potential for building a self-evolving system.

Figures

Figures reproduced from arXiv: 2506.04287 by the authors.

Figure 1
Figure 1. Overview of our framework for automatic skill discovery through exploration and iterative [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of EXIF with feedback at each iteration versus EF, which scales data by generating more samples per iteration without feedback, on Webshop and Crafter using Qwen2.5-7B. Increasing the amount of data alone does not improve performance without feedback. Webshop search[home office furniture sets for dining room living room color: blue | gray , shape: runner size: 21 in x 35 in price: < 60.00] [P… view at source ↗
Figure 3
Figure 3. Qualitative examples of action sequences generated by the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) The ratio of valid skill dataset among those generated using PF and EF approaches in Webshop and Crafter. (b) The average number of repeated actions (# R), average number of clicking attributes (# C), and average number of search keywords (# SW) by Bob, normalized …
Figure 5
Figure 5. Figure 5: (a) Performance of Bob using the Qwen2.5-7B model when Alice is Gpt-4o (red) or the Qwen2.5-7B (blue) model, investigating the potential of a self-evolving system (blue). (b) Ablation on whether using data from the previous iteration, where “Cumulative” means using dat…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 21 canonical work pages

  1. [1]

    Variational option discovery algorithms

    Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299, 2018

    arXiv 2018

  2. [2]

    Fireact: Toward language agent fine-tuning

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023

    arXiv 2023

  3. [3]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2023

    work page 2023

  4. [4]

    Emergent complexity and zero-shot transfer via unsupervised environment design

    Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 2020

    work page 2020

  5. [5]

    Guiding pretraining in reinforcement learning with large language models

    Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, 2023

    work page 2023

  6. [6]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018

    arXiv 2018

  7. [7]

    Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code

    Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code. arXiv preprint arXiv:2405.15568, 2024

    arXiv 2024

  8. [8]

    Automatic goal generation for reinforcement learning agents

    Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, 2018

    work page 2018

Show all 56 references
  1. [9]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    Benchmarking the spectrum of agent capabilities

    Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022

    2022

  3. [11]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  4. [12]

    Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhen- zhong Lan, and Dong Yu. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv preprint arXiv:2410.19609, 2024. 10

    arXiv

  5. [13]

    Pokéllmon: A human-parity agent for pokémon battles with large language models

    Sihao Hu, Tiansheng Huang, and Ling Liu. Pokéllmon: A human-parity agent for pokémon battles with large language models. arXiv preprint arXiv:2402.01118, 2024

    2024 arXiv

  6. [14]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118–9147. PMLR, 2022

    2022

  7. [15]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  8. [16]

    Illuminating generalization in deep reinforcement learning through procedural level generation

    Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv:1806.10729, 2018

    2018 arXiv

  9. [17]

    Dataenvgym: Data generation agents in teacher environments with student feedback

    Zaid Khan, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Dataenvgym: Data generation agents in teacher environments with student feedback. arXiv preprint arXiv:2410.06215, 2024

    2024 arXiv

  10. [18]

    Tree search for language model agents

    Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.01476, 2024

    2024

  11. [19]

    Autowebglm: A large language model- based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model- based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and D...

    2024

  12. [20]

    Unsupervised reinforcement learning with contrastive intrinsic control

    Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. Advances in Neural Information Processing Systems, 2022

    2022

  13. [21]

    Benchmarking mobile device control agents across diverse configurations

    Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations. arXiv preprint arXiv:2404.16660, 2024

    2024 arXiv

  14. [22]

    Competitive experience replay

    Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. arXiv preprint arXiv:1902.00528, 2019

    1902 arXiv

  15. [23]

    Weblinx: Real-world website navigation with multi-turn dialogue

    Xing Han Lù, Zdenˇek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930, 2024

    2024

  16. [24]

    Nnetscape navigator: Complex demonstrations for web agents without a demonstrator

    Shikhar Murty, Dzmitry Bahdanau, and Christopher D Manning. Nnetscape navigator: Complex demonstrations for web agents without a demonstrator. arXiv preprint arXiv:2410.02907, 2024

    2024 arXiv

  17. [25]

    Bagel: Bootstrapping agents by guiding exploration with language

    Shikhar Murty, Christopher D Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Bootstrapping agents by guiding exploration with language. In International Conference on Machine Learning. PMLR, 2024

    2024

  18. [26]

    Lift: Unsupervised reinforcement learning with foundation models as teachers

    Taewook Nam, Juyong Lee, Jesse Zhang, Sung Ju Hwang, Joseph J Lim, and Karl Pertsch. Lift: Unsupervised reinforcement learning with foundation models as teachers. arXiv preprint arXiv:2312.08958, 2023

    2023 arXiv

  19. [27]

    Asymmetric self-play for automatic goal discovery in robotic manipulation

    OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D’Sa, Arthur Petron, Henrique P d O Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882, 2021

    2021 arXiv

  20. [28]

    Balrog: Bench- marking agentic llm and vlm reasoning on games

    Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuci´nski, Lerrel Pinto, Rob Fergus, et al. Balrog: Bench- marking agentic llm and vlm reasoning on games. In International Conference on Learning Repres...

    2025

  21. [29]

    Explorer: Scaling exploration-driven web trajectory synthesis for multimodal web agents

    Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Awadallah. Explorer: Scaling exploration-driven web trajectory synthesis for multimodal web agents. arXiv preprint arXiv:2502.11357, 2025. 11

    2025 arXiv

  22. [30]

    Lipschitz- constrained unsupervised skill discovery

    Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Representa- tions, 2022

    2022

  23. [31]

    Accelerating reinforcement learning with learned skill priors

    Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, 2021

    2021

  24. [32]

    Skew-fit: State-covering self-supervised reinforcement learning

    Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019

    1903 arXiv

  25. [33]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. In International Conference on Learn...

    2025

  26. [34]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 2023

    2023

  27. [35]

    Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments

    Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arık. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893, 2025

    2025 arXiv

  28. [36]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning

    Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999

    1999

  29. [37]

    Open- ended learning leads to generally capable agents

    Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open- ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021

    2021 arXiv

  30. [38]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  31. [39]

    Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions

    Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019

    1901 arXiv

  32. [40]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  33. [41]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  34. [42]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 2022

    2022

  35. [43]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023

    2023

  36. [44]

    Skill reinforcement learning and planning for open-world long-horizon tasks

    Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Skill reinforcement learning and planning for open-world long-horizon tasks. arXiv preprint arXiv:2303.16563, 2023

    2023 arXiv

  37. [45]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023. 12

    2023 arXiv

  38. [46]

    Omni: Open-endedness via models of human notions of interestingness

    Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711, 2023

    2023 arXiv

  39. [47]

    Bootstrap your own skills: Learning to solve new tasks with large language model guidance

    Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint arXiv:2310.10021, 2023

    arXiv

  40. [48]

    Skillweaver: Web agents can self-improve by discovering and honing skills

    Boyuan Zheng, Michael Y Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, et al. Skillweaver: Web agents can self-improve by discovering and honing skills. arXiv preprint arXiv:2504.07079, 2025

    2025 arXiv

  41. [49]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Represent...

    2024

  42. [50]

    Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents

    Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran Li. Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents. arXiv preprint arXiv:2412.13194, 2024

    2024 arXiv

  43. [51]

    Find me double sided, machine washable decorative pillows with printing technology with size: 28

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024. 13 Automated Skill Discovery for Language Agents through Exploration and Iterative F eedback Su...

    2024 arXiv

  44. [53]

    HTML Element

    click[HTML Element]: Use this action to click on an HTML Element in the page content. “HTML Element” can be any clickable element in the page represented inside “[button]” and “[button_]”, such as an item id, action button, or attributes and options like color or size. Note th...

  45. [54]

    [button] Search [button_]

    search[keywords]: Use this action only when a “[button] Search [button_]” is present in the current web page content. You must replace “keywords” with any valid search query you want to search

  46. [55]

    HTML Element

    click[HTML Element]: Use this action to click on an HTML Element in the page content. “HTML Element” can be any clickable element in the page represented inside “[button]” and “[button_]”, such as an item id, action button, or attributes and options like color or size. Note th...

  47. [56]

    Refrain from B during your exploration

    The current low reward is due to B. Refrain from B during your exploration

  48. [57]

    last action

    The current low reward is due to not clicking C. Ensure to click diverse C during your exploration. D.1.5 Post-hoc reasoning prompt Post-hoc Reasoning Prompt You are an AI assistant tasked with explaining actions taken in a web environment. Given the instruction you need to fo...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.