Pith. sign in

REVIEW 6 major objections 6 minor 60 references

Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An agent with zero knowledge of a game—no rules, no APIs, no game-specific prompts—learns to play it from raw screenshots and outperforms rivals that were handed full game rules.

desk verdict A genuinely new zero-prior agent loop that works in two games, carrying a circular headline metric and an unisolated central claim — worth serious refereeing, not desk rejection. read the letter →

arxiv 2505.17673 v1 pith:USH7WACP submitted 2025-05-23 cs.AI

Jiawei Du , Jinlong Wu , Yuzheng Chen , Yucheng Hu , Bing Li , Joey Tianyi Zhou This is my paper
classification cs.AI
keywords bottom-upagentdesignskillevolutionLLMagentsopen-endedenvironmentsimplicitrewardexperience-drivenlearningvisualgroundingtrial-and-reasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that LLM-based agents can become competent in unfamiliar, open-ended environments without any human-designed workflow. It proposes a bottom-up design in which an agent starts with an empty skill library, tries out low-level mouse and keyboard actions, keeps only the sequences that visibly change the game screen, and reasons about those changes to build longer, reusable skills. The same game-agnostic codebase, with identical prompts, is dropped into Slay the Spire and Civilization V, and it makes meaningful progress in both. If the claim holds, the main engineering asset in agent systems shifts from task-specific workflows to a shared, evolving library of skills acquired from experience.

What carries the argument

The machine is the four-step skill-evolution loop centred on a shared, initially empty skill library $\mathbb{S}$. Augmentation composes longer action sequences by appending atomic actions to validated shorter skills, keeping a sequence only if it triggers observable environmental change; invocation asks the LLM to propose candidate skills for the current screen and ranks them with Monte Carlo Tree Search; evaluation scores each execution by the visual difference between pre- and post-screenshots, formalised as the semantic reward $R_{\mathrm{semantics}} = M(p_{\mathrm{differ}}, \sigma, x_t, x_{t+1})$; refinement prunes chronically poor skills and rewrites others by LLM reasoning. The formal objective that ties these together is the population-level maximisation of $R_{\mathrm{skill}} = R_{\mathrm{diversity}} + R_{\mathrm{efficiency}} + R_{\mathrm{semantics}}$ over the library, which turns open-ended play into a continual optimisation problem. Separately, the Segment Anything model converts raw pixels into candidate UI elements, giving the atomic actions a tractable target space.

What would settle it

Run the identical zero-prior agent on a turn-based game where the winning line begins with a preparatory move that changes nothing on screen for several turns—for instance a Slay the Spire-style fight that punishes attacking and rewards building block for an attack that lands later—and inspect which skills enter the library. The framework predicts such preparatory skills will be filtered out as no meaningful action, and if the agent's progression collapses on that floor while a copy of the agent given a second screenshot-difference threshold succeeds, the visual-change reward is the bottleneck the claim must survive.

Watch

Extended reading notes

Core claim

The central claim is that competence in an open-ended environment can be bootstrapped entirely from experience, with no priors, through a formalised skill-evolution loop. The environment is modelled as a partially observable Markov decision process whose observations are raw screenshots and whose actions are atomic mouse events, and the evolution of the skill library is written as the population-level objective $\max_{\mathbb{S}} \mathbb{E}[R_{\mathrm{skill}}(\sigma,\mathbb{S},x_t,\mathcal{T})]$, where $R_{\mathrm{skill}} = R_{\mathrm{diversity}} + R_{\mathrm{efficiency}} + R_{\mathrm{semantics}}$. The semantic term, the only one used to drive evaluation in the implemented system, is obtained by asking an LLM to compare screenshots before and after a skill executes, treating visual difference as implicit reward. Skills are composed incrementally, retained only when they produce recognisable environmental change, selected at each step by Monte Carlo Tree Search over LLM-proposed candidates, and pruned or rewritten when their rewards are poor. On this basis the agent clears 13 floors of Slay the Spire with a 98.56% execution responsive rate, and completes 50 turns and 8 researched technologies in Civilization V, exceeding every baseline including variants given task-specific priors.

Load-bearing premise

The design assumes that a visible change on screen is a trustworthy sign that a skill is worth keeping, which means any useful behaviour whose payoff is delayed or visually subtle is never learned and gets pruned away.

Editorial extensions

If this is right

  • Zero-prior agents can produce sustained, meaningful play in complex games: 13 cleared floors and score 81 in Slay the Spire, and 50 turns with 8 researched technologies in Civilization V.
  • Skill libraries improve with use: across training rounds the library grows through augmentation and pruning while execution responsiveness and in-game score rise, and the paper reports convergence by the fourth round.
  • MCTS-based skill selection is the linchpin of invocation: ablating it drops progression from several floors to a single floor.
  • A single unified codebase and prompt set transfers across structurally different games without modification, so deployment effort moves from per-game engineering to a general skill-evolution loop.
  • The paradigm is built for scale: with a shared library, every deployed agent's discoveries become immediately available to all others, so collective progress accelerates with population size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The screenshot-difference filter makes the reported execution responsive rate partly tautological: only skills that changed the screen were stored, so a high responsive rate is baked in by construction; a harder test is whether the retained skills are the ones that win games rather than merely move pixels.
  • The same reward choice biases the library toward greedy, immediately visible behaviours, so the paradigm as implemented is unlikely to discover defensive or setup play; pairing the visual signal with learned long-horizon credit assignment, which the authors list as future work, would directly test this ceiling.
  • A cheap, decisive experiment would run the codebase in a third environment engineered with delayed-payoff actions and vary the threshold on what counts as a recognisable change, measuring whether library composition and progression track the threshold.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes a bottom-up agent paradigm in which an LLM-powered agent perceives raw screenshots, acts via mouse and keyboard, and incrementally builds a skill library through trial-and-reasoning, without game-specific prompts or privileged APIs. The framework formalizes skill augmentation, MCTS-based invocation, and LLM-guided refinement under an implicit reward based on visual change. Experiments in Slay the Spire and Civilization V report that the bottom-up agent clears 13 floors and completes 50 turns with 8 technologies, outperforming prompted baselines. The authors conclude that agents can acquire competence from scratch through experience, with skill evolution and sharing as the key mechanism.

Significance. If the claims held, this would be a notable step toward experience-driven agents in open-ended environments, complementing top-down workflow designs. The paper deserves credit for using a unified game-agnostic codebase, raw visual inputs, low-level actions, and for releasing code. However, the current evaluation does not establish the central claim: the headline execution metric is partly circular by construction, the experimental design does not isolate skill evolution from the surrounding scaffolding, and single-run results without variance do not support the quantitative superiority claims. The paper's own Section 6 candidly acknowledges several of these issues, including delayed visual signals and reset/evaluation problems; these are not merely future work, because they bear directly on the reported numbers.

major comments (6)
  1. [§5.1, Table 1; §3.2, Algorithm 1] The 'Execution Responsive Rate' is circular with the skill-admission filter. Algorithm 1 retains a skill only if 'effect(s(k)) is recognizable,' that is, if it produces observable visual change, and §5.1 defines the metric as the percentage of skill invocations that lead to observable changes in game state. Therefore the reported 98.56% and 92.27% rates are nearly tautological and cannot support the claim that the discovered skills are 'meaningful' or 'highly functional.' Please define an independent metric, such as the contribution of each skill to floors cleared, in-game score, or technologies researched, or evaluate responsiveness on a set of skills that were not filtered by visual-change detection.
  2. [§3.1, Eq. (2); §4] The formal objective in Eq. (2) defines Rskill as a sum of Rdiversity, Refficiency, and Rsemantics, but Section 4 states that 'we use only the semantic reward to guide skill evaluation.' The experiments therefore do not optimize or test the formalized objective. Either implement all three terms and report their contributions, or present Eq. (2) as a proposed formulation rather than the objective that is actually evaluated.
  3. [§5.3, Table 3] No ablation removes the skill library while keeping the other components. The three ablations disable the visual filter, MCTS, or skill description, but in every condition the agent still accumulates and reuses skills. This leaves open the alternative explanation that the reported progression comes primarily from SAM-based perception, MCTS selection, and LLM reasoning, rather than from bottom-up skill evolution. Add a no-evolution control that keeps the same perception, MCTS, and LLM components but disables all skill storage and reuse.
  4. [§5.1, Table 1] All quantitative comparisons are single-run values with no seed variance or confidence intervals, and the footnotes indicate that some episodes terminated early at steps 301 and 232. Open-ended games are stochastic, so a one-episode difference, such as 13 floors versus 8 floors, cannot support the claim that the bottom-up agent 'outperforms all baselines.' Report multiple independent episodes per condition with mean and standard deviation, and state clearly whether the tabulated numbers are one representative run or an aggregate.
  5. [§5.2, Table 2] The skill-evolution-over-rounds evidence is not statistically grounded. Progression increases from 6 to 8 floors and score from 36 to 53, but the score is non-monotonic across rounds and no variance is reported. With only 100 steps per round and a single trajectory, the observed improvement could reflect stochasticity or simple library growth rather than skill refinement. Run repeated trials and report variance, or soften the claim to an anecdotal demonstration.
  6. [§5.1, Baselines and Figure 7] The baseline comparisons are confounded. The bottom-up agent receives SAM-based segmentation, MCTS, and a shared skill library, while the baselines receive only the text priors shown in Figure 7 and no equivalent perceptual or planning scaffolding. Consequently, the superiority in Table 1 does not identify which component is responsible, and the 'zero prior' framing is misleading because the bottom-up system uses a pretrained segmentation model. Add controlled baselines that share the same low-level interface and scaffolding, with and without skill evolution.
minor comments (6)
  1. [Table 1] In the bottom-up row for Civilization V, there appears to be a stray '1' before '50' in the Progression column; please correct the table formatting.
  2. [Abstract and §4] The abstract claims operation 'without any game-specific prompts or privileged APIs,' but Section 4 introduces SAM for visual grounding; please clarify explicitly whether SAM is considered part of the zero-prior setting or an external perceptual prior.
  3. [§5.1] Section 5.1 says each agent runs for three episodes per environment, but Table 1 reports single progression values with early termination at steps 301 and 232; please explain how the three episodes relate to the reported numbers.
  4. [§4 and Related Work] There are several typos, including 'reconized' and 'enviroments' in Section 4 and 'V oyager' in the related-work discussion; a careful copyedit pass is needed.
  5. [Figure 2(b)] Figure 2(b) needs axis labels and a legend; as printed, the visual comparison of game progression is difficult to parse.
  6. [§6, Exploration Overhead] The exploration overhead discussion states that bottom-up agents take about 12 hours versus 6 hours for baselines, while Table 1 reports bottom-up token costs below GPT-4o; please clarify the relationship between wall-clock time and token cost, since these are different quantities.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline Execution Responsive Rate is the skill-admission filter restated; the 98.56% figure is forced by the retention criterion rather than independent evidence of skill quality.

  1. self definitional [Section 3.2 (Skill Augmentation), Section 4 (Implicit Reward and Skill Refinement), Section 5.1 (Evaluation Metrics), Table 1]
    "To filter out meaningless behaviors, only sequences that trigger observable environmental changes are retained. ... we use only the semantic reward to guide skill evaluation. This is computed via the visual difference between screenshots before and after skill execution as a proxy for semantic reward. ... (c)Execution Responsive Rate: the percentage of skill invocations that lead to observable changes in game state, assessing functional validity of behaviors. ..."

    The skill-admission criterion (Section 3.2) keeps only sequences that produce observable environmental changes; the implicit-reward/evaluation rule (Section 4) scores skills by the visual difference between pre- and post-execution screenshots. The headline Execution Responsive Rate then measures the fraction of invocations that lead to observable changes in game state. Hence the 98.56% figure is not an independent measure of skill quality: it is the same visual-change test used to decide which skills are stored in the library. An agent whose library was filtered by this criterion will, by construction, show a high Execution Responsive Rate; the number is largely fixed by the filter, not by the quality of evolutionary learning.

full rationale

The central progression claims (floors cleared, technologies researched) are grounded in external game state and are not circular: they measure outcomes rather than the agent's own admission filter. However, the paper repeatedly uses the Execution Responsive Rate as evidence that discovered skills are 'meaningful' and 'highly functional,' and that metric is defined by the same observable-environmental-change criterion used to filter skills into the library. This makes the 98.56% headline number a restatement of the retention rule rather than an independent behavioral measurement. The missing no-evolution control (keeping SAM and MCTS but disabling the shared skill library) is a genuine experimental confound for the attribution of progress to skill evolution, but it is not itself a circularity; it is mentioned here only to separate confounds from definitional circularity. No load-bearing self-citation chain was found, and no uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework rests on the compositionality of atomic UI actions, the sufficiency of visual-change rewards, and the reliability of LLM reasoning; none of these are externally calibrated. The evaluation metric 'execution responsive rate' is the same visual-change signal used to filter skills, creating a circularity burden.

free parameters (5)
  • Visual change threshold / effect recognizability criterion
    Whether a skill sequence counts as 'recognizable effect' during augmentation (Section 3.2) is a hand-chosen criterion; no threshold value is given. It is the gate that decides which skills enter the library.
  • Reward weights (alpha, beta, gamma) for Rskill = 0, 1, 0 (semantic only)
    Equation 2 defines a three-term objective, but Section 4 states 'we use only the semantic reward', setting diversity and efficiency weights to zero. This is a modeling choice not derived from data.
  • Maximum skill length kmax
    Skill augmentation iterates k = 1 .. kmax (Algorithm 1); kmax is not specified and affects search depth.
  • MCTS budget/rollouts
    Skill invocation uses MCTS with a 'lightweight' budget (Section 3.2), but the number of rollouts or simulations is not given; it affects selection behavior.
  • Refinement threshold (skill count trigger)
    Algorithm 1 triggers refinement 'if skill count < threshold'; the threshold value is unspecified.
assumptions (4)
  • domain assumption All complex skills can be composed from low-level human-like actions (mouse clicks, drags, key presses)
    Section 3.1 states this compositionality as the theoretical basis for bottom-up competence; if some useful behaviors require more than ordered atomic sequences, the framework cannot represent them.
  • domain assumption Visual difference between pre- and post-execution screenshots is a sufficient proxy for semantic reward
    Section 4: 'we use only the semantic reward to guide skill evaluation. This is computed via the visual difference between screenshots'. The paper's own Limitations note that subtle or delayed effects will be missed.
  • ad hoc to paper Skills that produce no observable environmental change are meaningless and can be discarded
    Section 3.2: 'only sequences that trigger observable environmental changes are retained'. This is a design axiom that conflates visibility with usefulness.
  • domain assumption LLM reasoning (GPT-4o with VLM) can reliably select, describe, cluster, and refine skills from visual input
    The whole framework relies on the LLM to map screenshots to sensible skill selections and semantic descriptions; no calibration of this mapping is provided beyond final game scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution." pith.science (2026). https://pith.science/paper/USH7WACP

@misc{pith2026250517673,
  author       = {Pith},
  title        = {Pith review of: Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USH7WACP}},
  note         = {Machine review of arXiv:2505.17673}
}
read the original abstract

Most LLM-based agent frameworks adopt a top-down philosophy: humans decompose tasks, define workflows, and assign agents to execute each step. While effective on benchmark-style tasks, such systems rely on designer updates and overlook agents' potential to learn from experience. Recently, Silver and Sutton(2025) envision a shift into a new era, where agents could progress from a stream of experiences. In this paper, we instantiate this vision of experience-driven learning by introducing a bottom-up agent paradigm that mirrors the human learning process. Agents acquire competence through a trial-and-reasoning mechanism-exploring, reflecting on outcomes, and abstracting skills over time. Once acquired, skills can be rapidly shared and extended, enabling continual evolution rather than static replication. As more agents are deployed, their diverse experiences accelerate this collective process, making bottom-up design especially suited for open-ended environments. We evaluate this paradigm in Slay the Spire and Civilization V, where agents perceive through raw visual inputs and act via mouse outputs, the same as human players. Using a unified, game-agnostic codebase without any game-specific prompts or privileged APIs, our bottom-up agents acquire skills entirely through autonomous interaction, demonstrating the potential of the bottom-up paradigm in complex, real-world environments. Our code is available at https://github.com/AngusDujw/Bottom-Up-Agent.

Figures

Figures reproduced from arXiv: 2505.17673 by the authors.

Figure 1
Figure 1. Two paradigms of agent design. Most exist￾ing agent frameworks can be categorized as Top-down agents, which rely on pre-engineered architectures: they begin with high-level goals, decompose them into sub￾tasks, and execute workflows using task-specific APIs and tools. In contrast, we propose Bottom-Up agents to function as explorers: starting from zero prior knowl￾edge, they gradually acquire skills through trial-an… view at source ↗
Figure 2
Figure 2. Left: The bottom-up agent operates solely on raw visual input and simulates low-level mouse and keyboard actions. Without explicit rewards, it learns and refines skills based on implicit signals like visual changes or game progression. Right: Game progression measured by Civilization V’s tech tree and visual changes. Our bottom-up agent (blue) outperforms all baselines, including those with task-related priors. The … view at source ↗
Figure 3
Figure 3. Overview of Bottom-Up Skill Evolution. The agent begins with no predefined skills and gradually builds its library S through interaction. Left: New skills are incrementally composed by extending existing routines with atomic actions. Middle: Skills are evaluated by a visual-language model (VLM) comparing pre- and post-execution states; ineffective ones are refined or discarded via LLM reasoning. Right: At each times… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Analysis of skill evolution and reuse. (a) Skill library size increases over time through augmentation (+) and pruning (–). (b)Top-10 most frequently invoked skills in Slay the Spire. (c) Examples of compositional skill inheritance across environments, showing how high…
Figure 5
Figure 5. Figure 5: Prompting and execution visualization of the bottom-up agent. (a) Environment-agnostic prompts used for skill augmentation and invocation, enabling reasoning without access to game-specific APIs. (b) We design a GUI to visualized execution state of the agent during gam…
Figure 6
Figure 6. Figure 6: Prompting and execution visualization of the bottom-up agent. (a) Environment-agnostic prompts used for skill augmentation and invocation, enabling reasoning without access to game-specific APIs. We use the same codebase and prompts in both Slay the Spire and Civilizai…
Figure 7
Figure 7. Figure 7: Baseline prior knowledge provided to top-down agents. To enable task execution in Slay the Spire (left) and Civilization V (right), baseline agents are given structured game-specific summaries detailing rules, objectives, and UI controls. These priors are necessary for…
Figure 8
Figure 8. Figure 8: Prompts for skill augmentation via clustering and merging. The left prompt guides the agent to group functionally equivalent actions into clusters; the right prompt extends this by merging with existing clusters. These prompts enable semantic consolidation of skills du…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 16 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, K...

    arXiv 2022

  3. [3]

    Claude 3.7 sonnet and claude code

    Anthropic. Claude 3.7 sonnet and claude code. Anthropic Blog, February 2025. https: //www.anthropic.com/news/claude-3-7-sonnet

    work page 2025

  4. [4]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    2022

  5. [5]

    Windows agent arena: Evaluating multi-modal os agents at scale

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264, 2024

    arXiv 2024

  6. [6]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalash- n...

    arXiv 2023

  7. [7]

    Rt-1: Robotics transformer for real-world control at scale, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, ...

    arXiv 2023

  8. [8]

    Manus ai: General ai agent, March 2025

    Butterfly Effect AI. Manus ai: General ai agent, March 2025. URL https://manus.im/. Accessed: 2025-05-14

    work page 2025

Show all 60 references
  1. [9]

    Iteration head: A mechanistic study of chain-of-thought

    Vivien Cabannes, Charles Arnal, Wassim Bouaziz, Xingyu Yang, Francois Charton, and Ju- lia Kempe. Iteration head: A mechanistic study of chain-of-thought. Advances in Neural Information Processing Systems, 37:109101–109122, 2024

    2024

  2. [10]

    Why do multi- agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi- agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025

    2025 arXiv

  3. [11]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  4. [12]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, V olodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018. URL http...

    2018 arXiv

  5. [13]

    Diversity is all you need: Learning skills without a reward function, 2018

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function, 2018. URL https://arxiv.org/abs/1802. 06070

    2018

  6. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  7. [15]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  8. [16]

    Mastering diverse control tasks through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1–7, 2025

    2025

  9. [17]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  10. [18]

    Metagpt: Meta programming for multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6, 2023

    2023 arXiv

  11. [19]

    Video prediction policy: A generalist robot policy with predictive visual representations, 2025

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations, 2025. URL https://arxiv.org/abs/2412. 14803

    2025

  12. [20]

    Vima: General robot manipulation with multimodal prompts, 2023

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts, 2023. URL https://arxiv.org/abs/2210.03094

    2023 arXiv

  13. [21]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  14. [22]

    Leibo, Edgar Duéñez-Guzmán, Alexander Sasha Vezhnevets, John P

    Joel Z. Leibo, Edgar Duéñez-Guzmán, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot, 2021. URL https: //arxiv....

    2021 arXiv

  15. [23]

    Camel: Communicative agents for "mind" exploration of large language model society,

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society,

  16. [24]

    Code as policies: Language model programs for embodied control, 2023

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023. URL https://arxiv.org/abs/2209.07753

    2023 arXiv

  17. [25]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  18. [26]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015

  19. [27]

    Lill- icrap, Tim Harley, David Silver, and Koray Kavukcuoglu

    V olodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lill- icrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, 2016. URL https://arxiv.org/abs/1602.01783

    2016 arXiv

  20. [28]

    Gpt-4o: Openai’s multimodal flagship model

    OpenAI. Gpt-4o: Openai’s multimodal flagship model. OpenAI Blog, May 2024. https: //openai.com/index/hello-gpt-4o/

    2024

  21. [29]

    Introducing operator: Openai’s autonomous web agent

    OpenAI. Introducing operator: Openai’s autonomous web agent. OpenAI Blog, January 2025. https://openai.com/index/introducing-operator/

    2025

  22. [30]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  23. [31]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023

    2023 arXiv

  24. [32]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  25. [33]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  26. [34]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  27. [35]

    On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning

    Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061, 2022

    2022 arXiv

  28. [36]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  29. [37]

    Welcome to the era of experience

    David Silver and Richard S Sutton. Welcome to the era of experience. 2025

    2025

  30. [38]

    Open-world object manipulation using pre-trained vision-language models

    Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023. 13

    2023 arXiv

  31. [39]

    Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arık. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments, 2025. URL https://arxiv.org/abs/2501.10893

    2025 arXiv

  32. [40]

    Monte carlo tree search: A review of recent modifications and applications

    Maciej ´Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Ma´ndziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):2497–2562, 2023

    2023

  33. [41]

    Open-ended learning leads to generally capable agents, 2021

    Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes- Fitt, Valenti...

    2021 arXiv

  34. [42]

    Feudal networks for hierarchical reinforcement learning,

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning,

  35. [43]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  36. [44]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091, 2023

    2023 arXiv

  37. [45]

    Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions, 2019. URL https://arxiv.org/abs/1901.01753

    2019 arXiv

  38. [46]

    Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration

    Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration. arXiv preprint arXiv:2503.18891, 2025

    2025 arXiv

  39. [47]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  41. [49]

    Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023. URL https...

    2023 arXiv

  42. [50]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  43. [51]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  44. [52]

    Auto-gpt for online decision making: Benchmarks and additional opinions

    Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023

    2023 arXiv

  45. [53]

    Hierarchical multi- agent skill discovery

    Mingyu Yang, Yaodong Yang, Zhenbo Lu, Wengang Zhou, and Houqiang Li. Hierarchical multi- agent skill discovery. Advances in Neural Information Processing Systems, 36:61759–61776, 2023. 14

    2023

  46. [54]

    The second half

    Shunyu Yao. The second half. ysymyth.github.io, April 2025. https://ysymyth.github. io/The-Second-Half/

    2025

  47. [55]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  48. [56]

    S 2-mad: Breaking the token barrier to enhance multi-agent debate efficiency

    Yuting Zeng, Weizhe Huang, Lei Jiang, Tongxuan Liu, Xitai Jin, Chen Tianying Tiana, Jing Li, and Xiaohua Xu. S 2-mad: Breaking the token barrier to enhance multi-agent debate efficiency. arXiv preprint arXiv:2502.04790, 2025

    2025 arXiv

  49. [57]

    Magent: A many-agent reinforcement learning platform for artificial collective intelligence, 2017

    Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence, 2017. URL https://arxiv.org/abs/1712.00600

    2017 arXiv

  50. [58]

    manager” that sets goals from a “worker

    Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran Li. Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents. arXiv preprint arXiv:2412.13194, 2024. 15 Algorithm 1: Bottom-Up Skill Evol...

    2024 arXiv

  51. [2017]

    URL https://arxiv.org/abs/1703.01161

    arXiv

  52. [2023]

    URL https://arxiv.org/abs/2303.17760

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.