Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that scaling the number of environment interaction steps is a more compute-efficient test-time scaling axis than scaling per-step reasoning for interactive web agents.

desk verdict A promising new axis for agent test-time scaling, but the headline compute-efficiency comparison is undercut by ambiguous token accounting that likely ignores observation tokens. read the letter →

arxiv 2506.07976 v2 pith:PZIGSFXX submitted 2025-06-09 cs.LG cs.AI

Junhong Shen , Hao Bai , Lunjun Zhang , Yifei Zhou , Amrith Setlur , Shengbang Tong , Diego Caples , Nan Jiang
show 3 more authors
Tong Zhang Ameet Talwalkar Aviral Kumar
This is my paper
classification cs.LGcs.AI
keywords test-timescalingwebagentscurriculumreinforcementlearninginteractionhorizonchain-of-thoughtadaptiveexplorationonlinefilteredbehaviorcloningArena
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that interactive agents should be scaled along a new axis: the number of steps they are allowed to interact with the environment, rather than only the amount of reasoning they do per step. On web-agent benchmarks, giving a Gemma 3 12B model more interaction steps improves task success more than spending the same token budget on longer chain-of-thought or best-of-n action sampling. To go beyond prompting, the authors introduce TTI, a curriculum-based online reinforcement learning method that grows the allowed rollout horizon over training and trains the agent with filtered behavior cloning on successful rollouts. TTI reaches state-of-the-art results among open-source agents trained on open data on WebVoyager and WebArena. If the claim is right, interaction is a compute axis that should be traded off against per-step reasoning in agentic settings, not ignored.

What carries the argument

The central object is the interaction horizon $h$, the maximum number of environment steps in a rollout. The mechanism that carries the argument is TTI: each training iteration sets $h$ by a multiplicative schedule ($10, 20, 30, \ldots$), rolls out the agent in the environment, labels each trajectory with a prompting-based verifier, and updates the policy only on successful rollouts by maximizing action log-likelihood, weighting recent trajectories more heavily. This curriculum is what lets the agent consolidate exploitation skills at short horizons before it is exposed to long-horizon exploration, avoiding both premature termination and aimless wandering.

What would settle it

Train TTI on WebArena using the ground-truth evaluator instead of the Gemma 3 27B verifier to filter rollouts; if the gains over fixed-horizon and zero-shot baselines shrink or disappear, the reported advantage is an artifact of verifier bias rather than interaction scaling.

Watch

Extended reading notes

Core claim

Under a fixed token budget, gaining new information by taking more environment steps outperforms spending the same compute on deeper per-step reasoning: on a WebArena subset, interaction scaling raises task success from 23.81% to at least 28%, while per-step budget forcing and best-of-n add less than 3%. Since prompting-based re-checking alone is unstable and can confuse the agent, the paper trains agents with TTI, which uses filtered behavior cloning with a multiplicative curriculum over the maximum horizon ($h = 10, 20, 30$) and a replay buffer weighted toward recent rollouts. The resulting 12B agent scores 64.8% on WebVoyager and 26.1% on full WebArena, the best open-source, open-data results reported on these benchmarks, and its trajectories show it learning to backtrack and search in complex tasks while staying direct in simple ones.

Load-bearing premise

That the prompting-based verifier labels rollouts accurately enough (88.9% agreement with WebArena's ground truth) and that synthetic tasks generated by the base model are representative of real user tasks; if either fails, training may reward behavior that satisfies the verifier rather than genuinely completing the task.

Editorial extensions

If this is right

  • Under a fixed token budget, agents should allocate compute across more interaction steps rather than only within-step reasoning, because each step supplies new information that longer chain-of-thought cannot.
  • Training at a fixed long horizon is not enough: TTI's multiplicative curriculum beats both fixed $h=10$ and fixed $h=30$ training on WebVoyager and WebArena.
  • Agents trained with TTI learn to trade off thinking for acting, shortening per-step reasoning as trajectory length grows.
  • Combining trained interaction scaling with inference-time re-checking adds further gains, especially early in training.
  • Self-improvement with synthetic tasks generated by the base model and filtered by a verifier is sufficient to produce the best open-source, open-data web agent reported on these benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the advantage of interaction scaling comes from information gain under partial observability, the horizon-curriculum idea should transfer to robotics or computer-use domains, with a testable prediction that TTI's margin over per-step compute grows as observations become more incomplete.
  • The verifier's 88.9% agreement with WebArena's ground truth leaves room for part of the reported gains to reflect optimizing the verifier's notion of success; comparing TTI trained with verifier labels against TTI trained with ground-truth labels would separate these.
  • The observed drop in per-step tokens as interaction grows suggests an explicit coordinator that decides, at each step, whether to think longer or act could make the trade-off even more compute-efficient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes scaling test-time interaction, i.e., increasing the number of environment steps an agent may take within a single rollout, as a new dimension of test-time compute scaling for web agents. It first studies a purely inference-time 'check-again' prompt on a 62-task WebArena subset and claims that, under a token budget, interaction scaling outperforms per-step budget forcing and best-of-n action sampling (Finding 1). It then introduces TTI, a curriculum-based online filtered-behavior-cloning method that gradually increases the maximum rollout horizon during training. Using Gemma 3 12B, TTI reports state-of-the-art results among open-weight, open-data agents on WebVoyager (64.8% success) and WebArena (26.1%), with an analysis of learned exploration behaviors such as GoBack and Bing actions. The central thesis is that agents gain new information through interaction more effectively than by deeper reasoning per step.

Significance. If the thesis holds, the paper identifies a genuinely new and practically relevant test-time scaling axis for interactive agents, complementing per-step reasoning scaling. The work is notable for its reproducible setup (released code and data), the controlled design of the initial experiments to isolate horizon effects, and the honest documentation of failure modes. The learning-dynamics analysis showing the emergence of backtracking and search behavior is a valuable contribution. However, the central Finding 1 relies on a compute metric that is not precisely specified, and the main benchmark results are reported without statistical uncertainty, which limits the strength of the empirical claims until these are addressed.

major comments (4)
  1. [§4.2, Figure 3, Finding 1] The manuscript claims that interaction scaling 'beats thinking more a single step' under a fixed compute budget, measured by 'the number of tokens per trajectory' (Section 4.2). It never states whether this count includes input/observation tokens (accessibility tree, screenshot) or only generated tokens. The bottom panel of Figure 3 labels the y-axis 'tokens per step,' and later in Figure 6(c) 'tokens per step' is defined as 'the average CoT length,' both of which are consistent with output-only accounting. In this environment, each interaction step re-encodes a long observation, so excluding input tokens systematically understates the cost of interaction scaling relative to budget forcing and best-of-n, which reuse the same observation. Because Finding 1 is the paper's headline conceptual claim, the authors must either report input+output token counts (or a FLOP proxy) or explicitly reframe the claim as an output-token-efficiency result and re-examine whether the conclusion survives after accounting for re-encoding costs.
  2. [§4.2, Figure 3] The three scaling methods compared in Figure 3 differ not only in the number of interaction steps but also in the prompts and decoding procedures: interaction scaling uses 'check-again' prompts, budget forcing uses 'wait and think deeper' prompts, and best-of-n uses majority voting over sampled actions. The observed differences could therefore be driven by prompt quality or decoding strategy rather than by the number of interaction steps. To isolate the effect of interaction, the paper should include a control that uses the same 'check-again' wording but does not allow an additional environment step (e.g., asking the model to revise its answer within the same step), and a version that uses the budget-forcing prompt but lets the environment state advance. Without such controls, the causal attribution of the gains to interaction is not established.
  3. [§6.1, §6.3, Tables 3 and 4] The full WebVoyager and WebArena results are reported as single numbers from a single training run, with no error bars, confidence intervals, or significance tests. The improvements over fixed-horizon baselines are modest in absolute terms (WebVoyager: 64.8 vs 59.1 for fixed h=10; WebArena: 26.1 vs 23.8 for fixed h=10). The authors already report three-run averages for the 62-task subset (Table 1), so multiple seeds are feasible. Without uncertainty estimates, a reader cannot judge whether the claimed state-of-the-art results and curriculum gains are statistically reliable or due to noise in online RL training.
  4. [§6, Appendix E.1] Training relies on synthetic tasks generated by the base model itself and on a Gemma 3 27B prompting-based verifier with 88.9% agreement against WebArena's ground-truth evaluator. The paper reports only the aggregate agreement and does not analyze whether verifier accuracy varies with trajectory length, which is the very variable the curriculum manipulates. If longer rollouts are systematically mislabeled as successful, the learned preference for exploration could be an artifact of verifier bias rather than genuine task competence. A per-horizon verifier accuracy analysis and a sensitivity study (e.g., training with a stricter success threshold) would substantiate the claim that TTI improves through interaction rather than through verifier gamification.
minor comments (6)
  1. [§4.2 and Figure 3] The terms 'total compute,' 'tokens per trajectory,' and 'tokens per step' are used without a single explicit definition; please define in the main text or a dedicated appendix whether these include input tokens, output tokens, or both.
  2. [§4.1] The statement that the agent 'changes its action ~25% of the time after double-checking' lacks a confidence interval or sample size; please report the statistic over the three runs.
  3. [Table 3] The row 'Agent E' appears without a citation in the table; please add a reference to Agent-E [50] or remove the entry.
  4. [§6.1] The claim 'TTI also obtains the highest SR in 8 out of 13 domains' should clarify the comparison set (open-weight, open-data baselines only), since closed-source and UI-TARS rows exceed TTI in several of those domains.
  5. [Appendix E.1] There is a typo in the evaluator prompt: 'You should explicit consider' should read 'You should explicitly consider.'
  6. [Table 4] The 'Fixed h=30' row has inconsistent decimal formatting (28.57 in Maps vs. two-decimal format elsewhere); please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark results evaluated on external evaluators, not derived from fitted parameters or self-citations.

full rationale

No load-bearing circularity found. The paper's central claims are empirical benchmark results, not derived from its assumptions. The controlled experiments in Section 4 compare interaction scaling, budget forcing, and best-of-n under a common token metric; the comparison is an experiment, not a fit. TTI's training pipeline (Section 6, Appendix E.1) uses synthetic tasks generated by the base model and a separate Gemma 3 27B verifier with 88.9% agreement against WebArena's ground-truth evaluator; this is self-referential in data generation but not definitionally circular, because the headline results are measured on external WebVoyager/WebArena evaluators, and the verifier is a different model from the trained policy. Some citations to co-authored prior work (PAE [19], Digi-RL [20]) supply pipeline components, but the paper specifies the full algorithm and evaluates externally; these citations are not used to justify the empirical conclusions. The compute-accounting question about input vs. output tokens in Figure 3 is a measurement-validity concern, not a circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central results rest on benchmark evaluations and a training pipeline whose reward signal comes from a learned verifier and self-generated tasks; these are domain assumptions rather than first-principles derivations. The horizon schedule is the key hand-chosen design element, and the verifier accuracy and task-generation bias are the most significant unverified premises.

free parameters (4)
  • Interaction horizon schedule (multiplicative) = 10, 20, 20, 30, 30, 30, ...
    Chosen after comparing multiplicative vs additive schedules on a 62-task WebArena subset (Table 2); not derived from theory and selected using a portion of the later evaluation set.
  • Evaluation horizon h = 30
    Set generously above average task length (Section 4); this choice affects how much interaction is possible at test time.
  • Rollout size and update sample count = 512
    Chosen as a compute budget for online RL; affects the variance and coverage of the filtered BC update.
  • Learning rate = 1e-6 (WebArena), 4e-6 (WebVoyager)
    Standard hyperparameter chosen without a systematic search; reported in Appendix D.5 and E.3.
assumptions (5)
  • domain assumption Web navigation is a finite-horizon MDP with binary success reward.
    Section 3 formalizes the environment. If the reward or transition structure is not well-behaved, the filtered BC training signal is unreliable.
  • domain assumption Observation history truncated to the last three steps plus all past actions is sufficient for successful interaction.
    Section 3 states this truncation. Long-horizon credit assignment and information gathering may require more context than three observations.
  • domain assumption The synthetic task distribution generated by the base model is representative of real benchmark tasks.
    Section 6 uses PAE-style task generation. If the generated tasks are narrow or unrepresentative, generalization claims are weakened.
  • domain assumption The prompting-based verifier (Gemma 3 27B) accurately labels task success.
    Section 6 reports 88.9% agreement with WebArena ground truth. Verifier errors directly corrupt the filtered BC update, and the failure modes of the verifier are not analyzed.
  • domain assumption The six-action discrete action space is sufficient to solve all benchmark tasks.
    Section 3 excludes compound actions like goto[url]. If some tasks require richer action primitives, the measured performance ceiling is lower.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction." pith.science (2026). https://pith.science/paper/PZIGSFXX

@misc{pith2026250607976,
  author       = {Pith},
  title        = {Pith review of: Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZIGSFXX}},
  note         = {Machine review of arXiv:2506.07976}
}
read the original abstract

The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

    cs.CL 2026-07 conditional novelty 7.5 of 10

    Across ~38,000 hours on 134 ultra-long real-world tasks, aggregate agent performance follows a log-sigmoid of interaction time, and measured learning speed doubles about every three months.

  2. SSRL: Self-Search Reinforcement Learning

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    SSRL, a training pipeline that uses an LLM's own repeated sampling as a search environment for RL, improves question answering without external tools and transfers to real search engines.

Reference graph

Works this paper leans on

124 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    Webvoyager: Building an end-to-end web agent with large multimodal models,

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models,

  2. [2]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=oKn9c6ytLx

    2024

  3. [3]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024

    Claude. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024. URL https://www.anthropic.com/news/3-5-models-and-computer-use

    2024

  4. [4]

    Introducing operator, 2025

    OpenAI. Introducing operator, 2025. URL https://openai.com/index/ introducing-operator/

    2025

  5. [5]

    Browser use: Enable ai to control your browser, 2024

    Magnus Müller and Gregor Žunič. Browser use: Enable ai to control your browser, 2024. URL https://github.com/browser-use/browser-use

    2024

  6. [6]

    Cogagent: A visual language model for gui agents, 2023

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023

    2023

  7. [7]

    Your code’s new collaborator, 2025

    Claude. Your code’s new collaborator, 2025. URL https://www.anthropic.com/ claude-code

    2025

  8. [8]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch...

    2025

Show all 124 references
  1. [9]

    Fine-tuning large vision-language models as decision- making agents via reinforcement learning.ArXiv, abs/2405.10292, 2024

    Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision- making agents via reinforcement learning.ArXiv, abs/2405.10292, 2024. URLhttps://api....

    2024 arXiv

  2. [10]

    Emergence of pragmatics from referential game between theory of mind agents, 2021

    Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Zhu. Emergence of pragmatics from referential game between theory of mind agents, 2021. URLhttps://arxiv. org/abs/2001.07752

    2021 arXiv

  3. [11]

    Iterative teacher-aware learning

    Luyao Yuan, Dongruo Zhou, Junhong Shen, Jingdong Gao, Jeffrey L Chen, Quanquan Gu, Ying Nian Wu, and Song-Chun Zhu. Iterative teacher-aware learning. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Process...

    2021

  4. [12]

    Mixture- of-mamba: Enhancing multi-modal state-space models with modality-aware sparsity, 2025

    Weixin Liang, Junhong Shen, Genghan Zhang, Ning Dong, Luke Zettlemoyer, and Lili Yu. Mixture- of-mamba: Enhancing multi-modal state-space models with modality-aware sparsity, 2025. URL https://arxiv.org/abs/2501.16295

    2025 arXiv

  5. [13]

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models.ArXiv, abs/2502.19417, 2025

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchic...

    2025 arXiv

  6. [14]

    Scribeagent: Towards specialized web agents using production-scale workflow data.ArXiv, abs/2411.15004, 2024

    Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. Scribeagent: Towards specialized web agents using production-scale workflow data.ArXiv, abs/2411.15004, 2024. URL https://api.semanticscholar.org/CorpusID: 274192657

    2024 arXiv

  7. [15]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URLhttps:// openreview.net...

    2023

  8. [16]

    Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar

    Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: align then refine. InProceedings of the 40th International Conference on Machine Learning, 2023

    2023

  9. [17]

    Tag-llm: Repurposing general-purpose llms for specialized domains, 2024

    Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-llm: Repurposing general-purpose llms for specialized domains, 2024

    2024

  10. [18]

    Agudelo, Peter Qian, and Tianlong Chen

    Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Leandro Z. Agudelo, Peter Qian, and Tianlong Chen. Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms.ArXiv, abs/2502.07942, 2025

    2025 arXiv

  11. [19]

    Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran L. Li. Proposer-agent-evaluator(pae): Autonomous skill discovery for foundation model internet agents.ArXiv, abs/2412.13194, 2024

    2024 arXiv

  12. [20]

    Di- girl: Training in-the-wild device-control agents with autonomous reinforcement learning.ArXiv, abs/2406.11896, 2024

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- girl: Training in-the-wild device-control agents with autonomous reinforcement learning.ArXiv, abs/2406.11896, 2024

    2024 arXiv

  13. [21]

    Li, Sergey Levine, and Aviral Kumar

    Hao Bai, Yifei Zhou, Erran L. Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q-value functions for training device-control agents.ArXiv, abs/2502.15760, 2025

    2025 arXiv

  14. [22]

    Junhong Shen and Lin F. Yang. Theoretically principled deep rl acceleration via nearest neighbor function approximation.Proceedings of the AAAI Conference on Artificial Intelligence, 35(11):9558– 9566, May 2021. doi: 10.1609/aaai.v35i11.17151. URL https://ojs.aaai.org/index. p...

    2021 doi

  15. [23]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025. URL https:/...

    2025 arXiv

  16. [24]

    Claude takes research to new places, 2025

    Claude. Claude takes research to new places, 2025. URL https://www.anthropic.com/ news/research

    2025

  17. [25]

    Introducing deep research, 2025

    OpenAI. Introducing deep research, 2025. URL https://openai.com/index/ introducing-deep-research/

    2025

  18. [26]

    Gemini deep research, 2025

    Google Gemini. Gemini deep research, 2025. URL https://gemini.google/overview/ deep-research/?hl=en

    2025

  19. [27]

    Zettlemoyer, Percy Liang, Emmanuel J

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Li, Hanna Hajishirzi, Luke S. Zettlemoyer, Percy Liang, Emmanuel J. Candes, and Tatsunori Hashimoto. s1: Simple test- time scaling.ArXiv, abs/2501.19393, 2025. URL https://api.semanticscholar.org/ CorpusID:276079693

    2025 arXiv

  20. [28]

    Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/ forum...

    2025

  21. [29]

    Inference-aware fine- tuning for best-of-n sampling in large language models.ArXiv, abs/2412.15287, 2024

    Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine- tuning for best-of-n sampling in large language models.ArXiv, abs/2412.15287, 2024. URL https://...

    2024

  22. [30]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? ...

    2025

  23. [31]

    Webglm: Towards an efficient web-enhanced question answering system with human preferences, 2023

    Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. Webglm: Towards an efficient web-enhanced question answering system with human preferences, 2023

    2023

  24. [32]

    Multimodal web navigation with instruction-finetuned foundation models,

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models,

  25. [33]

    Agentoccam: A simple yet strong baseline for llm-based web agents, 2024

    Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm-based web agents, 2024. URLhttps://arxiv.org/abs/2410.13825

    2024 arXiv

  26. [34]

    URLhttps://arxiv.org/abs/2305.11854

    arXiv

  27. [35]

    UPS: Efficiently building foundation models for PDE solving via cross-modal adaptation.Transactions on Machine Learning Research, 2024

    Junhong Shen, Tanya Marwah, and Ameet Talwalkar. UPS: Efficiently building foundation models for PDE solving via cross-modal adaptation.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URLhttps://openreview.net/forum?id=0r9mhjRv1E

    2024

  28. [36]

    Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration, 2024

    Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration, 2024. URL https://arxiv.org/abs/2408.15978. 17 Thinking vs. Doing: Agents that Reason by Scaling Test-T...

    2024 arXiv

  29. [37]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.ArXiv, abs/2412.09605, 2024

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.ArXiv, abs/2412.09605, 2024

    2024 arXiv

  30. [38]

    Cat: Content-adaptive image tokenization, 2025

    Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, and Chunting Zhou. Cat: Content-adaptive image tokenization, 2025. URLhttps://arxiv.org/ abs/2501.03120

    2025 arXiv

  31. [39]

    Specialized foundation models struggle to beat supervised baselines, 2024

    Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, and Mikhail Khodak. Specialized foundation models struggle to beat supervised baselines, 2024. URL https://arxiv.org/abs/2411.02796

    2024 arXiv

  32. [40]

    Beyond browsing: Api-based web agents, 2024

    Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents, 2024. URLhttps://arxiv.org/abs/2410.16464

    2024 arXiv

  33. [41]

    Codepde: An inference framework for llm-driven pde solver generation, 2025

    Shanda Li, Tanya Marwah, Junhong Shen, Weiwei Sun, Andrej Risteski, Yiming Yang, and Ameet Talwalkar. Codepde: An inference framework for llm-driven pde solver generation, 2025. URL https://arxiv.org/abs/2505.08783

    2025

  34. [42]

    Mathematicalreconstruction of patient-specific vascular networks based on clinical images and global optimization.IEEE Access, 9:20648–20661, 2021

    JunhongShen, AbdulHannanFaruqi, YifanJiang, andNimaMaftoon. Mathematicalreconstruction of patient-specific vascular networks based on clinical images and global optimization.IEEE Access, 9:20648–20661, 2021. doi: 10.1109/ACCESS.2021.3052501

    2021

  35. [43]

    Autonomous evaluation and refinement of digital agents.arXiv preprint arXiv:2404.06474, 2024

    Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents.arXiv preprint arXiv:2404.06474, 2024

    2024 arXiv

  36. [44]

    Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, and Yu Su

    Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, and Yu Su. Skillweaver: Web agents can self-improve by discovering and honing skills. 2025. URLhttps://api.semanticscholar. org/Corp...

    2025

  37. [45]

    NAS-bench-360: Benchmarking neural architecture search on diverse tasks

    Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-bench-360: Benchmarking neural architecture search on diverse tasks. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022

    2022

  38. [46]

    Efficient architecture search for diverse tasks

    Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Efficient architecture search for diverse tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  39. [47]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URLhttps://arxiv.org/abs/2303.08774

    2024 arXiv

  40. [48]

    Autoguide: Automated generation and selection of state-aware guidelines for large language model agents.arXiv preprint arXiv:2403.08978, 2024

    Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guidelines for large language model agents.arXiv preprint arXiv:2403.08978, 2024

    2024 arXiv

  41. [49]

    Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked llm policies for web actions. InConference on Language Modeling (COLM), 2024. URLhttps://arxiv.org/ abs/2310.03720

    2024 arXiv

  42. [50]

    Introducing the next generation of claude, 2024

    Anthropic. Introducing the next generation of claude, 2024. URLhttps://www.anthropic. com/news/claude-3-family. 18 Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

    2024

  43. [51]

    L2g: Repurposing language models for genomics tasks.bioRxiv, 2024

    Wenduo Cheng, Junhong Shen, Mikhail Khodak, Jian Ma, and Ameet Talwalkar. L2g: Repurposing language models for genomics tasks.bioRxiv, 2024. doi: 10.1101/2024.12.09.627422. URL https://www.biorxiv.org/content/early/2024/12/11/2024.12.09.627422

    2024 doi

  44. [52]

    Agent-e: From autonomous web navigation to foundational design principles in agentic systems

    Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. ArXiv, abs/2407.13032, 2024

    2024 arXiv

  45. [53]

    Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Fau...

    2024 arXiv

  46. [54]

    Plan-and-act: Improving planning of agents for long-horizon tasks, 2025

    Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anu- manchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks, 2025

    2025

  47. [55]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=Pc8AU1aF5e

    2024

  48. [56]

    Tree search for language model agents.arXiv preprint arXiv:2407.01476, 2024

    Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents.arXiv preprint arXiv:2407.01476, 2024

    2024

  49. [57]

    Infogent: An agent-based framework for web information aggregation.ArXiv, abs/2410.19054, 2024

    Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji. Infogent: An agent-based framework for web information aggregation.ArXiv, abs/2410.19054, 2024

    2024 arXiv

  50. [58]

    Agent workflow memory.arXiv preprint arXiv:2409.07429, 2024

    Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory.arXiv preprint arXiv:2409.07429, 2024

    2024 arXiv

  51. [59]

    Shikhar Murty, Dzmitry Bahdanau, and Christopher D. Manning. Nnetscape navigator: Complex demonstrations for web agents without a demonstrator, 2024. URLhttps://arxiv.org/abs/ 2410.02907

    2024 arXiv

  52. [60]

    Bagel: Boot- strapping agents by guiding exploration with language.arXiv preprint arXiv:2403.08140, 2024

    Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Boot- strapping agents by guiding exploration with language.arXiv preprint arXiv:2403.08140, 2024

    2024 arXiv

  53. [61]

    Agent q: Advanced reasoning and learning for autonomous ai agents.ArXiv, abs/2408.07199, 2024

    PranavPutta, EdmundMills, NamanGarg, SumeetRameshMotwani, ChelseaFinn, DivyanshGarg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents.ArXiv, abs/2408.07199, 2024. URLhttps://api.semanticscholar.org/CorpusID:271865516. 19 Thinking vs. Doing...

    2024 arXiv

  54. [62]

    To- wards internet-scale training for agents.ArXiv, abs/2502.06776, 2025

    Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. To- wards internet-scale training for agents.ArXiv, abs/2502.06776, 2025. URL https://api. semanticscholar.org/CorpusID:276249229

    2025 arXiv

  55. [63]

    Distrl: An asyn- chronous distributed reinforcement learning framework for on-device control agents.ArXiv, abs/2410.14803, 2024

    Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asyn- chronous distributed reinforcement learning framework for on-device control agents.ArXiv, abs/2410.14803, 2024. URLhttps://api.semanticscholar.org/CorpusID:273501605

    2024 arXiv

  56. [65]

    Autowebglm: A large language model-based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery an...

    2024

  57. [66]

    Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arik. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments.ArXiv, abs/2501.10893, 2025

    2025 arXiv

  58. [67]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

  59. [68]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.ArXiv, abs/2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.ArXiv, abs/2408.03314, 2024. URL https://api.semanticscholar.org/CorpusID:271719990

    2024 arXiv

  60. [69]

    Rewarding progress: Scaling automated process verifiers for llm reasoning.ArXiv, abs/2410.08146, 2024

    Amrith Rajagopal Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning.ArXiv, abs/2410.08146, 2024. URL https: //api.semant...

    arXiv

  61. [70]

    URLhttps://api.semanticscholar.org/CorpusID:271601023

  62. [71]

    Training verifiers to solve math word problems.ArXiv, abs/2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.ArXiv, abs/2110.14168, 2021. URLhttps: //api.se...

    arXiv

  63. [72]

    Yuxiao Qu, Matthew Y. R. Yang, Amrith Rajagopal Setlur, Lewis Tunstall, Edward Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning.ArXiv,abs/2503.07572,2025. URL https://api.semanticscholar.org/CorpusID: 276928248

    2025 arXiv

  64. [73]

    Scaling test- time compute without verification or rl is suboptimal.ArXiv, abs/2502.12118, 2025

    Amrith Rajagopal Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test- time compute without verification or rl is suboptimal.ArXiv, abs/2502.12118, 2025. URL https://api.semanticscholar.org/CorpusID:276422443

    2025 arXiv

  65. [74]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learn- ing.ArXiv, abs/2501.12948, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learn- ing.ArXiv, abs/2501.12948, 2025. URL https://api.semanticscholar.org/CorpusID: 275789950

    2025 arXiv

  66. [75]

    Exact: Teachingaiagentstoexplorewithreflective-mctsandexploratorylearning.ArXiv, abs/2410.02052,

    XiaoYu, BaolinPeng, VineethVajipey, HaoCheng, MichelGalley, JianfengGao, andZhouYu. Exact: Teachingaiagentstoexplorewithreflective-mctsandexploratorylearning.ArXiv, abs/2410.02052,

    arXiv

  67. [76]

    Doing: Agents that Reason by Scaling Test-Time Interaction Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, 20 Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction Li Fei-Fei, Lijuan ...

    2025

  68. [77]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URLhttps://arxiv.org/ abs/2210.03629

    2023 arXiv

  69. [78]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.ArXiv, abs/2402.03300, 2024

    2024 arXiv

  70. [79]

    URLhttps://api.semanticscholar.org/CorpusID:273098809

  71. [80]

    Generative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240, 2024

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240, 2024

    2024 arXiv

  72. [81]

    Metaxas, and Tong Che

    Can Jin, Hongwu Peng, Qixin Zhang, Yujin Tang, Dimitris N. Metaxas, and Tong Che. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning. 2025. URL https://api.semanticscholar.org/CorpusID:277781795

    2025

  73. [82]

    Gemma 3 technical report.ArXiv, abs/2503.19786, 2025

    Gemma Team. Gemma 3 technical report.ArXiv, abs/2503.19786, 2025. URLhttps://api. semanticscholar.org/CorpusID:277313563

    2025 arXiv

  74. [83]

    Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for llms via reinforcement learning, 2025. URLhttps://arxiv.org/abs/ 2503.19470

    2025 arXiv

  75. [84]

    Think twice, act once: A co-evolution framework of llm and rl for large-scale decision making, 2025

    Xu Wan, Wenyue Xu, Chao Yang, and Mingyang Sun. Think twice, act once: A co-evolution framework of llm and rl for large-scale decision making, 2025. URLhttps://arxiv.org/abs/ 2506.02522

    2025 arXiv

  76. [85]

    Inducingprogrammaticskills for agentic tasks

    ZoraZhiruoWang,ApurvaGandhi,GrahamNeubig,andDanielFried. Inducingprogrammaticskills for agentic tasks. 2025. URLhttps://api.semanticscholar.org/CorpusID:277634286

    2025

  77. [86]

    Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, ICML 2024

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, ICML 2024

    2024

  78. [87]

    Recursive introspection: Teaching language model agents how to self-improve.Advances in Neural Information Processing Systems, 37:55249–55285, 2024

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve.Advances in Neural Information Processing Systems, 37:55249–55285, 2024

    2024

  79. [88]

    Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    1999

  80. [89]

    Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022. 21 Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

    2022

  81. [90]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009. URLhttps://api.semanticscholar. org/CorpusID:873046

    2009

  82. [91]

    Bayen, Sham M

    Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M. Bayen, Sham M. Kakade, Igor Mordatch, and P. Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines.ArXiv, abs/1803.07246, 2018. URL https://api.semanticscholar. org/CorpusID:4043645

    2018 arXiv

  83. [92]

    Analysis and improvement of policy gradient estimation.Neural networks : the official journal of the International Neural Network Society, 26:118–29, 2011

    Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. Analysis and improvement of policy gradient estimation.Neural networks : the official journal of the International Neural Network Society, 26:118–29, 2011. URLhttps://api.semanticscholar.org/CorpusID:2274728

    2011

  84. [93]

    Policy gradients with variance related risk criteria

    Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. arXiv: Learning, 2012. URLhttps://api.semanticscholar.org/CorpusID:3109162

    2012

  85. [94]

    Abbeel, and Wojciech Zaremba

    Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NeurIPS, 2017. URLhttps://api.semanticscholar.org/CorpusID:3532908

    2017

  86. [95]

    Taylor, and Peter Stone

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey.ArXiv, abs/2003.04960, 2020. URLhttps://api.semanticscholar.org/CorpusID:212657666

    2003 arXiv

  87. [96]

    A survey on curriculum learning.IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44:4555–4576, 2021

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning.IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44:4555–4576, 2021. URLhttps://api. semanticscholar.org/CorpusID:232362223

    2021

  88. [97]

    Teacher–student curriculum learning.IEEE Transactions on Neural Networks and Learning Systems, 31:3732–3740, 2017

    Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum learning.IEEE Transactions on Neural Networks and Learning Systems, 31:3732–3740, 2017. URL https://api.semanticscholar.org/CorpusID:8432394

    2017

  89. [98]

    Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446, 2024

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446, 2024

    2024 arXiv

  90. [99]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems ...

    2023

  91. [100]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters.Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020. URL...

    2020

  92. [101]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025. 22 Thinking vs. Doing: Agents that Reason...

    2025 arXiv

  93. [102]

    Shikhar Murty, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild. 2024. URL https://api. semanticscholar.org/CorpusID:273162280

    2024

  94. [103]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, nathan lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.ArXiv, abs/2503.01307, 2025. URLhttps://api.semanticscholar.org/CorpusID:276741915

    2025 arXiv

  95. [104]

    Towards enterprise-ready computer using generalist agent.ArXiv, abs/2503.01861,

    Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Aviad Sela, Asaf Adi, and Nir Mashkif. Towards enterprise-ready computer using generalist agent.ArXiv, abs/2503.01861,

    arXiv

  96. [106]

    Awa 1.5 achieves breakthrough performance on we- barena benchmark, 2024

    JaceAI. Awa 1.5 achieves breakthrough performance on we- barena benchmark, 2024. URL https://www.jace.ai/post/ awa-1-5-achieves-breakthrough-performance-on-webarena-benchmark

    2024

  97. [108]

    Dashboard

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017. 23 Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction Appendices A. Broader Impact This work contributes to t...

    2017 arXiv

  98. [109]

    25 Thinking vs

    Click a web element. 25 Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

  99. [113]

    STRICTLY follow the format

    Answer. This action should only be chosen when all questions in the task have been solved. Correspondingly, action should STRICTLY follow the format specified by one of the following lines: Click [numerical_label] Type [numerical_label] [content] Scroll [up/down] GoBack ANSWER...

  100. [114]

    Doing: Agents that Reason by Scaling Test-Time Interaction the online activity to be carried out

    Web Task Instruction: This is a clear and specific directive provided in natural language, detailing 32 Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction the online activity to be carried out

  101. [115]

    It serves as textual result in response to the instruction

    Result Response: This is a textual response obtained after the execution of the web task. It serves as textual result in response to the instruction

  102. [116]

    It serves as visual proof of the actions taken in response to the instruction

    Result Screenshots: This is a visual representation of the screen showing the result or interme- diate state of performing a web task. It serves as visual proof of the actions taken in response to the instruction. – You SHOULD NOT make assumptions based on information not pres...

  103. [117]

    Delete existing content in a textbox and then type content

  104. [118]

    Scroll up or down the whole window

  105. [119]

    Go back, returning to the previous webpage

  106. [120]

    Navigate to Bing’s homepage

  107. [121]

    This action should only be chosen when all questions in the task have been solved

    Answer. This action should only be chosen when all questions in the task have been solved. Correspondingly, action should STRICTLY follow the format specified by one of the following lines: Click [numerical_label] Type [numerical_label] [content] Scroll [up/down] GoBack Bing A...

  108. [122]

    Sometimes, accessibility tree or screenshot captures more elements than the other, but it’s fine to use either one

    The predicted action should be based on elements as long as it’s accessibility tree OR screenshot. Sometimes, accessibility tree or screenshot captures more elements than the other, but it’s fine to use either one

  109. [123]

    After typing, the system automatically hits ’ENTER’ key

    To input text for search bars, no need to click textbox first, directly type content. After typing, the system automatically hits ’ENTER’ key

  110. [124]

    Double check the formatting requirements in the task when ANSWER

    When a complex task involves multiple questions or steps, select ’ANSWER’ only at the very end, after addressing all of these questions or steps. Double check the formatting requirements in the task when ANSWER. Always think twice before using ’ANSWER’ action!!!

  111. [125]

    When specifying the content for ’Type’ and ’ANSWER’ actions, be sure to wrap the content with ’[]’

  112. [126]

    Use ‘GoBack‘ to return to the previous state, use it when you find the previous action incorrect

  113. [127]

    When you see a pop-up page, you should immediately ‘GoBack‘ to the previous page

  114. [128]

    STRICTLY follow the format

    Use ‘Bing‘ when you need to navigate to a different website or search for new information. Your reply should strictly follow the format: Thought: Your reasoning trace. A good practice is to follow this format: - Observation summary: where are you at now? list all elements that...

    2022

  115. [2024]

    URLhttps://arxiv.org/abs/2401.13919

    arXiv

  116. [2025]

    URLhttps://api.semanticscholar.org/CorpusID:276775684

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.