Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ARPO: replaying successful trajectories lets GRPO train a vision-language GUI agent end to end, raising OSWorld success from 23.5% to 29.9% and OSWorld Hard from 18.2% to 23.8%.

desk verdict ARPO is a useful RL recipe for GUI agents, but the replay-buffer off-policy issue is real and needs addressing before the headline gains can be taken at face value. read the letter →

arxiv 2505.16282 v1 pith:AA7GL5KX submitted 2025-05-22 cs.CV

Fanbin Lu , Zhisheng Zhong , Shu Liu , Chi-Wing Fu , Jiaya Jia This is my paper
classification cs.CV
keywords GUIagentsreinforcementlearningGroupRelativePolicyOptimizationexperiencereplayvision-languagemodelsOSWorldsparserewardslong-horizontasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that reinforcement learning can train a vision-language model to operate a real desktop GUI over long action sequences, a setting where rewards are sparse, feedback is delayed, and every rollout is expensive. Its method, ARPO, takes Group Relative Policy Optimization (GRPO), a policy-gradient algorithm that scores each response against other sampled responses, and adds a replay buffer that injects a previously successful trajectory whenever a whole training group fails, plus a task filter that keeps only tasks a baseline agent can sometimes solve. If the central claim holds, ARPO raises the UI-Tars-1.5 model's success rate on OSWorld from 23.5% to 29.9%, and on the paper's stricter OSWorld Hard protocol from 18.2% to 23.8%. This matters because most GUI agents are trained by imitation, and these results are evidence that sparse, delayed rule-based rewards from a live environment can drive end-to-end policy improvement.

What carries the argument

The load-bearing object is the experience replay buffer operating inside GRPO. GRPO computes token-level advantages by normalizing rewards within a group of sampled responses, so when every response in a group has zero reward, all advantages are zero and no gradient flows. The buffer stores one successful trajectory per task and, whenever a training group contains no successes, swaps in a stored successful trajectory so the group has nonzero reward variance. The second mechanism is the task-selection filter, which pre-runs each OSWorld task with the base agent and retains only tasks with at least one successful rollout among 16 attempts, yielding a 128-task training set with informative reward signals. Together the two mechanisms keep the GRPO advantage estimate nonzero even in sparse-reward GUI environments, and the 256-environment parallel rollout system makes collecting the underlying trajectories affordable.

What would settle it

Retrain ARPO with replayed trajectories either removed or reweighted by the current-policy probability ratio, and compare the final OSWorld success rate to the reported 29.9%; if the gain vanishes, the replay effect is an artifact of off-policy advantage bias rather than a genuine policy improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that a vision-language GUI agent can be trained end-to-end with rule-based rewards by extending GRPO with two mechanisms: a per-task replay buffer that swaps a stored successful trajectory into any GRPO group whose current rollouts all scored zero, and a task-selection filter that keeps only tasks a baseline agent completes at least once out of 16 attempts. On the OSWorld benchmark, applying this recipe to the UI-Tars-1.5 base model raises success from 23.5% to 29.9%, and from 18.2% to 23.8% on the authors' stricter OSWorld Hard variant, which forbids replacing the final action with FAIL when the step limit is reached. The replay buffer carries most of the gain: in-domain success on a 32-task subset climbs from 68.8% with GRPO alone to 81.25% with ARPO, and the average training reward ends at 0.75 versus 0.65. Out-of-domain performance stays roughly flat, at 56.3% versus 55.2% for the base model, which the paper reads as evidence that generalization still depends on task diversity and scale.

Load-bearing premise

The method assumes that a replayed successful trajectory, produced by an older version of the policy, can be used as an ordinary member of a GRPO training group without correcting for the policy change; if that assumption fails, the reported gains could come from biased advantage estimates rather than genuine policy improvement.

Editorial extensions

If this is right

  • ARPO improves UI-Tars-1.5 from 23.5% to 29.9% on standard OSWorld and from 18.2% to 23.8% on the stricter OSWorld Hard protocol.
  • Applying ARPO to an earlier UI-Tars-7B-DPO base lifts OSWorld success from 15.6% to 20.4%, so the gain is not specific to one checkpoint.
  • Replaying a successful trajectory whenever a GRPO group contains only failures keeps advantages nonzero, raising final average trajectory reward from 0.65 to 0.75.
  • On a 32-task in-domain subset, ARPO reaches 81.25% success versus 68.8% for GRPO and 43.8% for the base model, while out-of-domain performance stays near the base level.
  • On equal training data, trajectory-level policy optimization (ARPO at 27.3%) outperforms offline preference methods, whose best result (KTO) is 24.6%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The replay mechanism should transfer to other sparse-reward agentic settings, such as web navigation or robotics, whenever successful trajectories are rare and costly; the paper does not test this.
  • Because the reported RL gains are mostly in-domain, task diversity and compute may be the binding constraints for GUI agents, not the optimization algorithm; a larger, more varied task pool would put that to the test.
  • The stricter OSWorld Hard protocol, which removes the FAIL-at-step-limit hack, would make future GUI-agent comparisons more meaningful if adopted as a reporting standard.
  • An importance-corrected replay buffer, one that downweights old trajectories by their probability under the current policy, would isolate the genuine replay benefit from any off-policy bias in the advantages; the paper does not run this control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes ARPO, a reinforcement learning method for vision-language GUI agents that augments GRPO with a task selection strategy and an experience replay buffer. The method is applied to the UI-Tars-1.5 base model and evaluated on the OSWorld benchmark, reporting a success rate of 29.9% on standard OSWorld and 23.8% on a stricter 'OSWorld Hard' protocol, improving over the base model by 6.4% and 5.6%. The authors also present ablations on the replay buffer, task selection, comparison with offline preference optimization, and an analysis of out-of-domain generalization.

Significance. If the reported gains are robust, ARPO would be a meaningful step toward training GUI agents with end-to-end RL in live desktop environments, addressing sparse rewards and high rollout costs. The paper includes several useful engineering contributions: a distributed rollout system, a replay buffer design tailored to sparse-reward settings, and a task selection heuristic. The authors also release code and models, which supports reproducibility. However, the current experimental evidence has important caveats that are discussed below.

major comments (5)
  1. [Sec. 3.4 and Sec. 4.1] The replay buffer injects off-policy trajectories into GRPO training groups without any importance correction or consistency argument. In Sec. 3.1, the GRPO advantage is defined as (r_i - mu)/sigma, where mu and sigma are computed over a group of responses sampled from pi_old for the same query. Sec. 3.5 states that when all sampled rollouts for a task fail, a stored successful trajectory tau+ from an earlier policy is randomly substituted into the group. This violates the i.i.d.-from-pi_old assumption: the replayed trajectory is not a sample from the current behavior policy, and the group statistics are now dominated by an injected high-reward sample. The policy ratio for tau+ in the GRPO objective is therefore evaluated with respect to the wrong pi_old, and the advantages for both the replayed and the sampled trajectories are biased. The paper's own justification that this 'ensures meaningful reward variance' (Sec. 3.5) makes the mechanism load-bearing, but no importance weight, off-policy correction, or consistency proof is supplied. The reported gains over GRPO (Table 1: 29.9% vs. 26.0% on OSWorld; Fig. 4: 81.25% vs. 68.8% in-domain) could be at least partly an artifact of this biased advantage estimator. The authors should either provide a corrected estimator (e.g., importance weighting or a conservative baseline) or empirically demonstrate that the replay mechanism does not distort the policy gradient (for example, by comparing against a variant that uses the replayed trajectory only as a reference for reward normalization without updating the policy on it).
  2. [Sec. 4.4, Fig. 4] The training set is a subset of the evaluation benchmark. The task selection procedure in Sec. 3.4 selects 128 tasks from OSWorld that the baseline UI-Tars-1.5 can solve at least once in 16 rollouts, and training is performed on these tasks (Sec. 4.1). Evaluation, however, is on the full OSWorld benchmark (369 tasks). This creates a train/test overlap: the model is trained on a portion of the same benchmark it is evaluated on, whereas all comparison methods in Table 1 were not trained on OSWorld at all. Consequently, the headline improvements over UI-Tars-1.5 and other baselines may be inflated by the model's exposure to the evaluation task distribution. The paper should either evaluate on a held-out set of tasks that are disjoint from training (e.g., the remaining 241 OSWorld tasks, or another GUI benchmark), or compare against methods that were also trained on the same 128-task subset. The internal OOD analysis in Sec. 4.4 does not resolve this issue because the 'OOD' split is within the already-selected 128 tasks, as discussed in the next comment.
  3. [Sec. 4.2-4.8, Tables 1 and Figs. 3-7] The out-of-domain generalization claim is not supported by the reported experiment. The paper selects 32 tasks from the training task set as 'in-domain' and uses the remaining 96 as 'OOD', but all 128 tasks were selected by the same baseline-success criterion (Sec. 3.4). This is a split within a curated, easy-task distribution, not a genuinely out-of-domain evaluation. Moreover, the OOD comparison shows ARPO at 56.3% versus the base model's 55.2%, a 1.1-point difference. No error bars, standard deviations, or multiple seeds are reported, so this difference is within the likely noise range. The statement that ARPO 'recovers generalization capability' (Sec. 4.4) is therefore an overinterpretation. The authors should evaluate on tasks that were not used in training and were not selected by the same criterion, and report variance across seeds or runs.
  4. [Sec. 4.6, Fig. 6] No experiment reports error bars, standard deviations, or multiple seeds. All numbers in Table 1, Figure 4, Figure 6, and the qualitative analysis in Sec. 4.8 are single-run results. Given that several key comparisons are small (e.g., ARPO vs. GRPO on OSWorld: 29.9% vs. 26.0%; OOD: 56.3% vs. 55.2%), it is impossible to assess whether these differences are statistically meaningful. This is particularly important because the replay buffer and task selection are stochastic procedures that could introduce run-to-run variance. The authors should report mean and variance over at least 3 seeds for the main results and ablations, or provide a clear justification for why variance is negligible.
  5. [Sec. 4.6, Fig. 6] The comparison with offline preference optimization methods (DPO, KTO, reject sampling) in Sec. 4.6 is not sufficiently specified to be reproducible. The text says 'all methods are trained on the same task set with an equal number of rollouts,' but it does not state whether the same 128-task selected set is used, how DPO pairs are constructed from multi-turn trajectories, or how KTO's binary labels are derived from the scalar OSWorld rewards (which are not binary). Without these details, it is unclear whether the higher performance of ARPO/GRPO reflects a genuine advantage of on-policy RL or a mismatch in training setups. Please provide the exact training configurations for these baselines, including the reward preprocessing and the number of training steps.
minor comments (5)
  1. [Table 1] The header row contains a typo: 'UI-Tars-7B-1.5 + GPRO' should read 'GRPO'. Please correct this throughout the table.
  2. [Fig. 8] The trajectory in Figure 8 is displayed in reverse chronological order, with 'Action #3' appearing before 'Action #2' and 'Action #1'. This is confusing for the reader. Please reorder the actions chronologically or add a clear time indicator, and align the thoughts with their corresponding actions.
  3. [Sec. 3.1 and Sec. 4.1] The GRPO objective in Sec. 3.1 uses a single clip parameter epsilon, but Sec. 4.1 states that the clipping parameters are set to epsilon_low = 0.2 and epsilon_high = 0.3, following DAPO. Please clarify this asymmetry in the objective or define both parameters in Sec. 3.1.
  4. [Sec. 3.4] In the Reward Design paragraph, the sentence 'A reward of rt = 1 is assigned if ... and ri = 0 otherwise' uses inconsistent subscripts (rt and ri). Please use the same symbol for the trajectory reward throughout.
  5. [Fig. 2] In Figure 2, the condition 'if σr=0' is not explained in the caption or in the text. Please clarify in the caption that the replay buffer is used when the group reward variance is zero (i.e., all sampled rewards are equal).

Circularity Check

2 steps flagged · score 2.0 of 10

Minor selection-induced self-reference in task filtering; central ARPO result is empirical and not formally circular.

  1. other [Sec. 3.4 (Valuable Tasks Selection) and Sec. 4.2 (Table 1 / OSWorld results)]
    "we evaluate each task in OSWorld using the UI-Tars-1.5 model, performing 16 rollouts per task. A task is retained in the GRPO training set if the agent completes it successfully in at least one of these attempts. This method yields a curated set of 128 tasks... applying ARPO to the UI-Tars-1.5 base model results in a success rate of 29.9% on the standard OSWorld setting and 23.8% on the stricter OSWorld Hard variant—improving upon the original UI-Tars-1.5 model by 6.4% and 5.6%, respectively."

    The training distribution is selected by running the very baseline model that ARPO is compared against, keeping only tasks the baseline can already solve at least once in 16 rollouts. The headline OSWorld performance is then reported on the full OSWorld benchmark, which contains these 128 selected tasks (OSWorld is a fixed 369-task benchmark, Sec. 4.1). Part of the measured gain is therefore an artifact of training and testing on the same easy-for-baseline tasks, rather than a prediction from first principles. The paper itself shows the gain is mostly in-domain (81.25% vs 43.8% base on 32 in-domain tasks, Sec. 4.4), confirming that the selected tasks drive the improvement. This is selection bias / benchmark leakage rather than a formal equation-level circularity, so it is scored as minor.

  2. self definitional [Sec. 3.4 (definition of valuable tasks) vs Sec. 4.5 (Fig. 5b)]
    "we introduce a task filtering procedure to identify a subset of 'valuable' tasks, those capable of producing successful trajectories under a baseline agent... Fig. 5b shows that the standard deviation of rewards within GRPO groups is consistently higher when training on the curated task set."

    The filter is defined as selecting tasks with at least one successful baseline rollout and rejecting tasks that gave no reward in baseline rollouts. Therefore higher within-group reward variance on the curated set is guaranteed by the selection criterion: chosen tasks contain at least one positive trajectory while the discarded tasks are all-zero in the baseline screening. Presenting this variance increase as an empirical benefit of task selection restates the definition of the selection rather than providing independent evidence.

full rationale

No equation in the paper reduces to its own input: GRPO's objective (Sec. 3.1) is standard, the replay-buffer mechanism is a stated algorithmic modification, and the reported OSWorld success rates are measured empirically. The off-policy issue flagged by the skeptic (replaying trajectories from older policies without importance correction) is a real correctness concern about the GRPO estimator, but it is not circularity: the method does not define its result in terms of the replay-buffer advantage. The only self-referential elements are (i) task selection based on baseline solvability combined with evaluation on a benchmark containing those tasks, and (ii) validating task selection by reward variance that the selection rule guarantees. These are minor and do not collapse the central claim, because ARPO's improvement is still an empirical outcome that could in principle have gone the other way; hence score 2.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or ontological entities. It does introduce a new evaluation protocol (OSWorld Hard) and a replay-buffer mechanism, but these are methodological constructs rather than invented entities. The main burdens come from hand-chosen selection thresholds and the unstated replay-buffer capacity, plus the domain assumptions about OSWorld reward reliability and the validity of off-policy replay in GRPO.

free parameters (5)
  • Task selection threshold = at least 1 success in 16 rollouts
    Hand-chosen criterion in Sec. 3.4 that determines the 128-task training set; changing it changes which tasks are trained on and the measured gains.
  • Training task count = 128
    Result of the selection filter; the number is not independently justified as optimal or sufficient.
  • Rollout count per task (group size) = 8
    Chosen in Sec. 4.1 for GRPO groups; it directly affects reward variance and the cost of rollouts.
  • Replay buffer capacity = not reported in the paper
    Sec. 3.5 imposes a fixed-size limit with oldest-entry eviction but never reports the capacity, leaving a hidden hyperparameter that affects how often old trajectories are reused.
  • Clip parameters = epsilon_low=0.2, epsilon_high=0.3
    Borrowed from DAPO [28]; they shape the GRPO update and are not tuned for GUI environments in this paper.
assumptions (5)
  • domain assumption OSWorld execution-based scoring gives a reliable binary reward for task success
    Used throughout Sec. 3.4 as the trajectory reward rt; if the OSWorld evaluator is noisy or gameable, the training signal is compromised.
  • ad hoc to paper GRPO group normalization remains valid when one group member is a replayed trajectory from an earlier policy
    Sec. 3.5 injects buffer trajectories into groups without off-policy importance correction, while the GRPO advantage formula in Sec. 3.1 assumes samples from the current policy.
  • domain assumption The subset of 128 tasks selected by baseline solvability is representative of OSWorld's full benchmark
    Headline results in Table 1 are computed over the whole OSWorld set, but training is on the filtered subset; the paper's own OOD analysis shows poor transfer.
  • ad hoc to paper Binary trajectory reward plus format penalty provides sufficient learning signal
    Sec. 3.4 defines r = rt + rf with only 1, 0, and -1 values; there is no shaped or auxiliary reward, and the paper does not analyze whether this signal is sufficient.
  • domain assumption UI-Tars-1.5 / Qwen2.5-VL is a suitable base policy for RL fine-tuning
    All experiments start from this pretrained model, and the method's success may depend on the quality of this initialization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay." pith.science (2026). https://pith.science/paper/AA7GL5KX

@misc{pith2026250516282,
  author       = {Pith},
  title        = {Pith review of: ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AA7GL5KX}},
  note         = {Machine review of arXiv:2505.16282}
}
read the original abstract

Training large language models (LLMs) as interactive agents for controlling graphical user interfaces (GUIs) presents a unique challenge to optimize long-horizon action sequences with multimodal feedback from complex environments. While recent works have advanced multi-turn reinforcement learning (RL) for reasoning and tool-using capabilities in LLMs, their application to GUI-based agents remains relatively underexplored due to the difficulty of sparse rewards, delayed feedback, and high rollout costs. In this paper, we investigate end-to-end policy optimization for vision-language-based GUI agents with the aim of improving performance on complex, long-horizon computer tasks. We propose Agentic Replay Policy Optimization (ARPO), an end-to-end RL approach that augments Group Relative Policy Optimization (GRPO) with a replay buffer to reuse the successful experience across training iterations. To further stabilize the training process, we propose a task selection strategy that filters tasks based on baseline agent performance, allowing the agent to focus on learning from informative interactions. Additionally, we compare ARPO with offline preference optimization approaches, highlighting the advantages of policy-based methods in GUI environments. Experiments on the OSWorld benchmark demonstrate that ARPO achieves competitive results, establishing a new performance baseline for LLM-based GUI agents trained via reinforcement learning. Our findings underscore the effectiveness of reinforcement learning for training multi-turn, vision-language GUI agents capable of managing complex real-world UI interactions. Codes and models:https://github.com/dvlab-research/ARPO.git.

Figures

Figures reproduced from arXiv: 2505.16282 by the authors.

Figure 1
Figure 1. Agentic Replay Policy Optimization (ARPO) enables effective end-to-end policy opti￾mization for GUI agents. (a) Our vision-language agent processes long-horizon visual observations and interaction histories to generate sequential actions and receive policy gradients from sparse, delayed rewards. (b) ARPO significantly boosts in-domain task success rates compared to baseline and GRPO-only training. (c) Average traini… view at source ↗
Figure 2
Figure 2. Illustration of the reinforcement learning procedure for our multi-turn GUI agent. For a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation study of the replay buffer [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Training performance comparison for RL training with selected subset and full set. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparisons with offline preference optimization methods [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Trajectory showing the ARPO-trained agent correcting a misclick error. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

    cs.CL 2025-12 reject novelty 5.0 of 10

    AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.

  2. Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A plan-then-reason SFT plus a plan-quality reward in GRPO improves math-reasoning accuracy by small but consistent margins over GRPO and DAPO.

  3. MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment

    cs.LG 2025-07 conditional novelty 5.0 of 10

    MobileGUI-RL applies online RL with self-generated and filtered tasks plus trajectory-level rewards to mobile GUI agents, reporting improved success rates on AndroidWorld and AITW benchmarks.

Reference graph

Works this paper leans on

29 extracted references · 5 canonical work pages · cited by 3 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 3

    arXiv 2025

  2. [2]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935,

    arXiv

  3. [3]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. 8

    arXiv 2024

  4. [4]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024. 2, 4, 5

    arXiv 2024

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 2, 3

    arXiv 2025

  6. [6]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14281–14290, 2024. 2

    work page 2024

  7. [7]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025. 3

    arXiv 2025

  8. [8]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 3

    arXiv 2024

Show all 29 references
  1. [9]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553, 2024. 2

    2024 arXiv

  2. [10]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles , ...

  3. [11]

    Code-r1: Reproducing r1 for code with reliable rewards

    Jiawei Liu and Lingming Zhang. Code-r1: Reproducing r1 for code with reliable rewards. arXiv preprint arXiv:2503.18470, 2025. 3

    2025

  4. [12]

    Seg-zero: Reasoning- chain guided segmentation via cognitive reinforcement

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning- chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025. 3

    2025 arXiv

  5. [13]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  6. [14]

    Screenagent: A vision language model-driven computer control agent

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. arXiv preprint arXiv:2402.07945,

    arXiv

  7. [15]

    Toolrl: Reward is all tool learning needs

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025. 2, 3, 5

    2025 arXiv

  8. [16]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025. 2, 3, 4, 5, 6

    2025 arXiv

  9. [17]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 8

    2023

  10. [18]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 2 10

    2017 arXiv

  11. [19]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv. org/abs/2402.03300. 2, 3, 5

    2024 arXiv

  12. [20]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256,

    arXiv

  13. [21]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025. 3

    2025 arXiv

  14. [22]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 4

    2022

  15. [23]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024. 7

    arXiv

  16. [24]

    Gui-r1: A generalist r1-style vision-language action model for gui agents

    Xiaobo Xia and Run Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025. 2

    2025 arXiv

  17. [25]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  18. [26]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024. 4, 7

    2024 arXiv

  19. [27]

    Aria-ui: Visual grounding for gui instructions

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024. 7

    2024 arXiv

  20. [28]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 5, 6

    2025 arXiv

  21. [29]

    Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXiv preprint arXiv:2503.15478,

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXiv preprint arXiv:2503.15478,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.