Pith. sign in

REVIEW 5 major objections 4 minor 59 references

Episode-level rewards allow GUI agents to decode vague user requests into correct personalized recommendations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:37 UTC pith:NYTIJRPL

load-bearing objection Plausible recipe for episode-level RL in personalized GUI agents, but the key advantage term as written is a constant offset and cannot do the credit assignment the paper claims; experiments are too thin to carry it. the 5 major comments →

arxiv 2607.25369 v1 pith:NYTIJRPL submitted 2026-07-28 cs.AI

ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning

Jiaqi Zhang , Tong Chen , Junliang Yu , Quoc Viet Hung Nguyen , Hongzhi Yin This is my paper
classification cs.AI
keywords personalized agentic reasoningreinforcement fine-tuningepisode-wise policy optimizationGRPOGUI agentsChain-of-User-Thought rewardlong-horizon reasoninguser intent decoding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a new challenge it calls personalized agentic reasoning: agents that start from a vague user request and must interact with a graphical interface across many steps to infer what the user actually wants. It argues that existing training methods reward either the final answer or individual actions in isolation, which fails to connect early clues to later decisions. To fix this, the paper proposes training the agent on whole episodes, with an episode-level reward that measures both how confident the inferred user intent is and how much that intent contributed to the final recommendation. This reward is broadcast back to every step, so early GUI actions are optimized for their eventual effect on the recommendation. On a long-horizon personalized GUI benchmark, the episode-wise method outperforms both step-wise reinforcement learning and standard supervised training, especially on final recommendation accuracy.

Core claim

The central discovery is that episode-wise policy optimization—specifically, the proposed ESPO (an extension of the GRPO policy-gradient method)—improves personalized agentic reasoning. Instead of assigning each interaction step its own relative score, ESPO computes a Chain-of-User-Thought (COUT) reward for the whole episode, combining an 'intent confidence' term (is the predicted user intent supported by reliable earlier GUI actions?) with an 'intent contribution' term (did that intent actually lead to a correct final recommendation?). This episode-level reward is then added to each step's local advantage, so gradients flow from the final decision back through the entire trajectory. An epis

What carries the argument

The load-bearing mechanism is the Chain-of-User-Thought (COUT) reward combined with episodic advantage broadcasting. COUT is an episode-level scalar computed from two sigmoid gates: a GUI-action gate that aggregates how reliable the preceding GUI actions were, and a decision gate that aggregates how successful the final recommendation step was. COUT equals the sum of these gates multiplied by an intent-matching reward, yielding two terms—Intent Confidence and Intent Contribution. This reward is then broadcast, meaning the same episode-level advantage is added to every action step's locally computed advantage, so all steps share credit for the episode's outcome. The episodic batch sampler ens

Load-bearing premise

The whole result rests on the assumption that the benchmark's ground-truth labels and hand-crafted reward equations faithfully capture what real users actually want when they make an ambiguous request.

What would settle it

A user study where real users rate the final recommendations of an ESPO-trained agent versus a step-wise-trained agent on fresh ambiguous requests would falsify the personalization claim if users show no systematic preference for the episode-wise agent's choices.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Replacing vanilla step-wise GRPO with episode-wise ESPO improves all four evaluated metrics, showing that episode-level supervision helps both low-level GUI interaction and high-level recommendation.
  • Supervised fine-tuning alone, even with additional iterations, achieves zero recommendation accuracy on the benchmark, while the reinforcement stage with ESPO brings recommendation accuracy to 28.57%—evidence that reinforcement feedback is necessary for personalization.
  • Removing either the Intent Confidence or the Intent Contribution term from the COUT reward causes at least a 15% drop in recommendation accuracy, showing that both terms are essential for cross-step alignment.
  • Training for more than six epochs or raising the reward-gate temperature degrades performance, indicating that the episode-level reward needs a smooth, stable shape rather than a sharp one.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If episode-level broadcasting generalizes beyond GUI tasks, it could be applied to any sequential decision-making problem where a final outcome depends on a chain of intermediate actions—such as web navigation or household robotics; the paper does not test these domains.
  • The hand-designed COUT reward uses a cosine-similarity threshold to judge intent matching; a natural extension would be to tune or learn that threshold, since its value likely changes how much credit the episode reward assigns to early steps.
  • A strong test of the personalization claim would be to hold out entire users rather than episodes during training; if the model cannot generalize to an unseen user's ambiguous requests, the benchmark may be measuring reward-fitting rather than true personalization.
  • The paper reports no user study or external validation of whether recommended items are genuinely preferred by real users; such a study would separate the method's real-world value from its benchmark score.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper frames personalized agentic reasoning as a long-horizon GUI task with ambiguous user requests. It proposes ODYSSE, a two-stage reinforced fine-tuning (RFT) framework: an SFT cold-start followed by ESPO, an episode-level extension of GRPO. ESPO defines stage-specific verifiable rewards, aggregates them into a Chain-of-User-Thought (COUT) reward, broadcasts this reward as an episodic advantage across action steps, and uses an episodic batch sampler to preserve episode structure. Experiments on the SmartSpot benchmark claim consistent gains over generalist LVLMs and specialist GUI agents, including GUI-R1, and ablations attribute the gains to ESPO.

Significance. If the central claim holds, episode-wise advantage estimation over full interaction trajectories is a plausible and useful direction for personalized agentic reasoning, going beyond step-wise GRPO. The formulation of stage-specific rewards and the intent-confidence/intent-contribution gates is concrete and testable. However, the empirical evidence is currently too weak and partly internally inconsistent to establish the claim. The paper would benefit from a corrected episodic-advantage definition, a larger or at least properly seeded evaluation, and legible numeric results.

major comments (5)
  1. [Sec. 4.2.3, Eqs. (13)-(14)] The COUT advantage as written is a constant across the G-rollout group. In Eq. (13), A_COUT(e_i) = Broadcast_{a_j in e_i, g=1..G}(R_COUT^(i)), and R_COUT^(i) in Eq. (10) carries no rollout index g. Therefore A_COUT is the same scalar for every rollout and every step. A GRPO-style advantage must be relative across the group to provide credit assignment; a constant offset does not favor any rollout and, at importance ratio 1, contributes zero in expectation to the policy gradient. Consequently, the central claim that broadcasting the episode-level reward 'enables upstream evidence to effectively guide downstream personalized decisions' (abstract, Sec. 5.4) is not what Eq. (14) implements. Please either make R_COUT rollout-dependent and group-normalize it, or explicitly identify the episodic batch sampler and the clip/KL interaction as the operative mechanism and revise the claims according
  2. [Sec. 5.1.1, Table 2] The evaluation rests on 102 SmartSpot episodes split 8:1:1, i.e., roughly 10 test episodes. Table 2 reports single point estimates with no error bars, and the text uses phrases like 'consistent improvements' without variance information. With a test set this small, the RecAcc difference between 28.57% and 25.50% is fragile. Please report means and standard deviations over at least 3-5 seeds, and ideally add a second benchmark or a larger held-out set to support the generalization claim.
  3. [Secs. 5.1.3 and 5.6] The paper never defines the cosine threshold τ in Eq. (6), which is a load-bearing part of the intent-matching reward. Moreover, k and w are set to 1.0 in Sec. 5.1.3, and Sec. 5.6 selects the epoch count and k on the same test benchmark. This is selection on the test set. The hand-chosen reward constants in Eqs. (4)-(7) and the unspecified τ, w, and reward-scale sensitivity are not analyzed on a validation split. The risk that the gains reflect fitting SmartSpot's reward structure rather than better personalized reasoning is therefore not addressed.
  4. [Figures 2-6 and Secs. 5.2-5.6] In the submitted text, Figures 2, 3, 4, 5, and 6 appear as unreadable '/uni000...' encoded strings. These figures contain the RQ1, RQ3, and RQ4 empirical evidence, including the key ESPO-vs-GRPO comparison and the COUT ablations. No numeric tables accompany these figures, and the text reports only qualitative statements such as 'consistent improvements' or 'at least a 15% gap on RecAcc'. This makes the central empirical claims impossible to verify. Please replace the figures with legible versions and include a numeric table for all reported comparisons and ablations.
  5. [Sec. 5.5, Eq. (10)] The COUT reward in Eq. (10) is written as phi*Intent_acc + psi*Intent_acc, with the two terms referred to as 'Intent Confidence' and 'Intent Contribution'. As written, the two terms are algebraically identical apart from the gates phi and psi. The ablation in Sec. 5.5.2 treats them as separate components, but the manuscript does not clarify whether the two Intent_acc symbols are meant to be different quantities or whether the distinction is entirely in the gates. Please define the two terms explicitly and explain why removing one gate causes the reported cascade.
minor comments (4)
  1. [Eqs. (8)-(9)] The denominators R_max^GUI-accuracy and R_max^dec-accuracy are not defined. State whether they are the maximum possible sum of the respective rewards over an episode.
  2. [Algorithm 1 and Eq. (15)] Line 21 says 'Update theta using GRPO with KL regularization', while Eq. (15) is the ESPO objective. Use consistent terminology. Also, the ratio in Eq. (16) uses y_{j,g,t} but Eq. (15) writes rho_{j,i,t}; align the indices.
  3. [Sec. 5.1.3] No code or data release is mentioned. Given that the reward functions and evaluation are built on an in-group benchmark, releasing code and reward definitions is essential for reproducibility.
  4. [Appendix A.2, Fig. 7] The case study shows a recommendation step with ground-truth 'No' and model output 'Yes' (Step 13), and another with ground-truth 'Yes' and model output 'click' (Step 16). These errors are not discussed; a brief analysis would improve the interpretability of the case study.

Circularity Check

0 steps flagged

No circular derivation; empirical claims rest on a benchmark from overlapping prior work but are not equivalent to their inputs.

full rationale

The paper's core derivation is explicit: stage rewards (Eqs. 4–7) -> gates (Eqs. 8–9) -> COUT reward (Eq. 10) -> advantages (Eqs. 12–14) -> ESPO objective (Eq. 15). None of these equations reintroduces the target result as an input; the method is an empirical RL procedure rather than a fitted predictor of its own evaluation. The evaluation uses SmartSpot [42] and SmartAgent [42], authored by overlapping researchers, but this is a benchmark/baseline rather than a load-bearing derivation; per the rubric, self-citation in evaluation is not circularity unless the central claim reduces to it. A separate internal-consistency concern is that A_COUT in Eq. 13 is a group-constant broadcast because R_COUT in Eq. 10 has no rollout index, so the episodic term as written cannot provide group-relative credit assignment; this is a correctness risk, not a circularity, since the empirical comparison could still be valid through the episodic batch sampler or other mechanisms.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

The central claim rests on hand-chosen reward scales, tuned hyperparameters (k, w), and a self-created benchmark, which together determine most of the reported gains.

free parameters (5)
  • k (temperature coefficient) = 1.0
    Controls sigmoid sharpness of gates φ and ψ in Eqs (8)-(9); set to 1.0 after sensitivity analysis in Fig 6.
  • w (COUT advantage weight) = 1.0
    Weights the broadcast COUT advantage in Eq (14); set to 1.0 in implementation details (§5.1.3).
  • format/accuracy reward constants = 0.5, 0.2, -0.2, -0.5
    Hand-chosen reward scales in Eqs (4)-(7); no justification or ablation.
  • τ (cosine threshold in Intent_acc) = unspecified
    Threshold for intent matching in Eq (6); never given a value in the paper.
  • training schedule (SFT epochs, ESPO epochs, batch size) = 1 epoch SFT, 6 epochs ESPO, 1 episode per batch
    Chosen in §5.1.3 and sensitivity-tuned in §5.6; may affect results.
axioms (6)
  • standard math GRPO/PPO clipped surrogate objective is a valid policy optimization objective
    Used in Eq (2) and Eq (15); standard RL background.
  • domain assumption SmartSpot episode labels and ground-truth GUI intents y*_j represent genuine user personalization
    Rewards in Eqs (5)-(7) treat these labels as ground truth; benchmark from [42].
  • domain assumption GUI screenshot and action history at each step contain sufficient information to infer user intent
    Problem definition in §3; if false, the whole task is unlearnable.
  • domain assumption 102 episodes from SmartSpot are representative of real-world personalized agentic reasoning
    Evaluation §5.1.1; small scale, no external validation.
  • ad hoc to paper R_COUT = φ·Intent_acc + ψ·Intent_acc is an appropriate model of cross-step dependency
    Eq (10) is hand-designed; both terms multiply the same Intent_acc, with no derivation.
  • ad hoc to paper Broadcasting the same COUT advantage to all steps/rollouts is a valid credit assignment scheme
    Eqs (13)-(14); a constant shift may not differentiate step-level credit as claimed.
invented entities (1)
  • Chain-of-User-Thought (COUT) reward no independent evidence
    purpose: Episode-level reward designed to tie GUI intent prediction to the final personalized decision
    The reward is hand-composed from the paper's own gates and intent accuracy; no external validation that it measures user intent.

pith-pipeline@v1.3.0-alltime-deepseek · 21122 in / 14449 out tokens · 138131 ms · 2026-08-01T02:37:25.317880+00:00 · methodology

0 comments
read the original abstract

Agentic systems have rapidly advanced in their ability to interact with real-world environments, leverage external tools, and provide services for users. However, unlike natural-world tasks that assume well-defined instructions, human-centered scenarios are characterized by ambiguous requests that lead to large, open-ended solution spaces. Decoding users' personalized preferences is therefore essential for narrowing the candidate solution space. This introduces a new challenge, personalized agentic reasoning, which requires agents to jointly interact with both users and environments to deliver personalized services. In this paper, we present ODYSSE, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning. At its core, ODYSSE proposes Episode-wise GRPO (ESPO), a novel extension of Group Relative Policy Optimization (GRPO) designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning. Rather than optimizing individual steps independently, ESPO introduces an episode-level reward mechanism together with episodic advantage estimation, enabling upstream evidence to effectively guide downstream personalized decisions and allowing agents to progressively resolve ambiguous user requests across multiple interaction steps. We further propose an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO. We evaluate ODYSSE on realistic long-horizon personalized GUI reasoning tasks. Experimental results demonstrate that ODYSSE consistently outperforms both specialist and general-purpose LVLMs, highlighting its effectiveness for personalized agentic reasoning.

Figures

Figures reproduced from arXiv: 2607.25369 by Hongzhi Yin, Jiaqi Zhang, Junliang Yu, Quoc Viet Hung Nguyen, Tong Chen.

Figure 1
Figure 1. Figure 1: ESPO organizes 𝐿 isolated action steps into episodes. The stacked 𝑜𝑖,𝑗 and 𝐴𝑖,𝑗 blocks represent𝐺 rollout-level quantities. 3 Problem Definition In this work, we formulate Personalized Agentic Reasoning in the context of digital personal assistants (e.g., flight booking and prod￾uct purchasing assistants). In this setting, an ambiguous request 𝑞 is resolved through agent-GUI (Graphical User Interface) inte… view at source ↗
Figure 2
Figure 2. Figure 2: Our RFT-based ODYSSE vs. its SFT variants. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Our ESPO vs. GRPO. understanding alone is sufficient for personalized factor percep￾tion without task-specific adaptation. Specialist Agentsexamine whether optimization for specific environments alone is sufficient for capturing personalized factors. Together, these two groups test whether personalized agentic reasoning can be achieved by either general-purpose multimodal capability or environment-specific… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study of Intent Contribution and Intent Confidence in 𝑅COUT. The y-axis presents the score Δ relative to ODYSSE (%). 1.0 2.0 3.0 3 6 9 Epoch k 0.130 0.140 0.119 0.132 0.127 0.122 0.125 0.111 0.102 EleAcc 1.0 2.0 3.0 3 6 9 k 0.629 0.661 0.699 0.824 0.775 0.743 0.778 0.773 0.803 Op F1 1.0 2.0 3.0 3 6 9 Epoch k 0.084 0.110 0.112 0.161 0.137 0.115 0.127 0.127 0.134 Step SR 1.0 2.0 3.0 3 6 9 k 0.079 0.… view at source ↗
Figure 6
Figure 6. Figure 6: Hyper-parameter sensitivity analysis of ODYSSE [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Case Study of our proposed ODYSSE [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 25 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al . 2025. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631(2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Hyungjoo Chae, Namyoung Kim, Kai Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. 2025. Web agents with world models: Learning and leveraging environment dynamics in web navigation. InInternational Conference on Learning Representations, Vol. 2025. 63707–63738

    2025

  4. [4]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning.arXiv preprint arXiv:2310.05915(2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 9313–9332

    2024

  6. [6]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx

  8. [8]

    Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, and Weipeng Chen. 2024. From novice to expert: Llm agent policy optimization via step-wise reinforcement learning.arXiv preprint arXiv:2411.03817(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. 2026. Agentic Reinforced Policy Optimization. InThe Fourteenth International Conference on Learning Representations

    2026

  10. [10]

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation mat- ters in deep policy gradients: A case study on ppo and trpo.arXiv preprint arXiv:2005.12729(2020)

    Pith/arXiv arXiv 2020

  11. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al . 2025. DeepSeek-R1 in- centivizes reasoning in LLMs through reinforcement learning.Nature645, 8081 (2025), 633–638

    2025

  12. [12]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6864–6890

    2024

  13. [13]

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2024. An embodied generalist agent in 3D world. InProceedings of the 41st International Conference on Machine Learning. 20413–20451

    2024

  14. [14]

    Jeonghye Kim, Sojeong Rhee, Minbeom Kim, Dohyung Kim, Sangmook Lee, Youngchul Sung, and Kyomin Jung. 2025. Reflact: World-grounded decision making in llm agents via goal-state reflection. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 33421–33453

    2025

  15. [15]

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Jia Li, Yuxin Su, and Michael R Lyu. 2026. From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level. arXiv preprint arXiv:2601.03731(2026)

    Pith/arXiv arXiv 2026

  17. [17]

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security.arXiv preprint arXiv:2401.05459(2024)

    Pith/arXiv arXiv 2024

  18. [18]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. InInternational Conference on Learning Representations, Vol. 2024. 39578–39601

    2024

  19. [19]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  20. [20]

    Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, and Xiaobo Xia. 2025. Gui-r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, et al. 2024. Habitat 3.0: A co-habitat for humans, avatars, and robots. In International Conference on Learning Representations, Vol. 2024. 15306–15336

    2024

  22. [22]

    Xiaoming Ren, Ru Zhen, Chao Li, Yang Song, Qiuxia Hou, Yanhao Zhang, Peng Liu, Qi Qi, Quanlong Zheng, Qi Wu, et al. 2026. X-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction.arXiv preprint arXiv:2605.05765(2026)

    Pith/arXiv arXiv 2026

  23. [23]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems36 (2023), 68539–68551

    2023

  24. [24]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  25. [25]

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2025. Rewarding progress: Scaling automated process verifiers for llm reasoning. In International Conference on Learning Representations, Vol. 2025. 60808–60838

    2025

  26. [26]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  27. [27]

    Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. 2025. Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441(2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping

  29. [29]

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. 2021. Habitat 2.0: training home a...

    2021

  30. [30]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents.Frontiers of Computer Science18, 6 (2024), 186345

    2024

  31. [31]

    Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, and Ling Yang. 2026. Openclaw-rl: Train any agent simply by talking.arXiv preprint arXiv:2603.10165 (2026)

    Pith/arXiv arXiv 2026

  32. [32]

    Yinjie Wang, Tianbao Xie, Ke Shen, Mengdi Wang, and Ling Yang. 2026. RLAny- thing: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System.arXiv preprint arXiv:2602.02488(2026)

    arXiv 2026

  33. [33]

    Yuhang Wang, Feiming Xu, Zheng Lin, Guangyu He, Yuzhe Huang, Haichang Gao, Zhenxing Niu, Shiguo Lian, and Zhaoxiang Liu. 2026. From assistant to double agent: Formalizing and benchmarking attacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412(2026)

    arXiv 2026

  34. [34]

    Zongwei Wang, Min Gao, Hongzhi Yin, Junliang Yu, Tong Chen, Shazia Sadiq, and Tianrui Li. 2026. Self-distilled reinforcement learning for co-evolving agentic recommender systems.arXiv preprint arXiv:2604.10029(2026)

    Pith/arXiv arXiv 2026

  35. [35]

    Zehong Wang, Fang Wu, Hongru Wang, Xiangru Tang, Bolian Li, Zhenfei Yin, Yijun Ma, Yiyang Li, Weixiang Sun, Xiusi Chen, et al. 2026. Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Jiaqi Zhang, Tong Chen, Junliang Yu, Quoc Viet Hung Nguyen, and Hongzhi ...

    arXiv 2026

  36. [36]

    Tianxin Wei, Ting-Wei Li, Zhining Liu, Xuying Ning, Ze Yang, Jiaru Zou, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Dongqi Fu, et al. 2026. Agentic reasoning for large language models.arXiv preprint arXiv:2601.12538(2026)

    Pith/arXiv arXiv 2026

  37. [37]

    Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhenglin Wang, Zhengwei Tao, Ding-Chu Zhang, Zekun Xi, Robert Tang, et al . 2026. Webdancer: Towards autonomous information seeking agency.Advances in Neural Information Processing Systems38 (2026), 120957–120985

    2026

  38. [38]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  39. [39]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629(2022)

    Pith/arXiv arXiv 2022

  40. [40]

    Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. 2023. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306.11565(2023)

    Pith/arXiv arXiv 2023

  41. [41]

    Jiaqi Zhang, Yu Cheng, Yongxin Ni, Yunzhu Pan, Zheng Yuan, Junchen Fu, Youhua Li, Jie Wang, and Fajie Yuan. 2024. Ninerec: A benchmark dataset suite for evaluating transferable recommendation.IEEE Transactions on Pattern Analysis and Machine Intelligence(2024)

    2024

  42. [42]

    Jiaqi Zhang, Chen Gao, Liyuan Zhang, Quoc Viet Hung Nguyen, and Hongzhi Yin. 2026. Smartagent: Chain-of-user-thought for embodied personalized agent in cyber world. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 17993–18001

    2026

  43. [43]

    Jiaqi Zhang, Junliang Yu, Zongwei Wang, Wei Yuan, Tong Chen, Quoc Viet Hung Nguyen, Bin Cui, and Hongzhi Yin. 2025. Towards Reasoning-Aware Recom- mender Systems: A Survey in the LLM Era.Authorea Preprints(2025)

    2025

  44. [44]

    Yu Zhang, Shutong Qiao, Jiaqi Zhang, Tzu-Heng Lin, Chen Gao, and Yong Li

  45. [45]

    Yuyang Zhao, Wentao Shi, Fuli Feng, and Xiangnan He. 2025. Appagent-pro: A proactive gui agent system for multidomain information integration and user assistance. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 6767–6771

    2025

  46. [46]

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. 2025. Easyr1: An efficient, scalable, multi-modality rl training framework.arXiv preprint arXiv:2501.12345(2025)

    Pith/arXiv arXiv 2025

  47. [47]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning, acting, and planning in language models. InProceedings of the 41st International Conference on Machine Learning. 62138–62160

    2024

  48. [48]

    A survey of large language model empowered agents for recommenda- tion and search: Towards next-generation information retrieval.arXiv preprint arXiv:2503.05659(2025)

    Pith/arXiv arXiv 2025

  49. [49]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479(2025)

    Pith/arXiv arXiv 2025

  50. [50]

    Ningyan Zhu, Huacan Wang, Jie Zhou, Feiyu Chen, Shuo Zhang, Ge Chen, Chen Liu, Jiarou Wu, Wangyi Chen, Xiaofeng Mou, et al . 2026. SemaClaw: A Step Towards General-Purpose Personal AI Agents through Harness Engineering. arXiv preprint arXiv:2604.11548(2026). A Appendix A.1 Detailed Prompt Prompt designs of our ODYSSE You are a personal GUI agent. In this ...

    Pith/arXiv arXiv 2026

  51. [52]

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. 2025. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXiv preprint arXiv:2503.15478(2025)

    Pith/arXiv arXiv 2025

  52. [55]

    For’click’: •output the click point[x, y] •input_textmust be’no input text’

  53. [56]

    For’type’: •output the target point[x, y] •input_textmust be the exact text to type

  54. [57]

    For ’scroll up’ , ’scroll down’ , ’scroll left’ , ’scroll right’,’exit’: •pointmust be[-100, -100] •input_textmust be’no input text’

  55. [58]

    For’pool_found’: •pointmust be[-100, -100] •input_text must be a short summary of the user’s final intent

  56. [59]

    •Do not output multiple<think>/<answer>blocks

    For’recommendation’: •pointmust be[-100, -100] •input_text must be exactly one English word: ’Yes’or’No’ • do not output any explanation, summary, or extra text Important constraints: •Output exactly one action. •Do not output multiple<think>/<answer>blocks. •Do not output multiple dictionaries. • For ’recommendation’, input_text must be ex- actly’Yes’or ...

    2018

  57. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  58. [2025]

    Agent-rlvr: Training software engineering agents via guidance and envi- ronment rewards.arXiv preprint arXiv:2506.11425(2025)

    Pith/arXiv arXiv 2025

  59. [2026]

    InThe Fourteenth International Conference on Learning Representations

    The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs. InThe Fourteenth International Conference on Learning Representations