pith. machine review for the scientific record. sign in

arxiv: 2604.18401 · v1 · submitted 2026-04-20 · 💻 cs.CL

Recognition: unknown

StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning

Daoyu Wang , Qingchuan Li , Mingyue Cheng , Jie Ouyang , Shuo Yu , Qi Liu , Enhong Chen
Authors on Pith no claims yet

Pith reviewed 2026-05-10 04:23 UTC · model grok-4.3

classification 💻 cs.CL
keywords agentic reinforcement learningstep-level MDPLLM agentspolicy optimizationcredit assignmentmulti-turn interactionstool usedecision making
0
0 comments X

The pith

LLM agents need step-level MDP and credit assignment rather than token-level modeling for multi-turn RL.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Traditional token-level MDP modeling inherited from standard LLM training struggles to capture the delayed rewards, sparse signals, and long variable contexts that arise in multi-turn agent interactions. The paper advances a step-level MDP formulation in which entire steps function as the atomic actions, with corresponding step-level credit assignment to propagate rewards at the natural granularity of agent decisions. This alignment lets policy optimization target core capabilities such as decision making and tool use directly. If the claim holds, agentic RL would become substantially more effective at training general agents.

Core claim

The paper claims that the conventional token-level Markov Decision Process (MDP) should be advanced to a step-level MDP formulation, and that the step, rather than the token, should be regarded as the proper action representation for LLM agents. Step-level credit assignment is introduced as the matching optimization method, aligning policy updates and reward propagation with the scale of actual agent behavior. Preliminary experiments supply initial support for the approach.

What carries the argument

Step-level MDP formulation in which steps serve as actions, together with step-level credit assignment that propagates rewards at the granularity of agent decisions.

If this is right

  • Credit assignment operates at the scale of real agent decisions rather than individual tokens.
  • Policy optimization directly targets multi-turn behaviors such as tool use and planning.
  • Sparse and delayed rewards become easier to propagate across variable-length interactions.
  • Training focuses on decision-level outcomes instead of token prediction accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • New training pipelines could track decision boundaries explicitly to enable step-level logging and replay.
  • Benchmarks may shift toward evaluating complete steps rather than token sequences to better match the new MDP.
  • The formulation could combine with hierarchical RL techniques to scale to still longer agent horizons.
  • Agent harnesses might need updated interfaces that expose step boundaries for reward shaping.

Load-bearing premise

Redefining the MDP and credit assignment at step granularity will meaningfully address delayed and sparse rewards plus long-context challenges in multi-turn agent settings.

What would settle it

A head-to-head experiment on a multi-turn tool-use benchmark in which step-level credit assignment produces measurably higher success rates than matched token-level baselines.

read the original abstract

General agents have given rise to phenomenal applications such as OpenClaw and Claude Code. As these agent systems (a.k.a. Harnesses) strive for bolder goals, they demand increasingly stronger agentic capabilities from foundation Large Language Models (LLMs). Agentic Reinforcement Learning (RL) is emerging as a central post-training paradigm for empowering LLMs with these capabilities and is playing an increasingly pivotal role in agent training. Unlike single-turn token-level alignment or reasoning enhancement, as in RLHF and RLVR, Agentic RL targets multi-turn interactive settings, where the goal is to optimize core agentic capabilities such as decision making and tool use while addressing new challenges including delayed and sparse rewards, as well as long and variable context. As a result, the token-centric modeling and optimization paradigm inherited from traditional LLM RL is becoming increasingly inadequate for capturing real LLM agent behavior. In this paper, we present StepPO as a position on step-level Agentic RL. We argue that the conventional token-level Markov Decision Process (MDP) should be advanced to a step-level MDP formulation, and that the step, rather than the token, should be regarded as the proper action representation for LLM agents. We then propose step-level credit assignment as the natural optimization counterpart of this formulation, thereby aligning policy optimization and reward propagation with the granularity of agent decisions. Finally, we discuss the key systems designs required to realize step-level Agentic RL in practice and preliminary experiments provide initial evidence for the effectiveness of this perspective. We hope that the step-aligned, step-level paradigm embodied in StepPO offers the Agentic RL community a useful lens for understanding agent behavior and helps advance LLMs toward stronger general-agent capabilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents StepPO as a position paper on agentic reinforcement learning for LLMs. It argues that the conventional token-level MDP formulation is inadequate for multi-turn interactive agent settings and should be replaced by a step-level MDP in which steps (rather than tokens) constitute the atomic actions; it proposes step-level credit assignment as the corresponding optimization mechanism to better align with agent decisions and mitigate delayed/sparse rewards, discusses required systems-level designs, and reports preliminary experiments as initial supporting evidence.

Significance. If the step-level MDP can be rigorously formalized and implemented, the perspective could supply a more natural granularity for credit assignment and policy optimization in long-horizon LLM agent tasks, potentially improving sample efficiency and addressing limitations of token-centric RLHF/RLVR approaches in harness-style agent training.

major comments (2)
  1. [Step-level MDP formulation (proposal section)] The central claim that advancing to a step-level MDP yields a well-defined Markovian process whose credit assignment directly mitigates sparse/delayed rewards is load-bearing yet lacks an explicit transition kernel P(s_{t+1}|s_t, step) or intra-step reward decomposition. Without these, it remains unclear whether step boundaries chosen post-hoc preserve the Markov property or whether partial observability inside a step invalidates the claimed advantage over token-level MDPs.
  2. [Preliminary experiments] The preliminary experiments are invoked to provide 'initial evidence' for the effectiveness of the step-aligned paradigm, but no quantitative results, baselines, task descriptions, or evaluation protocols are supplied. This leaves the empirical support too thin to substantiate the position's practical claims.
minor comments (2)
  1. [Abstract and introduction] The abstract and introduction could more explicitly contrast the proposed step-level credit assignment with existing hierarchical or option-based RL methods to clarify novelty.
  2. [Notation and definitions] Notation for the step-level action space and state representation is introduced informally; adding a compact table or equation block defining A_step, S_step, and the reward function at step granularity would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for acknowledging the potential significance of a step-level MDP perspective for agentic RL. We address the two major comments point by point below and will revise the manuscript to incorporate the requested clarifications and details.

read point-by-point responses

  1. Referee: [Step-level MDP formulation (proposal section)] The central claim that advancing to a step-level MDP yields a well-defined Markovian process whose credit assignment directly mitigates sparse/delayed rewards is load-bearing yet lacks an explicit transition kernel P(s_{t+1}|s_t, step) or intra-step reward decomposition. Without these, it remains unclear whether step boundaries chosen post-hoc preserve the Markov property or whether partial observability inside a step invalidates the claimed advantage over token-level MDPs.

    Authors: We agree that the current presentation would benefit from greater formal rigor. In the revised manuscript we will explicitly define the step-level transition kernel P(s_{t+1} | s_t, a_step), where a_step denotes the atomic step action (e.g., a tool invocation or a complete response turn). We will also supply an intra-step reward decomposition that assigns terminal rewards at step boundaries while allowing optional dense signals inside steps. On the Markov property, we will clarify that step boundaries are not chosen post-hoc but are aligned with natural decision points in agent harnesses (after environment feedback), thereby ensuring the state representation at each step boundary captures the necessary history to reduce partial observability relative to token-level modeling. This formulation directly supports more effective credit assignment for delayed rewards. revision: yes

  2. Referee: [Preliminary experiments] The preliminary experiments are invoked to provide 'initial evidence' for the effectiveness of the step-aligned paradigm, but no quantitative results, baselines, task descriptions, or evaluation protocols are supplied. This leaves the empirical support too thin to substantiate the position's practical claims.

    Authors: We acknowledge that the current description of the preliminary experiments is too brief to serve as convincing initial evidence. In the revision we will expand the section to report quantitative metrics, explicit task descriptions (multi-turn tool-use and decision-making benchmarks), comparison baselines (token-level PPO and standard RLHF variants), and the evaluation protocols employed. These additions will be presented as preliminary while preserving the position-paper character of the work. revision: yes

Circularity Check

0 steps flagged

No circularity: position paper advances conceptual reformulation without self-referential reduction

full rationale

The manuscript is explicitly framed as a position paper proposing a shift from token-level to step-level MDP for agentic RL. The core argument identifies limitations of existing token-centric modeling for multi-turn settings and advocates redefining actions and credit assignment at step granularity. No equations, fitted parameters, or derivations are supplied that reduce by construction to the inputs (e.g., no self-defined MDP transition kernel or prediction that is statistically forced by a fit). Preliminary experiments are cited only as initial evidence, not as the load-bearing justification. No self-citation chains, uniqueness theorems, or ansatzes imported from prior author work appear in the provided text. The proposal remains a forward-looking perspective rather than a closed derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central proposal rests on one domain assumption about the superiority of step-level modeling; no free parameters or invented entities are described in the abstract.

axioms (1)
  • domain assumption Step-level MDP formulation better captures LLM agent behavior and enables effective credit assignment than token-level MDP
    This is the load-bearing premise stated in the abstract for advancing agentic RL.

pith-pipeline@v0.9.0 · 5623 in / 1135 out tokens · 53313 ms · 2026-05-10T04:23:57.354913+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Brief Overview: Agentic Reinforcement Learning In Large Language Models

    cs.AI 2026-04 unverdicted novelty 2.0

    This review synthesizes conceptual foundations, methods, challenges, and future directions for agentic reinforcement learning in large language models.

  2. A Brief Overview: Agentic Reinforcement Learning In Large Language Models

    cs.AI 2026-04 unverdicted novelty 2.0

    The paper surveys the conceptual foundations, methodological innovations, challenges, and future directions of agentic reinforcement learning frameworks that embed cognitive capabilities like meta-reasoning and self-r...

Reference graph

Works this paper leans on

40 extracted references · 23 canonical work pages · cited by 1 Pith paper · 15 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Agent-r1 github repository.https://github.com/AgentR1/Agent-R1, 2026

    AgentR1. Agent-r1 github repository.https://github.com/AgentR1/Agent-R1, 2026

    2026

  3. [3]

    Claw-r1 github repository.https://github.com/AgentR1/Claw-R1, 2026

    AgentR1. Claw-r1 github repository.https://github.com/AgentR1/Claw-R1, 2026

    2026

  4. [4]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024

    2024

  5. [5]

    Claude code: Build, debug, and ship from your terminal.https://claude.ai/product/claude-code, 2025

    Anthropic. Claude code: Build, debug, and ship from your terminal.https://claude.ai/product/claude-code, 2025

    2025

  6. [6]

    Qwen Technical Report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    arXiv preprint arXiv:2511.14460 , year=

    Mingyue Cheng, Jie Ouyang, Shuo Yu, Ruiran Yan, Yucong Luo, Zirui Liu, Daoyu Wang, Qi Liu, and Enhong Chen. Agent-r1: Training powerful LLM agents with end-to-end reinforcement learning.arXiv preprintarXiv:2511.14460, 2025

    work page arXiv 2025

  8. [8]

    A comprehensive survey of the llm-based agent: The contextual cognition perspective

    Mingyue Cheng, Daoyu Wang, Shuo Yu, Qingchuan Li, Jie Ouyang, Yucong Luo, Yiju Zhang, Qi Liu, and Enhong Chen. A comprehensive survey of the llm-based agent: The contextual cognition perspective. 2026

    2026

  9. [9]

    Group-in-Group Policy Optimization for LLM Agent Training

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for LLM agent training. arXiv preprint arXiv:2505.10978, 2025

    work page internal anchor Pith review arXiv 2025

  10. [10]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298,

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. AReaL: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298, 2025

    work page arXiv 2025

  11. [11]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv e-prints, pages arXiv–2501, 2025

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv e-prints, pages arXiv–2501, 2025

    2025

  12. [12]

    Tree search for llm agent reinforcement learning, 2026

    Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu. Tree search for LLM agent reinforcement learning.arXiv preprint arXiv:2509.21240, 2025. 12

    work page arXiv 2025

  13. [13]

    arXiv preprint arXiv:2509.01055 , year=

    Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. VerlTool: Towards holistic agentic reinforcement learning with tool use.arXiv preprint arXiv:2509.01055, 2025

    work page arXiv 2025

  14. [14]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    work page internal anchor Pith review arXiv 2025

  15. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  16. [16]

    AgentBench: Evaluating LLMs as Agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    work page internal anchor Pith review arXiv 2023

  17. [17]

    Agent lightning: Train any ai agents with reinforcement learning,

    Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent lightning: Train ANY AI agents with reinforcement learning.arXiv preprint arXiv:2508.03680, 2025

    work page arXiv 2025

  18. [18]

    Forge: Scalable agent rl framework and algorithm

    MiniMax. Forge: Scalable agent rl framework and algorithm. https://www.minimax.io/news/ forge-scalable-agent-rl-framework-and-algorithm, 2026

    2026

  19. [19]

    openclaw: Your own personal ai assistant

    openclaw. openclaw: Your own personal ai assistant. any os. any platform. the lobster way.https://github. com/openclaw/openclaw, 2024

    2024

  20. [20]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advancesin Neural Information Processing Systems, 2022

    2022

  21. [21]

    Paperscout: An autonomous agent for academic paper search with process-aware sequence-level policy optimization

    Tingyue Pan, Jie Ouyang, Mingyue Cheng, Qingchuan Li, Zirui Liu, Daoyu Wang, Mingfan Pan, Shuo Yu, and Qi Liu. Paperscout: An autonomous agent for academic paper search with process-aware sequence-level policy optimization. arXiv preprint arXiv:2601.10029, 2026

    work page arXiv 2026

  22. [22]

    rllm documentation.https://docs.rllm-project.com/, 2026

    rLLM. rllm documentation.https://docs.rllm-project.com/, 2026

    2026

  23. [23]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    2023

  24. [24]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  25. [25]

    Sglang: Efficient structured generation for large language models

    SGLang Team. Sglang: Efficient structured generation for large language models. https://github.com/ sgl-project/sglang, 2024

    2024

  26. [26]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    HybridFlow: A Flexible and Efficient RLHF Framework , url=

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. InProceedings of the TwentiethEuropean Conference on Computer Systems, 2025. doi: 10.1145/3689031.3696075

    work page doi:10.1145/3689031.3696075 2025

  28. [28]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    work page internal anchor Pith review arXiv 1909

  29. [29]

    ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    work page internal anchor Pith review arXiv 2010

  30. [30]

    slime: An SGLang-native post-training framework for RL scaling.https://lmsys.org/blog/ 2025-07-09-slime/, 2025

    The slime Team. slime: An SGLang-native post-training framework for RL scaling.https://lmsys.org/blog/ 2025-07-09-slime/, 2025

    2025

  31. [31]

    Voyager: An Open-Ended Embodied Agent with Large Language Models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anand- kumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023. 13

    work page internal anchor Pith review arXiv 2023

  32. [32]

    SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

    Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. Sppo: Sequence-level ppo for long-horizon reasoning tasks, 2026. URLhttps://arxiv.org/abs/2604.08865

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  33. [33]

    Let it flow: Agentic crafting on rock and roll, building the rome model within an open agentic learning ecosystem.arXiv preprintarXiv:2512.24873, 2025

    Weixun Wang, XiaoXiao Xu, Wanhe An, Fangwen Dai, Wei Gao, Yancheng He, Ju Huang, Qiang Ji, Hanqi Jin, Xiaoyang Li, et al. Let it flow: Agentic crafting on rock and roll, building the rome model within an open agentic learning ecosystem. arXiv preprint arXiv:2512.24873, 2025

    work page arXiv 2025

  34. [34]

    RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073, 2025

    work page internal anchor Pith review arXiv 2025

  35. [35]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018

    2018

  36. [36]

    Webshop: Towards scalable real-world web interaction with grounded language agents.Advancesin Neural Information Processing Systems, 35:20744–20757, 2022

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advancesin Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  37. [37]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022

    2022

  38. [38]

    The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for LLMs: A survey.arXiv preprint arXiv:2509.02547, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  39. [39]

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    work page internal anchor Pith review arXiv 2023

  40. [40]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025. 14

    work page internal anchor Pith review arXiv 2025