pith. machine review for the scientific record. sign in

arxiv: 2605.00425 · v3 · submitted 2026-05-01 · 💻 cs.AI

Recognition: 2 theorem links

· Lean Theorem

AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

Daxiang Dong, Haotian Zhao, Jianmin Wu, Jingnan Gu, Lun Tian, Songlin Zhou, Stephen S.-T. Yau, Tianshu Zhu, Wenyu Zhang, Yifeng Huang, Yucheng Zeng, Yuxin Zhang

Authors on Pith no claims yet

Pith reviewed 2026-05-11 00:50 UTC · model grok-4.3

classification 💻 cs.AI
keywords adaptive entropy modulationreinforcement learningLLM agentscredit assignmentexploration exploitationmulti-turn tasksresponse level entropyadvantage rescaling
0
0 comments X

The pith

AEM rescales advantages with a response-level entropy proxy to improve credit assignment in multi-turn LLM agent RL without added supervision.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents AEM as a way to handle sparse rewards in reinforcement learning for language model agents by modulating entropy at the level of complete responses rather than single tokens. This adjustment uses the natural interaction between a response's advantage and its surprisal to create an uncertainty signal that automatically shifts training from exploration toward exploitation as positive and negative samples balance out. The approach avoids dense intermediate rewards or auxiliary models, reducing complexity while targeting how agents actually influence environments through full outputs. Experiments across navigation, shopping, and code tasks with models up to 32B parameters show consistent gains over standard RL baselines.

Core claim

AEM lifts entropy dynamics from token to response level to match the effective action scale of LLM agents. Under natural-gradient updates, entropy drift is shown to be governed by the interaction between the sampled-response advantage and its relative surprisal. This relation yields a practical response-level uncertainty proxy that rescales advantages, allowing the training process to leverage the changing ratio of positive to negative samples for an automatic exploration-to-exploitation transition.

What carries the argument

Response-level uncertainty proxy derived from advantage-surprisal interaction, used to rescale advantages during RL updates.

If this is right

  • Credit assignment in long agent trajectories can proceed without process reward models or extra self-supervised objectives.
  • The exploration-exploitation balance emerges automatically from sample statistics rather than requiring separate schedules or hyperparameters.
  • Training remains effective across model scales from 1.5B to 32B on navigation, web, and software-engineering benchmarks.
  • Supervision complexity stays constant while performance rises relative to unmodified RL baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same response-level proxy might reduce the need for hand-crafted dense rewards in other sequential agent settings where actions are extended outputs.
  • Token-level entropy signals could systematically understate uncertainty when the environment only observes complete responses.
  • The method's reliance on natural sample balance suggests it could adapt to changing task distributions during training without explicit detection.

Load-bearing premise

Lifting entropy analysis from tokens to full responses correctly captures uncertainty at the granularity that actually affects the environment, and this proxy reliably tracks the advantage-surprisal dynamic without being disrupted by token sampling noise.

What would settle it

Run standard RL and AEM side-by-side on a multi-turn task and check whether the AEM version fails to improve final success rate or shows no measurable reduction in entropy drift after the positive-negative sample balance stabilizes.

Figures

Figures reproduced from arXiv: 2605.00425 by Daxiang Dong, Haotian Zhao, Jianmin Wu, Jingnan Gu, Lun Tian, Songlin Zhou, Stephen S.-T. Yau, Tianshu Zhu, Wenyu Zhang, Yifeng Huang, Yucheng Zeng, Yuxin Zhang.

Figure 1
Figure 1. Figure 1: An example on a three-action policy simplex: entropy increases along the training direction view at source ↗
Figure 2
Figure 2. Figure 2: Empirical relationship be￾tween α − 1 and the Monte Carlo rela￾tive surprisal −(S − HMC resp). Analysis A: Consistency between α−1 and −(S−Hresp). To examine whether α − 1 matches the sign of −(S − Hresp), we conduct a Monte Carlo probing study on the relationship between α − 1 and S(a | s) − Hresp(s) on WebShop with Qwen2.5-1.5B. We probe n = 64 states, and for each state we sample K = 64 responses to est… view at source ↗
Figure 3
Figure 3. Figure 3: Two masking strategies lead to clearly diverging entropy trends. Analysis B: Validating the trend of entropy. To further demonstrate how A(α − 1) controls the trend of entropy dynamics, we illustrate the entropy dynamics during the first 50 training steps under two gradient-masking strategies in view at source ↗
Figure 3
Figure 3. Figure 3: Two masking strategies lead to clearly diverging entropy trends. Analysis B: Validating the trend of entropy. To further illustrate how A(α − 1) governs the direc￾tion of entropy dynamics, [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Entropy and success-rate dynamics for one pair of runs. 5.4 Computation Cost 45.9% 8.2% 8.6% 36.0% Total 500.7s per step Abase AEM Component share Rollout 45.9% 229.9s Old prob. 8.2% 41.2s Ref prob. 8.6% 42.9s Update 36.0% 180.4s Abase 0.2% 0.81s AEM 1.1% 5.62s view at source ↗
Figure 6
Figure 6. Figure 6: Training time breakdown of GRPO+AEM. This section analyzes the additional computa￾tional overhead introduced by AEM. The extra cost is limited to lightweight response-level uncertainty estimation and modulation, includ￾ing response-level entropy aggregation, group￾wise normalization, and advantage rescaling. Importantly, AEM requires neither extra roll￾outs nor additional policy or reference model forward … view at source ↗
Figure 7
Figure 7. Figure 7: Training Curves of Qwen2.5-1.5B Model on ALFWorld. view at source ↗
Figure 8
Figure 8. Figure 8: Training Curves of Qwen2.5-1.5B Model on WebShop. view at source ↗
Figure 9
Figure 9. Figure 9: Training Curves of Qwen2.5-7B Model on ALFWorld. view at source ↗
Figure 10
Figure 10. Figure 10: Training Curves of Qwen2.5-7B Model on WebShop. view at source ↗
Figure 11
Figure 11. Figure 11: Training reward curves of DeepSWE with and without AEM on the R2E dataset. view at source ↗
read the original abstract

Reinforcement learning (RL) has substantially improved the ability of large language model (LLM) agents to interact with environments and solve multi-turn tasks. However, effective agentic RL remains challenging: sparse outcome-only rewards provide limited guidance for assigning credit to individual steps within long interaction trajectories. Existing approaches often introduce dense intermediate supervision, such as process reward models or auxiliary self-supervised signals, which increases supervision and tuning complexity and may limit generalization across tasks and domains. We present AEM, a supervision-free credit assignment method that adaptively modulates entropy dynamics during RL training to improve the exploration-exploitation trade-off. Since in agentic RL the environment is typically affected by a complete response, rather than an individual token, our analysis lifts entropy dynamics from the token level to the response level, aligning uncertainty estimation with the effective action granularity of LLM agents and reducing sensitivity to token-level sampling noise. We further show that entropy drift under natural-gradient updates is governed by the interaction between the sampled-response advantage and its relative surprisal. Motivated by this result, AEM derives a practical response-level uncertainty proxy and uses it to rescale advantages, leveraging the evolving balance between positive and negative samples to naturally transition from exploration to exploitation. Extensive experiments on ALFWorld, WebShop, and SWE-bench-Verified with models ranging from 1.5B to 32B demonstrate that AEM consistently improves strong RL baselines, including a +1.4\% gain when integrated into a state-of-the-art software-engineering RL training framework.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes AEM, a supervision-free credit-assignment method for multi-turn agentic RL with LLMs. It lifts entropy dynamics from token to response level to align with the granularity at which the environment is affected by complete responses, derives that entropy drift under natural-gradient updates is governed by the interaction between sampled-response advantage and relative surprisal, obtains a practical response-level uncertainty proxy from this analysis, and uses the proxy to rescale advantages so that the balance between positive and negative samples drives a natural transition from exploration to exploitation. Experiments on ALFWorld, WebShop, and SWE-bench-Verified with models from 1.5B to 32B report consistent gains over strong RL baselines, including a +1.4% improvement when integrated into a state-of-the-art software-engineering RL framework.

Significance. If the response-level entropy-drift derivation holds without unstated approximations and the empirical gains are robust to proper controls, AEM would supply a lightweight, supervision-free mechanism for modulating exploration-exploitation in agentic RL that avoids the overhead of process reward models or auxiliary self-supervised signals. The alignment of uncertainty estimation with response-level actions rather than token-level noise is a conceptually attractive feature for multi-turn settings with sparse outcome rewards. The breadth of evaluation across three distinct environments and a wide range of model scales constitutes a positive empirical contribution.

major comments (2)
  1. [analysis of entropy dynamics] The central claim that entropy drift under natural-gradient updates is governed by the interaction between the sampled-response advantage and its relative surprisal (and that this interaction directly motivates a practical response-level uncertainty proxy) is load-bearing for the entire supervision-free credit-assignment mechanism. The manuscript summarizes this analysis at a high level without providing the explicit update equations, the natural-gradient derivation steps, or the approximations employed (e.g., neglect of higher-order terms or assumptions about unbiased advantage estimates at response granularity). Without these details it is impossible to verify whether the proxy truly reduces sensitivity to token-level sampling noise or whether it introduces circularity with fitted parameters.
  2. [experimental evaluation] The experimental claims of consistent improvement, including the reported +1.4% gain on SWE-bench-Verified, rest on comparisons whose details are not visible in the provided description. No information is given on the precise baselines, number of random seeds, statistical significance tests, or ablation controls that isolate the contribution of the advantage-rescaling step from other implementation choices. This absence makes it difficult to assess whether the observed gains are attributable to AEM or to uncontrolled variance.
minor comments (2)
  1. [method] The notation used for the response-level uncertainty proxy and the advantage-rescaling operation should be introduced with an explicit formula (ideally an equation) rather than a prose description, to facilitate reproducibility.
  2. The abstract and introduction would benefit from a short statement of the precise functional form of the proxy (e.g., whether it is a normalized surprisal term or an advantage-weighted entropy estimate) so that readers can immediately grasp the computational overhead.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and constructive review. We address each major comment below, providing clarifications on the theoretical derivation and committing to expanded experimental reporting in the revision.

read point-by-point responses

  1. Referee: [analysis of entropy dynamics] The central claim that entropy drift under natural-gradient updates is governed by the interaction between the sampled-response advantage and its relative surprisal (and that this interaction directly motivates a practical response-level uncertainty proxy) is load-bearing for the entire supervision-free credit-assignment mechanism. The manuscript summarizes this analysis at a high level without providing the explicit update equations, the natural-gradient derivation steps, or the approximations employed (e.g., neglect of higher-order terms or assumptions about unbiased advantage estimates at response granularity). Without these details it is impossible to verify whether the proxy truly reduces sensitivity to token-level sampling noise or whether it introduces circularity with fitted parameters.

    Authors: We agree that the high-level presentation in the main text requires expansion for verifiability. Appendix B of the manuscript contains the full natural-gradient derivation, starting from the policy gradient and deriving the entropy drift equation under the first-order approximation (neglecting higher-order terms) with the assumption of unbiased advantage estimates at response granularity. We will move the key update equations and steps into the main text. The proxy follows directly from the advantage-surprisal interaction using only quantities already present in standard RL (no additional fitted parameters), eliminating circularity while aligning modulation with response-level actions to reduce token noise sensitivity. revision: yes

  2. Referee: [experimental evaluation] The experimental claims of consistent improvement, including the reported +1.4% gain on SWE-bench-Verified, rest on comparisons whose details are not visible in the provided description. No information is given on the precise baselines, number of random seeds, statistical significance tests, or ablation controls that isolate the contribution of the advantage-rescaling step from other implementation choices. This absence makes it difficult to assess whether the observed gains are attributable to AEM or to uncontrolled variance.

    Authors: We acknowledge that the summary description omitted key controls. Section 4 and Appendix C detail the baselines (PPO, GRPO, and SOTA software-engineering RL framework), use of 5 random seeds with mean/std reporting, paired t-tests for significance, and ablations that isolate the advantage-rescaling component while holding all other hyperparameters fixed. The +1.4% gain on SWE-bench-Verified is from controlled integration into the SOTA framework. We will add a summary table of these controls to the main text to make attribution to AEM explicit. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected in derivation chain

full rationale

The paper claims to derive a response-level uncertainty proxy by first showing mathematically that entropy drift under natural-gradient updates is governed by the interaction between sampled-response advantage and relative surprisal, then using that result to motivate the proxy for rescaling advantages. This is presented as an analysis lifted from token to response level to align with agentic action granularity, without evidence of self-definition (e.g., defining the proxy in terms of itself), renaming a fitted input as a prediction, or load-bearing self-citation chains that reduce the central claim to unverified prior work by the same authors. The derivation is self-contained against the stated RL assumptions and does not reduce by construction to its inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Only abstract available so ledger is partial; main postulated element is the response-level uncertainty proxy derived from entropy analysis.

axioms (1)
  • domain assumption Environment is affected by a complete response rather than an individual token
    Explicitly stated as the reason for lifting entropy dynamics to response level.
invented entities (1)
  • response-level uncertainty proxy no independent evidence
    purpose: To rescale advantages for credit assignment in multi-turn RL
    Derived from entropy drift analysis; no independent evidence or falsifiable prediction provided in abstract.

pith-pipeline@v0.9.0 · 5618 in / 1191 out tokens · 47778 ms · 2026-05-11T00:50:54.908908+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

Reference graph

Works this paper leans on

68 extracted references · 15 canonical work pages · 9 internal anchors

  1. [1]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Policy Gradient Methods for Reinforcement Learning with Function Approximation , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  2. [2]

    2024 , booktitle =

    Huang, Jiangyong and Yong, Silong and Ma, Xiaojian and Linghu, Xiongkun and Li, Puhao and Wang, Yan and Li, Qing and Zhu, Song-Chun and Jia, Baoxiong and Huang, Siyuan , title =. 2024 , booktitle =

    2024

  3. [3]

    How to Train Your

    Dheeraj Vattikonda and Santhoshi Ravichandran and Emiliano Penaloza and Hadi Nekoei and Thibault Le Sellier de Chezelles and Megh Thakkar and Nicolas Gontier and Miguel Mu. How to Train Your. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  4. [4]

    Silver, David and Huang, Aja and Maddison, Chris J. and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others , journal =. Mastering the Game of

  5. [5]

    Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , journal=

  6. [6]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. Advances in Neural Information Processing Systems (NeurIPS) , year =

  7. [7]

    Proceedings of The 33rd International Conference on Machine Learning , pages =

    Asynchronous Methods for Deep Reinforcement Learning , author =. Proceedings of The 33rd International Conference on Machine Learning , pages =. 2016 , editor =

    2016

  8. [8]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  9. [9]

    International Conference on Learning Representations (ICLR) , year =

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  10. [10]

    Transactions on Machine Learning Research , year=

    The Landscape of Agentic Reinforcement Learning for LLMs: A Survey , author=. Transactions on Machine Learning Research , year=

  11. [11]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    WebEvolver: Enhancing Web Agent Self-Improvement with Co-evolving World Model , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  12. [12]

    The Fourteenth International Conference on Learning Representations , year =

    Test-Time Adaptation for LLM Agents via Environment Interaction , author=. The Fourteenth International Conference on Learning Representations , year =

  13. [13]

    Advances in Neural Information Processing Systems , volume=

    Avatar: Optimizing llm agents for tool usage via contrastive reasoning , author=. Advances in Neural Information Processing Systems , volume=

  14. [14]

    ICML 2025 Workshop on Computer Use Agents , year=

    Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment , author=. ICML 2025 Workshop on Computer Use Agents , year=

    2025

  15. [15]

    International Conference on Learning Representations (ICLR) , year =

    Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents , author =. International Conference on Learning Representations (ICLR) , year =

  16. [16]

    WebGPT: Browser-assisted question-answering with human feedback

    WebGPT: Browser-assisted Question-answering with Human Feedback , author =. arXiv preprint arXiv:2112.09332 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author =. arXiv preprint arXiv:2307.13854 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Conference on Robot Learning (CoRL) , year =

    Do As I Can, Not As I Say: Grounding Language in Robotic Affordances , author =. Conference on Robot Learning (CoRL) , year =

  19. [19]

    Voyager: An Open-Ended Embodied Agent with Large Language Models

    Voyager: An Open-Ended Embodied Agent with Large Language Models , author =. arXiv preprint arXiv:2305.16291 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    STaR: Bootstrapping Reasoning with Reasoning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  21. [21]

    Training language models to follow instructions with human feedback

    Training language models to follow instructions with human feedback , author =. arXiv preprint arXiv:2203.02155 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    2021 , url =

    Mohit Shridhar and Xingdi Yuan and Marc-Alexandre C\^ot\'e and Yonatan Bisk and Adam Trischler and Matthew Hausknecht , booktitle =. 2021 , url =

    2021

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Webshop: Towards scalable real-world web interaction with grounded language agents , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    2024 , url=

    Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan , booktitle=. 2024 , url=

    2024

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    Swe-agent: Agent-computer interfaces enable automated software engineering , author=. Advances in Neural Information Processing Systems , volume=

  27. [27]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    SWE-smith: Scaling Data for Software Engineering Agents , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  28. [28]

    Advances in Neural Information Processing Systems , volume=

    Taskbench: Benchmarking large language models for task automation , author=. Advances in Neural Information Processing Systems , volume=

  29. [29]

    Advances in neural information processing systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=

  30. [30]

    NeurIPS 2024 , year=

    Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making , author=. NeurIPS 2024 , year=

    2024

  31. [31]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Embodied multi-modal agent trained by an llm from a parallel textworld , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  32. [32]

    Forty-second International Conference on Machine Learning , year=

    Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks , author=. Forty-second International Conference on Machine Learning , year=

  33. [33]

    Advances in Neural Information Processing Systems , year=

    A-Mem: Agentic memory for llm agents , author=. Advances in Neural Information Processing Systems , year=

  34. [34]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Agent-pro: Learning to evolve via policy-level reflection and optimization , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  35. [35]

    The Fourteenth International Conference on Learning Representations , year=

    ATPO: ADAPTIVE TREE POLICY OPTIMIZATION FOR MULTI-TURN MEDICAL DIALOGUE , author=. The Fourteenth International Conference on Learning Representations , year=

  36. [36]

    The Fourteenth International Conference on Learning Representations , year=

    TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models , author=. The Fourteenth International Conference on Learning Representations , year=

  37. [37]

    The twelfth international conference on learning representations , year=

    Let's verify step by step , author=. The twelfth international conference on learning representations , year=

  38. [38]

    Qwen3 Technical Report

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [39]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  40. [40]

    The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

    The entropy mechanism of reinforcement learning for reasoning language models , author=. arXiv preprint arXiv:2505.22617 , year=

    work page internal anchor Pith review arXiv

  41. [41]

    Qwen2.5-Coder Technical Report

    Qwen2. 5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  42. [42]

    On Entropy Control in

    Han Shen , booktitle=. On Entropy Control in

  43. [43]

    2024 , url=

    John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press , booktitle=. 2024 , url=

    2024

  44. [44]

    arXiv preprint arXiv:2502.01600 , year=

    Reinforcement learning for long-horizon interactive llm agents , author=. arXiv preprint arXiv:2502.01600 , year=

    work page arXiv

  45. [45]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  46. [46]

    arXiv preprint arXiv:2509.09265 , year=

    Harnessing uncertainty: Entropy-modulated policy gradients for long-horizon llm agents , author=. arXiv preprint arXiv:2509.09265 , year=

    work page arXiv

  47. [47]

    Junbo Li, Peng Zhou, Rui Meng, Meet P

    Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Training , author=. arXiv preprint arXiv:2602.19225 , year=

    work page arXiv

  48. [48]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    MobileUse: A Hierarchical Reflection-Driven GUI Agent for Autonomous Mobile Operation , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  49. [49]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  50. [50]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  51. [51]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  52. [52]

    Hindsight credit assignment for long-horizon llm agents, 2026

    Hindsight Credit Assignment for Long-Horizon LLM Agents , author=. arXiv preprint arXiv:2603.08754 , year=

    work page arXiv

  53. [53]

    The Fourteenth International Conference on Learning Representations , year=

    Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks , author=. The Fourteenth International Conference on Learning Representations , year=

  54. [54]

    Epo: Entropy-regularized policy optimization for llm agents reinforcement learning.arXiv preprint arXiv:2509.22576,

    Epo: Entropy-regularized policy optimization for llm agents reinforcement learning , author=. arXiv preprint arXiv:2509.22576 , year=

    work page arXiv

  55. [55]

    2025 , howpublished=

    rLLM: A Framework for Post-Training Language Agents , author=. 2025 , howpublished=

    2025

  56. [56]

    arXiv preprint arXiv:2505.20732 , year=

    Spa-rl: Reinforcing llm agents via stepwise progress attribution , author=. arXiv preprint arXiv:2505.20732 , year=

    work page arXiv

  57. [57]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Group-in-Group Policy Optimization for LLM Agent Training , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  58. [58]

    DeepSWE: Training a State-of-the-Art Coding Agent from Scratch by Scaling RL , author =

  59. [59]

    ICML 2024 , year =

    R2E: Turning any Github Repository into a Programming Agent Environment , author=. ICML 2024 , year =

    2024

  60. [60]

    Group Sequence Policy Optimization

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  61. [61]

    2026 , booktitle =

    Dong, Guanting and Bao, Licheng and Wang, Zhongyuan and Zhao, Kangzhi and Li, Xiaoxi and Jin, Jiajie and Yang, Jinghan and Mao, Hangyu and Zhang, Fuzheng and Gai, Kun and Zhou, Guorui and Zhu, Yutao and Wen, Ji-Rong and Dou, Zhicheng , title =. 2026 , booktitle =

    2026

  62. [62]

    The Fourteenth International Conference on Learning Representations , year=

    Agentic Reinforced Policy Optimization , author=. The Fourteenth International Conference on Learning Representations , year=

  63. [63]

    2026 , eprint=

    Flexible Entropy Control in RLVR with Gradient-Preserving Perspective , author=. 2026 , eprint=

    2026

  64. [64]

    2000 , publisher =

    Methods of Information Geometry , author =. 2000 , publisher =

    2000

  65. [65]

    2020 , publisher =

    An Elementary Introduction to Information Geometry , author =. 2020 , publisher =

    2020

  66. [66]

    Advances in neural information processing systems , volume=

    A natural policy gradient , author=. Advances in neural information processing systems , volume=

  67. [67]

    The Fourteenth International Conference on Learning Representations , year=

    Entropy-preserving reinforcement learning , author=. The Fourteenth International Conference on Learning Representations , year=

  68. [68]

    Actor-Critic Algorithms , url =

    Konda, Vijay and Tsitsiklis, John , booktitle =. Actor-Critic Algorithms , url =