pith. machine review for the scientific record.
sign in

arxiv: 2510.00568 · v3 · submitted 2025-10-01 · 💻 cs.CL

ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards

Shiyu Li , Yang Tang , Yifan Wang , Peiming Li , Xi Chen This is my paper

Pith reviewed 2026-05-18 11:16 UTC · model grok-4.3

classification 💻 cs.CL
keywords search agentsself-correctionreinforcement learningLLM agentsprocess rewardreasoning benchmarksknowledge intensive tasks
0
0 comments X

The pith

Search agents can recover from bad reasoning paths by calling a judge action and using dense rewards that score both facts and usefulness.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to fix a core flaw in RL training for LLM search agents: sparse or rule-based rewards often lock agents into wrong paths with no way back. ReSeek adds a self-correction step where the agent can invoke a JUDGE action to spot errors in the information gathered so far and then re-plan the rest of its search. This process is steered by a dense reward that separately credits factual correctness and genuine utility for answering the original query. The authors also release FictionalHot, a fresh benchmark of recent questions meant to block the data contamination that affects older test sets. If the approach works, agents become able to treat complex searches as recoverable, iterative processes rather than all-or-nothing sequences.

Core claim

ReSeek is a self-correcting framework for LLM search agents that adds a JUDGE action the agent can call to evaluate gathered information and re-plan its strategy, paired with an instructive process reward split into a correctness component for factual retrieval and a utility component for query relevance; agents trained this way achieve higher task success rates and more faithful reasoning paths than prior methods on the FictionalHot benchmark.

What carries the argument

The self-correction mechanism, in which the agent invokes a JUDGE action mid-episode to identify errors and replan its search strategy.

If this is right

  • Agents stop committing permanently to suboptimal search paths and instead recover mid-episode.
  • Task success rates rise on multi-step knowledge queries that require locating and using several facts.
  • The final reasoning traces stay closer to information that actually helps answer the query.
  • Results remain strong on a benchmark built from recent questions to reduce contamination risk.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same judge-and-replan loop could be tested on other agent settings such as tool-use or multi-hop planning.
  • Dense process rewards that separate correctness from utility might reduce the amount of human preference data needed for agent training.
  • Making the JUDGE step fully internal rather than a separate call could lower latency in deployed systems.

Load-bearing premise

Calling the JUDGE action lets the agent recover from errors without creating new mistakes or adding prohibitive extra steps, and FictionalHot genuinely prevents the data contamination seen in earlier benchmarks.

What would settle it

Run the same ReSeek-trained agents on a held-out set of tasks while blocking the JUDGE action entirely and measure whether task success rate and path faithfulness fall to the level of the non-self-correcting baselines.

Figures

Figures reproduced from arXiv: 2510.00568 by Peiming Li, Shiyu Li, Xi Chen, Yang Tang, Yifan Wang.

Figure 1
Figure 1. Figure 1: A comparison of reasoning processes on a multi-hop question about an obscure entity. Standard RAG (a) fails as it cannot perform sequential reasoning. Vanilla agent like Search-R1 (b) reasons sequentially but gets stuck on its initial path. In contrast, our agent (c) demonstrates robust self-correction: it uses a low process reward (rp) to identify the unproductive intermediate step, triggers a JUDGE actio… view at source ↗
Figure 2
Figure 2. Figure 2: Training the agent’s self-evaluation capability. We train the agent via policy optimiza￾tion to master the JUDGE action. A reward signal is generated by comparing the agent’s judgment against an “ideal” one, which is determined by the rerank score between the current search observa￾tion and the GT answer. This reward guides the policy to learn effective self-correction. A key contribution of our framework … view at source ↗
Figure 3
Figure 3. Figure 3: The FictionalHot benchmark con￾struction process: transforming a real-world question answer sample into a fictional sample with fictional question and documents. The construction of FictionalHot follows a three-step pipeline, as illustrated in the Fig￾ure 3. First, we draw a 10% random sample of seed questions from the seven benchmarks mentioned before. Next, these questions are paraphrased by GPT-5. This … view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study on search-embedding choice and base/instruction models. We evalu￾ate our method on the Wiki18 corpus across dif￾ferent backbone and embedding models over all datasets. The dashed line denotes the mean per￾formance (excluding BM25). Interaction Turns Study. We perform an ablation over the number of turns to isolate the effect of the action budget and to test whether models can leverage iterat… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative analysis of our JUDGE action impact. We categorize each case as ‘Posi￾tive’ (beneficial intervention), ‘Negative’ (detrimental intervention), or ‘Normal’. To provide a deeper, qualitative understanding of our judge mechanism’s effectiveness beyond ag￾gregate scores, we conducted a fine-grained analysis of its behavior on a case-by-case basis. We classify the impact of each judge intervention in… view at source ↗
Figure 7
Figure 7. Figure 7: A baseline agent (Search-R1) failing the two-hop question. The agent attempts to solve the problem in a single step and incorrectly extracts the year of the shooting (1985) instead of the correct year of death (1987). intermediate result before initiating a second, focused search for the death year. This structured process prevents premature conclusions and leads to the correct answer where the baseline fa… view at source ↗
Figure 8
Figure 8. Figure 8: A case study of ReSeek on a two-hop question. The agent first identifies the shooter (“Dennis Allen”) and then finds his death year. The judge action is used to validate the intermediate finding before proceeding to the second reasoning step. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: A baseline agent (Search-R1) failing the two-hop question. While the agent’s search successfully identifies the creator, “Loren Bouchard,” it fails to perform the necessary follow-up search for their birth date. It prematurely concludes with a hallucinated and incorrect answer. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A case study of ReSeek on a two-hop question. The agent first attempts a broad search but correctly uses the judge action to determine the retrieved information is insufficient. It then extracts the creator’s name (“Loren Bouchard”) from the initial context and initiates a second, focused search for the birth date. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
read the original abstract

Search agents powered by Large Language Models (LLMs) have demonstrated significant potential in tackling knowledge-intensive tasks. Reinforcement learning (RL) has emerged as a powerful paradigm for training these agents to perform complex, multi-step reasoning. However, prior RL-based methods often rely on sparse or rule-based rewards, which can lead agents to commit to suboptimal or erroneous reasoning paths without the ability to recover. To address these limitations, we propose ReSeek, a novel self-correcting framework for training search agents. Our framework introduces a self-correction mechanism that empowers the agent to dynamically identify and recover from erroneous search paths during an episode. By invoking a special JUDGE action, the agent can judge the information and re-plan its search strategy. To guide this process, we design a dense, instructive process reward function, which decomposes into a correctness reward for retrieving factual information and a utility reward for finding information genuinely useful for the query. Furthermore, to mitigate the risk of data contamination in existing datasets, we introduce FictionalHot, a new and challenging benchmark with recently curated questions requiring complex reasoning. Being intuitively reasonable and practically simple, extensive experiments show that agents trained with ReSeek significantly outperform SOTA baselines in task success rate and path faithfulness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces ReSeek, a self-correcting framework for training LLM-based search agents. It incorporates a JUDGE action that enables the agent to identify errors in its current search path and re-plan, guided by a dense instructive process reward that decomposes into a correctness component for factual retrieval and a utility component for query-relevant information. The authors also contribute the FictionalHot benchmark of recently curated questions designed to reduce data contamination. The central empirical claim is that ReSeek-trained agents significantly outperform SOTA baselines on task success rate and path faithfulness.

Significance. If the results hold under rigorous validation, the work could advance RL training of reliable search agents by showing how self-correction and dense instructive rewards address the limitations of sparse or rule-based signals. The FictionalHot benchmark addresses a genuine practical concern in the field. The approach is described as intuitively reasonable and practically simple, with potential to influence agent design for knowledge-intensive tasks.

major comments (2)
  1. [Experimental Results] Experimental Results section: The claim that ReSeek agents significantly outperform baselines in task success rate and path faithfulness is load-bearing on the JUDGE self-correction mechanism enabling genuine recovery. No ablations isolate the JUDGE action from the dense correctness+utility rewards, and the manuscript provides no statistics on average JUDGE invocations per episode, correction success rate, or added token overhead. Without these, outperformance on FictionalHot could be driven primarily by the denser reward signal rather than self-correction.
  2. [Method] Method section (framework description): The integration of the JUDGE action as an LLM-based step within the same agent policy risks introducing new factual errors or unproductive re-plans. The manuscript does not detail how the policy is trained to invoke JUDGE productively or provide evidence that it preserves or improves path faithfulness, which directly underpins the self-correcting claim.
minor comments (2)
  1. [Abstract] Abstract: The statement that agents 'significantly outperform' would be strengthened by including at least one quantitative result or baseline name.
  2. [Method] Notation for the reward function: The decomposition into correctness and utility components could be presented with explicit equations or pseudocode to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and insightful comments on our manuscript. We have carefully reviewed the major concerns regarding the isolation of the JUDGE mechanism and the details of its training and impact on faithfulness. Below we provide point-by-point responses and indicate the revisions we will make to address these points.

read point-by-point responses

  1. Referee: [Experimental Results] Experimental Results section: The claim that ReSeek agents significantly outperform baselines in task success rate and path faithfulness is load-bearing on the JUDGE self-correction mechanism enabling genuine recovery. No ablations isolate the JUDGE action from the dense correctness+utility rewards, and the manuscript provides no statistics on average JUDGE invocations per episode, correction success rate, or added token overhead. Without these, outperformance on FictionalHot could be driven primarily by the denser reward signal rather than self-correction.

    Authors: We agree that isolating the contribution of the JUDGE action is important for substantiating the self-correction claim. Our current experiments evaluate the full ReSeek framework (JUDGE plus dense instructive rewards) against baselines, and the combined system yields the reported gains in success rate and faithfulness. To directly address the concern, we will add new ablation experiments in the revised manuscript that compare the full ReSeek agent against a variant using only the dense rewards without the JUDGE action. We will also report statistics on average JUDGE invocations per episode, the rate at which JUDGE invocations lead to successful corrections, and the additional token overhead incurred. These additions will clarify whether the performance improvements stem from self-correction or primarily from the denser reward signal. revision: yes

  2. Referee: [Method] Method section (framework description): The integration of the JUDGE action as an LLM-based step within the same agent policy risks introducing new factual errors or unproductive re-plans. The manuscript does not detail how the policy is trained to invoke JUDGE productively or provide evidence that it preserves or improves path faithfulness, which directly underpins the self-correcting claim.

    Authors: We acknowledge the potential risk that an LLM-based JUDGE step could introduce errors or unproductive re-plans. The policy is trained end-to-end with the dense instructive reward, which explicitly rewards paths that achieve higher factual correctness and query utility; this training signal encourages the agent to invoke JUDGE only when it leads to net improvement rather than indiscriminately. In the revised manuscript we will expand the Method section with additional details on the reward decomposition and training dynamics that guide productive JUDGE usage. We will also include further analysis of path faithfulness metrics, demonstrating that ReSeek agents achieve higher faithfulness scores than baselines, indicating that the self-correction mechanism preserves or improves overall path quality rather than degrading it. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework with independent experimental claims

full rationale

The paper introduces ReSeek as a practical framework combining a JUDGE-based self-correction action with a dense process reward (correctness plus utility) and evaluates it empirically against baselines on the newly introduced FictionalHot benchmark. No equations, derivations, fitted parameters, or mathematical claims are present that could reduce by construction to inputs. No load-bearing self-citations, uniqueness theorems, or ansatzes are invoked to justify core results. The performance claims rest on direct experimental comparisons rather than any self-referential chain, making the work self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review performed on abstract only; no explicit free parameters, background axioms, or newly postulated entities are identifiable from the provided text. The JUDGE action and reward function are introduced as part of the framework but lack sufficient detail to classify as free parameters or invented entities.

pith-pipeline@v0.9.0 · 5756 in / 1160 out tokens · 41730 ms · 2026-05-18T11:16:21.782168+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

    cs.CV 2026-04 unverdicted novelty 6.0

    A decision-based agent for KB-VQA learns to dynamically select retrieval or answer actions over multiple steps and achieves state-of-the-art results on InfoSeek and E-VQA after fine-tuning on automatically collected t...

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 1 Pith paper · 14 internal anchors

  1. [1]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection.ArXiv, abs/2310.11511,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

    URL https://api.semanticscholar.org/CorpusID:264288947. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Love- nia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V . Do, Yan Xu, and Pascale Fung. A mul- titask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and inter- activity.ArXiv, abs/2302.04023,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  4. [4]

    URLhttps://arxiv.org/abs/2310. 05915. Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Learning to reason with search for llms via reinforce- ment learning.arXiv preprint arXiv:2503.19470,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Training large language models for retrieval-augmented question answering through backtracking correc- tion

    Huawen Feng, ZekunYao, Junhao Zheng, and Qianli Ma. Training large language models for retrieval-augmented question answering through backtracking correc- tion. In Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.),International Con- ference on Representation Learning, volume 2025, pp. 51866–51884,

    work page 2025

  6. [6]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    URL https://proceedings.iclr.cc/paper_files/paper/2025/file/ 80790082a3b0e4fa9061730ee876f5ba-Paper-Conference.pdf. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Constructing A Multi- hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A Multi- hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. InProceedings of the 28th International Conference on Computational Linguistics, pp. 6609–6625. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.580. URLhttps://www. a...

    work page doi:10.18653/v1/2020.coling-main.580 2020

  8. [8]

    doi: 10.1145/3571730

    ISSN 1557-7341. doi: 10.1145/3571730. URL http://dx.doi.org/10.1145/3571730. Pengcheng Jiang, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. s3: You don’t need that much data to train a search agent via rl. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,

    work page doi:10.1145/3571730 2025

  9. [9]

    Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

    Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation.ArXiv, abs/2305.06983,

    work page arXiv

  10. [10]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    URLhttps://api.semanticscholar.org/CorpusID: 258615731. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [12]

    Conan-embedding-v2: Training an llm from scratch for text embeddings.arXiv preprint arXiv:2509.12892, 2025a

    Shiyu Li, Yang Tang, Ruijie Liu, Shi-Zhe Chen, and Xi Chen. Conan-embedding-v2: Training an llm from scratch for text embeddings.arXiv preprint arXiv:2509.12892, 2025a. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models.ArXiv, abs/2501.05366, ...

    work page arXiv

  12. [13]

    Negar Maleki, Balaji Padmanabhan, and Kaushik Dutta

    URLhttps://arxiv.org/abs/2505.07773. Negar Maleki, Balaji Padmanabhan, and Kaushik Dutta. Ai hallucinations: a misnomer worth clarifying. In2024 IEEE conference on artificial intelligence (CAI), pp. 133–138. IEEE,

    work page arXiv

  13. [14]

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non- Parametric Memories. URLhttps://arxiv.org/abs/2212.10511. 11 Preprint version. Work in Progress. OpenAI. Introducing chatgpt.CoRR,

    work page internal anchor Pith review arXiv

  14. [15]

    Measuring and Narrowing the Compositionality Gap in Language Models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and Narrowing the Compositionality Gap in Language Models. URLhttps://arxiv.org/ abs/2210.03350. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T ¨ur, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [16]

    ToolRL: Reward is All Tool Learning Needs

    URLhttps://arxiv. org/abs/2504.13958. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qi...

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [17]

    Qwen2.5 Technical Report

    URLhttps://arxiv.org/abs/2412.15115. Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond.Foundations and Trends® in Information Retrieval, 3(4):333–389,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [18]

    R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

    URLhttps://api.semanticscholar.org/CorpusID: 258833055. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforce- ment Learning. a. doi: 10.48550/arXiv.2503.05592. URLhttp://arxiv.org/abs/2503. 05592. Huatong Song, Jinhao Jiang, Wenqi...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.05592

  18. [19]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    URLhttps://api.semanticscholar.org/ CorpusID:278367823. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop Questions via Single-hop Question Composition. 10: 539–554. ISSN 2307-387X. doi: 10.1162/tacl a 00475. URLhttps:// direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00475/110996/ MuSiQue-Multihop-Questions-via-...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1162/tacl

  19. [20]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [21]

    ReAct: Synergizing Reasoning and Acting in Language Models

    12 Preprint version. Work in Progress. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question An- swering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 2369–2380. Association for ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/d18-1259 2018

  21. [22]

    What’s behind ppo’s collapse in long-cot? value optimization holds the secret.arXiv preprint arXiv:2503.01491,

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value optimization holds the secret.arXiv preprint arXiv:2503.01491,

    work page arXiv

  22. [23]

    Agenttuning: Enabling generalized agent abilities for llms

    URLhttps://arxiv.org/abs/ 2310.12823. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025a. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu,...

    work page arXiv

  23. [24]

    DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments.arXiv preprint arXiv:2504.03160,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [25]

    Our experiments are built upon the internalverlreinforcement learning framework and executed on a cluster equipped with Huawei Ascend NPUs

    A APPENDIX A.1 IMPLEMENTATIONDETAILS We provide a detailed description of our implementation to ensure the reproducibility of our results. Our experiments are built upon the internalverlreinforcement learning framework and executed on a cluster equipped with Huawei Ascend NPUs. Model and DataThe core of our agent is theQwen2.5-3B-Instructmodel, which serv...

    work page 2048

  25. [26]

    Dennis Allen

    To stabilize training and prevent the policy from deviating excessively from the reference model, we incorporated a KL divergence penalty with a coefficient (β) of0.001, calculated using thelow var klformulation. For credit assignment, we used a discount factor (γ) of 0.99 and Generalized Advantage Estimation (GAE) with aλof 0.95. During the rollout phase...

    work page arXiv 2025