Pith. sign in

REVIEW 3 major objections 7 minor 37 references

Information-gain tree rollouts reshape LLM agent training

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 12:41 UTC pith:UFG4QA67

load-bearing objection The theory is correct but generic; the practical gains rest on empirically-tuned components the theory doesn't cover. the 3 major comments →

arxiv 2607.06223 v1 pith:UFG4QA67 submitted 2026-07-07 cs.AI

Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents

Yijun Zhang , Fan Xu , Jiaxin Ding , Yule Xie , Shiqing Gao , Xin Ding , Haoxiang Zhang , Luoyi Fu
show 1 more author
Xinbing Wang
This is my paper
classification cs.AI
keywords rolloutpolicyoptimizationagentsbranchesbudgetgain-basedigrpo
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes IGRPO, a framework for training multi-turn search agents that reorganizes how rollout computation is spent. Instead of allocating search budget uniformly across branches, the method scores each intermediate state by how much it increases the policy's likelihood of producing the correct answer—a quantity called information gain—and preferentially expands nodes with higher scores. The central theoretical result is that this expansion rule induces a limiting distribution over trajectories proportional to the original policy probability times an exponential tilt toward higher cumulative information gain. This induced distribution serves as a teacher target for policy optimization, unifying adaptive tree-structured exploration with principled policy learning. Experiments on seven question-answering benchmarks show consistent gains over chain-based and tree-based baselines under identical rollout budgets, with the framework also producing more efficient search behavior at inference time.

Core claim

The paper's core result is that information-gain-based probabilistic node expansion in tree-structured rollouts induces a well-defined limiting teacher distribution over trajectories—specifically, a distribution proportional to the original policy probability multiplied by an exponential weight on cumulative information gain. This distribution is shown to be the unique optimizer of a variational objective balancing expected information value against KL divergence from the current policy. By training against samples drawn from this teacher distribution, the agent learns to concentrate its search behavior on informative intermediate states rather than wasting computation on unpromising paths.

What carries the argument

The central object is the information gain of an intermediate search node, defined as the marginal increase in the policy's normalized likelihood of the ground-truth answer when moving from a parent node to a child node. This signal drives a softmax-temperature expansion rule that probabilistically allocates rollout budget across active nodes, and the resulting tree-structured rollout process converges to an exponentially tilted trajectory distribution that serves as the optimization target.

Load-bearing premise

The entire framework rests on the assumption that the increase in the policy's likelihood of the ground-truth answer after reaching an intermediate state is a reliable proxy for whether that state will lead to a correct final answer. This correlation is supported by a preliminary observation on one dataset but is not formally guaranteed across tasks or training stages.

What would settle it

If the correlation between cumulative information gain and final reward were shown to be weak or unstable across diverse task types, model sizes, or training stages, the teacher distribution would no longer align with the reward-based objective, and the policy optimization target would lose its justification.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the information-gain signal reliably proxies node utility, then any tree-structured rollout for LLM agents could replace uniform or heuristic branching with a principled probabilistic allocation grounded in a variational objective.
  • The teacher-distribution framework could extend to other node-level scoring functions beyond information gain, as the paper itself notes, potentially covering domains where answer-likelihood signals are difficult to define.
  • The reduction in inference-time tool calls suggests that training against an informativeness-biased teacher distribution leads agents to internalize more selective search strategies, not just better final answers.
  • Smaller models appear to benefit more from information-gain-guided rollouts, which could make the framework particularly relevant for deploying capable search agents on resource-constrained hardware.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The dependence on ground-truth answer likelihood during training means the framework is naturally suited to tasks with known correct answers, but extending it to open-ended or subjective tasks would require a substitute node-utility signal whose correlation with downstream reward is comparably strong.
  • The theoretical limiting distribution requires per-layer expansion budgets approaching infinity; in practice the method uses small budgets, so the gap between the finite-sample rollout distribution and the theoretical teacher distribution deserves empirical quantification.
  • If information gain and final reward are strongly correlated, the variational bridge suggests that information-gain-guided rollouts approximate reward-weighted trajectory sampling, which connects this work to a broader family of reward-aware tree search methods.
  • The entropy dynamics observed during training—initial decrease followed by marginal increase—suggest that the method implicitly regulates exploration-exploitation trade-offs, which could be studied as a standalone mechanism for preventing premature convergence in multi-turn agent training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper proposes Information Gain-based Rollout Policy Optimization (IGRPO), a framework that allocates rollout budget in tree-structured search according to node-level information gain (IG) derived from answer-likelihood changes. The authors prove that this rollout mechanism induces a limiting teacher distribution μ_θ(o) ∝ π_θ(o) exp(γV(o)) (Theorem 1) and that this distribution is the optimizer of a variational objective (Theorem 2). They then use this induced distribution as a sampling target for group-based policy optimization, evaluated on seven search-augmented QA benchmarks. The theoretical results are mathematically correct and self-contained. However, the bridge from the variational objective (optimizing E[V(o)]) to the reward-based training objective (optimizing E[R(o)]) is informal, relying on a preliminary empirical observation (Fig. 3) rather than a formal guarantee. Additionally, several practical design choices that materially affect performance—heuristic val(·), reward schedules, per-node expansion caps—are not covered by the theory. The empirical results are solid and the code is released, but the 'principled' framing is stronger than what the theory supports.

Significance. The paper addresses a real and important problem: efficient allocation of rollout budget in multi-turn LLM agent training. The idea of using answer-likelihood information gain as a node-level utility signal is well-motivated, and the theoretical characterization of the induced limiting distribution (Theorems 1–2) is a clean contribution. The experiments span seven benchmarks with strong baselines, and the code release supports reproducibility. The entropy analysis (Fig. 4) and tool-call efficiency comparison (Table 2) add practical value. However, the significance is tempered by the gap between theory and practice: the theorems are generic (they hold for any val(·)), while the empirical gains depend on heuristic choices the theory is silent about. The paper would be more impactful if this gap were acknowledged more carefully and if the ablations isolated the contribution of the principled limiting distribution from the post-hoc-tuned components.

major comments (3)
  1. §4.2, Theorem 2 and the bridge to Eq. (2): Theorem 2 shows μ_θ optimizes E[V(o)] − (1/γ)KL(μ||π_θ), not the actual training objective E[R(o)] − βKL(π_θ||π_ref) in Eq. (2). The paper bridges this gap by claiming V(o) correlates with R(o), citing Fig. 3. However, Fig. 3 shows only that average cumulative IG differs between successful and failed trajectory groups on one dataset (IGPO training on Qwen2.5-3B). This does not establish a trajectory-level correlation between V(o) and R(o), nor does it show the correlation holds across all seven evaluation benchmarks. The claim that μ_θ is a 'reward-aligned teacher' (§4.2) is thus stronger than what is empirically demonstrated. The authors should either (a) soften the framing to acknowledge this as a heuristic bridge supported by preliminary evidence, or (b) provide a more systematic correlation analysis across datasets and trajectory levels.
  2. §4.2, Eq. (10) and Appendix C.2: The specific val(·) = (s_θ(h;a) + IG(h))/2 is a heuristic average of cumulative and instantaneous IG with no theoretical justification. Since Theorems 1–2 hold for any val(·), the theory does not predict why this particular form is preferred. The γ=0 ablation (Fig. 5, 42.2% vs 45.4%) shows IG contributes ~3%, but this does not isolate whether the gain comes from the principled limiting distribution or from the post-hoc-tuned val(·) and reward schedule. An ablation comparing val(·) = s_θ(h;a) alone vs. the proposed average, or val(·) = IG(h) alone, would help disentangle the contribution of the principled framework from the heuristic design choices.
  3. Appendix C.1: The per-node expansion cap of 2 ('each node is expanded at most twice at each stage') modifies the i.i.d. sampling assumed in the Theorem 1 proof. Under this cap, the number of times a node is selected is bounded, so the law-of-large-numbers convergence in Eq. (21) may not apply when the active set is small relative to the stage budget. The authors should clarify whether Theorem 1 is intended as an asymptotic idealization (B_t → ∞ without caps) and discuss whether the finite-budget, capped regime used in practice still approximates μ_θ. This is important because the 'principled' framing of the method rests on the limiting distribution being a good approximation of the actual sampling process.
minor comments (7)
  1. §4.1, Eq. (3): The answer score s_θ(h;a) is defined as the geometric mean of per-token probabilities (exp of average log-prob). This is a specific normalization choice; the paper should briefly justify why geometric mean rather than, e.g., the raw likelihood or arithmetic mean, especially since the choice affects the scale of IG(h) in Eq. (4).
  2. Table 1: StepSearch results are missing for NQ, TriviaQA, PopQA, and Bamboogle (shown as '–'). The paper should state whether this is because StepSearch was not evaluated on these datasets in the original paper or due to implementation limitations.
  3. §5.1: The statement 'IGRPO expanding temperature γ is set to 1.0 without further tuning' could be more precise. Fig. 5 shows γ ∈ {0, 1, 5} were compared, so γ=1 was selected based on ablation results. Saying 'without further tuning' is slightly misleading; 'selected from {0, 1, 5} based on ablation' would be more accurate.
  4. Fig. 3: The caption mentions 'IGPO training steps' and 'color intensity corresponds to the trajectory turn,' but the figure lacks axis labels for which training steps are shown and does not specify which dataset is used. Adding this information would make the figure self-contained.
  5. Appendix C.2, Reward Design section: The four reward designs are described in prose but not summarized in a table. A small table comparing the four designs and their resulting average scores would make the ablation easier to follow.
  6. §3.1: The reward R(o) = I[â=a] is defined as a binary indicator, but later in §4.2 and Appendix C.2, terminated paths receive rewards based on cumulative IG (G(o) or 0.5G(o)). The paper should clarify upfront that the reward function is modified for terminated paths, perhaps with a unified reward definition in §3.1 or §4.2.
  7. The paper uses both 'information gain' and 'informativeness' somewhat interchangeably. Standardizing terminology (e.g., 'information gain' for IG(h) and 'informativeness' for val(h)) would improve readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee correctly identifies that Theorems 1–2 are mathematically correct but characterizes a gap between the variational objective (optimizing E[V(o)]) and the reward-based training objective (optimizing E[R(o)]), and notes that several practical design choices are not covered by the theory. We address each major comment below and agree that the manuscript should be revised to acknowledge the theory–practice gap more carefully and to add the requested ablations.

read point-by-point responses

  1. Referee: §4.2, Theorem 2 and the bridge to Eq. (2): Theorem 2 shows μ_θ optimizes E[V(o)] − (1/γ)KL(μ||π_θ), not the actual training objective E[R(o)] − βKL(π_θ||π_ref) in Eq. (2). The paper bridges this gap by claiming V(o) correlates with R(o), citing Fig. 3. However, Fig. 3 shows only that average cumulative IG differs between successful and failed trajectory groups on one dataset (IGPO training on Qwen2.5-3B). This does not establish a trajectory-level correlation between V(o) and R(o), nor does it show the correlation holds across all seven evaluation benchmarks. The authors should either (a) soften the framing to acknowledge this as a heuristic bridge supported by preliminary evidence, or (b) provide a more systematic correlation analysis across datasets and trajectory levels.

    Authors: The referee is correct that Fig. 3 provides only preliminary, group-level evidence on a single training configuration, and that the claim that μ_θ is a 'reward-aligned teacher' is stronger than what is formally established. We will revise the manuscript to adopt option (a): we will soften the framing in §4.2 to explicitly acknowledge that the bridge from the variational objective (E[V(o)]) to the reward-based objective (E[R(o)]) is a heuristic step supported by preliminary empirical evidence, not a formal guarantee. Specifically, we will replace the language 'reward-aligned teacher' with more precise wording such as 'informativeness-aligned teacher distribution, empirically correlated with reward,' and add an explicit caveat that Theorem 2 characterizes the optimizer of E[V(o)] − (1/γ)KL(μ||π_θ), while the training objective in Eq. (2) optimizes E[R(o)] − βKL(π_θ||π_ref), and the connection between the two rests on the empirical V–R correlation rather than a formal proof. We will also expand the discussion of Fig. 3 to clearly state its limitations: it shows group-level (successful vs. failed) differences in cumulative IG on the IGPO training run (Qwen2.5-3B, NQ+HotpotQA), not trajectory-level correlations, and not results on all seven evaluation benchmarks. We believe option (a) is the honest and appropriate response given the current evidence, and we will implement it in the revision. revision: yes

  2. Referee: §4.2, Eq. (10) and Appendix C.2: The specific val(·) = (s_θ(h;a) + IG(h))/2 is a heuristic average of cumulative and instantaneous IG with no theoretical justification. Since Theorems 1–2 hold for any val(·), the theory does not predict why this particular form is preferred. The γ=0 ablation (Fig. 5, 42.2% vs 45.4%) shows IG contributes ~3%, but this does not isolate whether the gain comes from the principled limiting distribution or from the post-hoc-tuned val(·) and reward schedule. An ablation comparing val(·) = s_θ(h;a) alone vs. the proposed average, or val(·) = IG(h) alone, would help disentangle the contribution of the principled framework from the heuristic design choices.

    Authors: We agree that the current ablation does not disentangle the contribution of the principled limiting distribution from the specific heuristic choice of val(·). The referee's requested ablation—comparing val(·) = s_θ(h;a) alone, val(·) = IG(h) alone, and the proposed average—is well-motivated and directly addresses whether the gains come from the framework or from the particular val(·) instantiation. We will run these ablations and report the results in the revised manuscript. We note that since Theorems 1–2 hold for any val(·), all three variants induce a principled limiting distribution of the form μ_θ(o) ∝ π_θ(o)exp(γV(o)); they differ only in what V(o) measures. Thus the ablation will help clarify which informativeness signal is most effective in practice, while all variants benefit from the principled framework. We will also add an explicit statement in §4.2 that the specific form of val(·) in Eq. (10) is a heuristic design choice not predicted by the theory, and that the theory only guarantees the limiting distribution result for any valid val(·). revision: yes

  3. Referee: Appendix C.1: The per-node expansion cap of 2 ('each node is expanded at most twice at each stage') modifies the i.i.d. sampling assumed in the Theorem 1 proof. Under this cap, the number of times a node is selected is bounded, so the law-of-large-numbers convergence in Eq. (21) may not apply when the active set is small relative to the stage budget. The authors should clarify whether Theorem 1 is intended as an asymptotic idealization (B_t → ∞ without caps) and discuss whether the finite-budget, capped regime used in practice still approximates μ_θ.

    Authors: The referee is correct that the per-node expansion cap of 2 modifies the i.i.d. sampling assumption underlying the Theorem 1 proof. We will clarify in the revised manuscript that Theorem 1 is intended as an asymptotic idealization: it characterizes the limiting distribution when B_t → ∞ without caps, and the law-of-large-numbers convergence in Eq. (21) applies in that regime. The per-node cap is a practical heuristic used to encourage exploration under a limited rollout budget, and it means the actual finite-budget sampling process only approximately follows μ_θ. We will add an explicit discussion of this point, both in §4.2 (briefly) and in Appendix C.1 (in detail), acknowledging that: (1) the cap introduces a deviation from the i.i.d. assumption, (2) the quality of the approximation depends on the ratio of the stage budget to the active set size, and (3) the empirical results (including the IGRPO-low variant in Table 3, which uses a reduced per-stage budget) suggest the approximation remains useful in practice, but we do not claim it is exact. We will also soften the 'principled' framing to make clear that the theory provides the idealized target distribution, while practical rollout collection approximates it under finite-budget constraints. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; theorems are self-contained and empirical design choices are acknowledged as heuristic

full rationale

The paper's two theoretical results (Theorem 1 and Theorem 2) are self-contained mathematical derivations from the stated rollout mechanism. Theorem 1's proof (Appendix A.1) is a straightforward induction: given expansion probability p(h) ∝ exp(γval(h)), the limiting distribution is μθ(o) ∝ πθ(o)exp(γV(o)) by construction of the sampling process. Theorem 2 uses the Donsker-Varadhan variational formula (external citation [37]) to show this distribution optimizes E[V(o)] − (1/γ)KL(μ||πθ). Neither theorem depends on fitted parameters or self-citation. The specific val(·) instantiation (Eq. 10: val(h) = (sθ(h;a) + IG(h))/2) is presented as a heuristic design choice, not as a theoretical prediction. The bridge from V(o) to R(o) is informal (motivated by Fig. 3's preliminary observation), but the paper does not claim this as a derivation — it explicitly calls it a 'preliminary observation' and a 'practical bridge.' The reward schedule tuning (0.5G then G, Appendix C.2) and per-node expansion cap (Appendix C.1) are hyperparameter choices not covered by the theory, but they are presented as such, not as predictions. The concerns about the theory-practice gap (val(·) heuristic, reward schedule, expansion cap violating i.i.d. assumption) are correctness risks, not circularity. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities or postulated objects. The 'teacher distribution' μθ is a derived mathematical object, not an invented entity. The framework operates entirely within standard RL and LLM training formalisms.

free parameters (4)
  • γ (expansion temperature) = 1.0
    Controls how strongly expansion probability favors informative nodes (Eq. 5). Set to 1.0 without systematic tuning; ablation in §5.3 tests γ∈{0,1,5}.
  • val(·) averaging weight = 0.5 (implicit)
    Eq. 10 uses an unweighted average of cumulative and instantaneous IG. The 0.5 weight is not justified or ablated.
  • Reward schedule for terminated paths = 0.5G(o) for steps 1-150, G(o) for steps 151-200
    Appendix C.2 ablates four reward designs; the best-performing one is selected post-hoc.
  • Per-node expansion cap = 2
    Appendix C.1: 'each node is expanded at most twice at each stage.' This practical constraint affects the induced distribution but is not reflected in the theoretical analysis.
axioms (4)
  • domain assumption Ground-truth answer likelihood sθ(h;a) is a valid proxy for node informativeness
    §4.1, Eq. 3-4. The entire framework depends on this being a meaningful signal. Supported empirically by Fig. 3 but not proven.
  • domain assumption Cumulative information gain V(o) is positively correlated with trajectory reward R(o)
    §4.2, paragraph after Theorem 2. The bridge from variational objective to reward-based GRPO relies on this correlation, supported only by the preliminary observation in Fig. 3.
  • standard math Donsker-Varadhan variational formula applies to the trajectory distribution setting
    Appendix A.2. Standard result [37], correctly applied.
  • domain assumption Per-layer budget Bt → ∞ is a reasonable asymptotic regime for analysis
    Theorem 1 assumes Bt→∞, but experiments use Bt=5. The gap between theory and practice is not formally addressed.

pith-pipeline@v1.1.0-glm · 20895 in / 2874 out tokens · 514565 ms · 2026-07-08T12:41:59.442507+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning has become a promising paradigm for improving large language model (LLM) agents on long-horizon search tasks, where the agent must make a sequence of intermediate decisions before receiving a final outcome. However, existing methods still face a key limitation: the rollout budget is often allocated without explicitly assessing the utility of intermediate states. As a result, substantial computation may be spent on low-value states, even though different branches can vary drastically in their informativeness. In this paper, we propose Information Gain-based Rollout Policy Optimization (IGRPO), a policy optimization framework that treats intermediate-state informativeness as the organizing principle of rollout collection. Specifically, IGRPO performs budget-aware tree-structured rollouts by allocating expansion budget according to node-level informativeness, so that more informative branches are expanded more frequently while unpromising branches are progressively suppressed. We further demonstrate that the information gain-based rollout induces an explicit limiting teacher distribution over trajectories, which naturally yields a clear policy optimization target, thereby unifying adaptive tree-structured exploration with principled policy learning under a single framework. Experiments on seven challenging search-augmented QA benchmarks demonstrate that IGRPO consistently outperforms strong baselines under the same rollout budget constraints, validating the effectiveness of leveraging the induced teacher distribution to guide policy optimization for long-horizon search agents.

Figures

Figures reproduced from arXiv: 2607.06223 by Fan Xu, Haoxiang Zhang, Jiaxin Ding, Luoyi Fu, Shiqing Gao, Xinbing Wang, Xin Ding, Yijun Zhang, Yule Xie.

Figure 1
Figure 1. Figure 1: Illustration of different rollout patterns. Red nodes represent unpromising or even mis [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of IGRPO. Left: During rollout, the expansion budget is adaptively allocated according to node-level informativeness, so that more informative nodes are expanded with higher probability, thereby inducing a distribution µθ that biases sampling toward more promising trajectories. Right: the expansion process between tree-level 1 and tree-level 2 is governed by an informativeness￾aware strategy, wher… view at source ↗
Figure 3
Figure 3. Figure 3: Average cumulative informa￾tion gain at different trajectory turns across IGPO training steps. Blue and red denote successful and failed trajectory groups, respectively, and color intensity corresponds to the trajectory turn. Node-level Information Gain. To instantiate the idea of budget-aware tree expansion, we first require a crite￾rion for assessing how promising each intermediate node is. We employ inf… view at source ↗
Figure 4
Figure 4. Figure 4: Entropy loss during training on Qwen2.5-3B-Instruct. We further examine the training dynamics by comparing the entropy loss of different rollout strategies in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation results. The y-axis shows success rate (%) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt template for the search agent. IGRPO expanding temperature γ is set to 1.0 without further tuning. Qwen2.5-3B-Instruct [35] is trained on 4×H20 GPUs, while Qwen2.5-7B-Instruct is trained on 8×H20 GPUs. Implementation Details. We build our codebase on top of the verl-agent framework [15]. In particular, on top of this framework, we reproduce IGPO [16] and adapt it to use the E5 retriever in our searc… view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics of IGRPO-3B and IGPO-3B. Dashed vertical lines indicate the steps [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Case study of the overall tree-structured rollout behaviors of IGRPO. The number on each [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Detailed analysis of the three highlighted nodes in the case study. Node 1 issues an [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · 17 internal anchors

  1. [1]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020. 9

    work page 1901

  2. [2]

    Palm: Scaling language modeling with pathways.Journal of machine learning research, 24(240): 1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of machine learning research, 24(240): 1–113, 2023

    work page 2023

  3. [3]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arxiv 2023.arXiv preprint arXiv:2302.13971, 10, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    ReAct: Synergizing Reasoning and Acting in Language Models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  5. [5]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

    work page 2023

  6. [6]

    The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey.arXiv preprint arXiv:2509.02547, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Deep Research Agents: A Systematic Examination And Roadmap

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, et al. Deep research agents: A systematic examination and roadmap.arXiv preprint arXiv:2506.18096, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Searching for best practices in retrieval- augmented generation

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for best practices in retrieval- augmented generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17716–17736, 2024

    work page 2024

  9. [9]

    Search-o1: Agentic search-enhanced large reasoning models

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5420–5438, 2025

    work page 2025

  10. [10]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    ToolRL: Reward is All Tool Learning Needs

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  13. [13]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    work page 2022

  14. [14]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    Group-in-Group Policy Optimization for LLM Agent Training

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    arXiv preprint arXiv:2510.14967 , year=

    Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents.arXiv preprint arXiv:2510.14967, 2025. 10

    work page arXiv 2025

  17. [17]

    Tree search for llm agent reinforcement learning.arXiv preprint arXiv:2509.21240, 2025

    Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu. Tree search for llm agent reinforcement learning.arXiv preprint arXiv:2509.21240, 2025

    work page arXiv 2025

  18. [18]

    Agentic entropy-balanced policy optimization.arXiv preprint arXiv:2510.14545, 2025

    Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jing- han Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, et al. Agentic entropy-balanced policy optimization.arXiv preprint arXiv:2510.14545, 2025

    work page arXiv 2025

  19. [19]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024

    work page 2024

  21. [21]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    ZeroSearch: Incentivize the Search Capability of LLMs without Searching

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    Smart-searcher: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning

    Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. Smart-searcher: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 13572–13586, 2025

    work page 2025

  26. [26]

    StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization

    Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization.arXiv preprint arXiv:2505.15107, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Agentic Reinforced Policy Optimization

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [28]

    Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019

    work page 2019

  29. [29]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017

    work page 2017

  30. [30]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pages 9802–9822, 2023

    work page 2023

  31. [31]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018. 11

    work page 2018

  32. [32]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, 2020

    work page 2020

  33. [33]

    Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    work page 2022

  34. [34]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, 2023

    work page 2023

  35. [35]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv e-prints, pages arXiv–2412, 2024

    work page 2024

  36. [36]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  37. [37]

    Asymptotic evaluation of certain markov process expectations for large time, i.Communications on pure and applied mathematics, 28 (1):1–47, 1975

    Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i.Communications on pure and applied mathematics, 28 (1):1–47, 1975. A Derivations and Proofs A.1 Proof of Theorem 1 Setup.Recall that a complete trajectory is denoted by o= (h 0 τ1 − →h1 τ2 − → · · · τT − − →hT ),(11) where all traject...

    work page 1975