Pith. sign in

REVIEW 4 major objections 5 minor 82 references

Contextual Information Policy Optimization for Search Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CIPO claims agents fail when rewards ignore whether retrieved evidence steers the next step; its dense EALR reward plus answer correctness raises average F1 to 0.504 and 0.456 across seven QA benchmarks.

desk verdict CIPO's attention-masked reward is a genuinely new training signal for search agents; the experiments are broad and the counterfactual analysis is honest, but the evidence that it teaches genuine grounding rather than suggestibility is not yet conclusive. read the letter →

arxiv 2608.06128 v1 pith:I627NOJ7 submitted 2026-08-06 cs.AI

Xingyu Guo , Wei Chen , Linlin Yang , Baochang Zhang This is my paper
classification cs.AI
keywords searchagentsreinforcementlearningevidencegroundingretrieval-augmentedgenerationprior-drivenreasoningmulti-turncreditassignmentquestionansweringEALR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main bottleneck in reinforcement-learned search agents is not retrieval quality but reward misalignment: existing methods reward a correct final answer or stepwise progress, so an agent can be trained to confirm its parametric guess with retrieval rather than to be changed by it. To fix this, the authors propose Contextual Information Policy Optimization (CIPO), which adds a dense, turn-level reward called the Evidence-Access Log-Likelihood Ratio (EALR). EALR computes, for each post-retrieval and action, how much more likely that action is when the preceding block is visible than when its attention to that block is masked out. CIPO combines EALR with a terminal outcome reward and optimizes a clipped policy objective, requiring no human process annotations and no reward model. On seven question-answering benchmarks, including three out-of-domain sets, CIPO reports the highest average F1 for both Qwen2.5-3B and Qwen2.5-7B backbones, and its trajectories are less frequently classified by an LLM judge as prior-driven.

What carries the argument

The load-bearing object is EALR (Eq. 5), a token-averaged log-likelihood ratio $$$s^{{\mathrm{ealr}}$}_{i,t} = \frac{1}{m}\log\frac{\pi_{\theta_{\mathrm{old}}}(a_{i,t+1}\mid h_{i,t}, E_{i,t})}{\pi_{\theta_{\mathrm{old}}}(a_{i,t+1}\mid h_{i,t}, M(E_{i,t}))},$$ where $a_{i,t+1}$ is the next turn's <think> and <search> token block, $E_{i,t}$ is the just-retrieved <information> block, and $M(E_{i,t})$ is the same block with its attention to the action masked out. A positive value means the evidence made the observed action more probable; a value near zero means the agent would have taken the same action without ever seeing the evidence. The paper converts each turn's EALR score into a group-normalized discounted advantage, sums it with the terminal outcome reward, and optimizes a clipped objective with a KL penalty. Because both likelihoods are computed by teacher-forcing the frozen rollout policy, EALR requires no extra rollout, no reward model, and no human labels, just one additional masked scoring pass per transition.

What would settle it

Run CIPO on a QA benchmark whose retrieval service returns fluent, plausible passages that contradict the correct answer while still being on-topic. If the trained agent increases its EALR scores and its measured evidence-dependence while its F1 falls relative to an outcome-only baseline, the claim that EALR captures genuine task-relevant grounding is refuted; if it learns to ignore the misleading passages, the grounding assumption holds.

Watch

Extended reading notes

Core claim

CIPO's central claim is that an agent's reasoning steps should be rewarded for being sensitive to retrieved evidence, not merely for converging on the right answer. The paper's contribution is a local, annotation-free measure of that sensitivity, EALR, and a training objective that couples it with a global correctness reward. In CIPO's experiments, this coupling yields macro-average F1 of 0.456 with Qwen2.5-3B and 0.504 with Qwen2.5-7B, beating the best prior reinforcement learning method by 4.7 F1 points at both model scales, while a GPT-5.5-based judge classifies a higher share of correct trajectories as evidence-driven (70.2% versus 59.4% for IGPO). The paper interprets these results as showing that explicitly rewarding evidence grounding discourages confirmation bias and lets retrieved facts revise rather than merely confirm the agent's line of reasoning.

Load-bearing premise

The paper assumes that the likelihood contrast between an action generated with retrieved evidence visible and the same action generated with it masked is a valid, sufficient proxy for whether the agent is genuinely using that evidence in a task-relevant way; if the contrast rewards sensitivity to noisy or misleading passages instead, or misses evidence that acts two or more turns later, the reward signal is misaligned with the behavior CIPO claims to promote.

Editorial extensions

If this is right

  • Training only on four in-domain benchmarks transfers to all three out-of-domain sets: the 7B CIPO model leads on MuSiQue and Bamboogle, and the 3B model leads on all three.
  • EALR alone raises supportive-evidence utilization from roughly 19-22% to 44-49%; adding outcome reward pushes it to 55-61% while cutting irrelevant-evidence utilization from about 29-32% to 9-11%.
  • The measured evidence-driven rate on test trajectories rises from 29.9% to 41.9% overall, and from 59.4% to 70.2% among correctly answered trajectories.
  • The added training cost is modest: per-step wall-clock time increases 9.2% for the 3B model and 4.6% for the 7B model, against relative average-F1 gains of 12.0% and 17.5%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the visible-versus-masked likelihood contrast is a general test of evidence dependence, so the same reward recipe could apply to other tool-use agents (code interpreters, calculators, database queries) where the next action should be changed by the tool's output; the paper only demonstrates it on search-based QA.
  • Editorial: the ablation shows EALR is a sensitivity signal, not a relevance signal, since EALR-only training raises irrelevant-evidence utilization to 31.7% (3B) and 28.9% (7B); the outcome reward is what selects task-relevant dependencies, so in noisy-retrieval settings the EALR term may need an explicit relevance filter.
  • Editorial: EALR credits only the immediately following action, so evidence whose influence appears two or more turns later is never rewarded; a windowed or multi-turn attribution variant is a natural extension the paper has not tested.
  • Editorial: the paper's prior-driven rate is measured by an LLM judge rather than by direct counterfactuals; a cheaper behavioral check would delete the retrieved passage and see whether the final answer changes, and then compare that rate with EALR scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Contextual Information Policy Optimization (CIPO), an RL framework for training search agents to use retrieved evidence in their reasoning. For each post-retrieval transition, CIPO computes an Evidence-Access Log-Likelihood Ratio (EALR, Eq. 5) by teacher-forcing the observed next-turn <think>+<search> action under evidence-visible and evidence-masked conditions, forming a dense turn-level reward. This is normalized within a rollout group, combined with a terminal outcome reward, and optimized with a clipped GRPO-style objective plus a KL penalty. The evaluation covers seven QA benchmarks (four in-domain, three out-of-domain) with Qwen2.5-3B/7B backbones, comparing against outcome-only and process-supervised RL baselines, plus ablations, counterfactual masking experiments, training dynamics, and efficiency analyses. The central claims are that CIPO achieves the highest average F1 on most benchmarks and reduces prior-driven reasoning as measured by an LLM judge.

Significance. If the empirical claims hold, CIPO is an appealingly simple and annotation-free dense credit signal that could improve evidence grounding in agentic search RL. The derivation of EALR is clean and requires no additional reward model; the code is released; the evaluation spans seven benchmarks and two model sizes; and the counterfactual masking and target-scope analyses are thoughtful diagnostics. However, the central empirical claims currently rest on single runs without variance estimates, and the behavioral claim about prior-driven reasoning relies on an LLM judge whose criteria may conflate evidence-suggestibility with genuine evidence use. The missing corruption/suggestibility test is the largest gap, as it directly determines whether EALR rewards task-relevant grounding or mere sensitivity to retrieved text.

major comments (4)
  1. [Section 3.1, Eq. (5)] EALR rewards any increase in next-turn action likelihood when the preceding <information> block is visible relative to masked, which is a measure of sensitivity to retrieved text, not of task-relevant grounding. The paper's own Table 9 shows that a variant masking only non-supportive evidence yields negative Evidence Selectivity (-20.5) and the lowest average F1 (0.345), confirming that pure sensitivity to unhelpful content is harmful. The full CIPO objective combines EALR with outcome reward, but no experiment deliberately corrupts, swaps, or contradicts the retrieved passages to test whether the trained policy becomes suggestible (i.e., lets misleading evidence redirect its answer) or remains robustly selective. I ask the authors to add such counterfactual/corruption tests, for example by injecting false or contradictory passages at inference and measuring answer-change rates, or to explicitly weaken the mechanism claim.
  2. [Section 4.4 and Appendix F] The behavioral claim that CIPO reduces prior-driven reasoning rests entirely on GPT-5.5 classifications whose stated criterion for 'evidence-driven' is that retrieved evidence 'substantially influences' intermediate reasoning or the final answer. By that criterion, a trajectory that blindly follows misleading evidence would also be labeled evidence-driven, so the measured reduction in prior-driven rate is not evidence that the policy uses evidence selectively and correctly. No validation of the judge against human labels, no inter-annotator agreement statistics, and no robustness analyses are reported. Since this metric is load-bearing for the central claim, please provide a human-validated subset with agreement statistics, and/or split the metric into (a) whether evidence influenced the trajectory and (b) whether that influence led to a correct answer.
  3. [Tables 1-3, Figures 3-6] All performance comparisons appear to be single runs with no error bars, confidence intervals, or significance tests. The headline claims of consistency (e.g., 'CIPO consistently achieves the highest macro-average F1' and the 4.7-point margins over the strongest baseline) assert reliability without variance estimates, which is particularly risky given the modest margins on several individual benchmarks and the stochasticity of RL training. Please report multiple seeds with means and variances, and ideally significance tests for the aggregate gains, or explicitly state that the reported numbers are from one run.
  4. [Eq. (5) and Appendix D] EALR scores only the immediate next-turn <think>+<search> action against the most recent information block, so evidence whose influence first appears two or more turns later receives no direct dense credit. The Appendix D comparison against the full-trajectory variant does not isolate this next-turn-only limitation, because the two variants differ on many dimensions simultaneously; a control that varies only the scoring window (one, two, or three future turns) would be more informative. As written, the claim that CIPO assigns dense, turn-level credit to evidence use is overstated for delayed evidence effects.
minor comments (5)
  1. [Figure 1] The figure contains 'V on Braun' and 'supervises NASA program' in the illustrative trajectories; 'V on Braun' should be 'von Braun' to match standard spelling.
  2. [Table 1 header] The header 'Qwen2.5-7B-Base/Instruct' and 'Qwen2.5-3B-Base/Instruct' is ambiguous; Section 4.1 states that all methods use the Instruct variants, so the header should be aligned with the text.
  3. [Appendix C, prompt template] The prompt template includes the literal strings '* Today is {today} *' and 'deep research' boilerplate; please verify that the placeholder is instantiated and that the template shown matches the released code.
  4. [Section 4.7] The G/C ratio is computed from relative F1 gains and relative time increases; reporting absolute F1 deltas alongside relative percentages would make the efficiency comparison easier to interpret.
  5. [References] The reference for the training framework (Sheng et al. 2025) is titled 'Hybridflow', which appears inconsistent with the name 'verl' used in the text; please verify the citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CIPO's reward is not fitted to benchmark answers; F1 gains are measured on external datasets, and the behavioral evaluation uses an independent LLM judge whose rubric is aligned with, but not identical to, the EALR objective.

full rationale

The claimed derivation chain is self-contained. EALR (Eq. 5) is a reward signal computed from the frozen rollout policy's likelihood contrast under evidence-visible versus attention-masked conditions; it is not a predictor fitted to the benchmark answers, and the terminal outcome reward is combined with it rather than derived from it. The central F1 claim (Table 1) is evaluated on seven external QA benchmarks using word-level F1, which is independent of the training objective. The behavioral claim that CIPO reduces prior-driven reasoning is measured by an external LLM judge (GPT-5.5) with a rubric requiring retrieved evidence to substantially influence reasoning; although this rubric is conceptually aligned with EALR, the judge is not built from Eq. (5), and the judged property is broader (whole-trajectory, semantic influence) than the next-turn likelihood contrast that EALR scores. Optimizing EALR therefore does not logically force a particular judge classification, so the evaluation provides independent evidence rather than a restatement of the reward. The paper's own Table 9 counterfactual masking experiments and its admitted limitation that EALR alone increases irrelevant-evidence utilization further show that the authors tested the alternative interpretation; that concern is a construct-validity or robustness issue, not circularity. No load-bearing self-citation, imported uniqueness theorem, or fitted-input-called-prediction step appears in the derivation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces no physical or conceptual entities. The main free parameters are standard RL hyperparameters. The key assumptions are the validity of the attention-masked likelihood contrast as a measure of evidence dependence, and the reliability of the LLM-judge-based behavioral evaluation.

free parameters (5)
  • discount_factor_gamma = 1.0
    Set to 1.0 for full-trajectory credit; a design choice that affects how future rewards propagate.
  • kl_coefficient_beta = 0.001
    Standard KL penalty coefficient to limit policy drift.
  • group_size_G = 16
    Number of rollouts per prompt for group-relative normalization.
  • max_turns = 5
    Maximum interaction turns per rollout.
  • rollout_temperature = 1.0
    Sampling temperature for rollouts.
assumptions (5)
  • standard math Attention masking with -inf logits yields a valid autoregressive conditional distribution for the masked condition.
    Used to compute the masked log-likelihood in Eq. (4).
  • domain assumption The trajectory format with distinct <think>, <search>, and <information> blocks, where information tokens are environment-generated, holds for training and evaluation.
    Required for the EALR definition in Section 3.1.
  • domain assumption Increasing EALR (evidence sensitivity) is a valid proxy for desirable evidence use that improves or preserves task performance when combined with outcome supervision.
    Core premise of the reward design; tested empirically through ablations and counterfactual masking.
  • domain assumption The LLM-based judge and evidence annotation procedure in Appendix F provide a reliable measure of prior-driven vs evidence-driven reasoning.
    Used for the behavioral analysis in Section 4.4; no inter-annotator agreement reported.
  • standard math The expectation of the log-likelihood ratio under the evidence-visible policy equals the KL divergence in Eq. (7).
    Motivates EALR as a policy-level evidence-access signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contextual Information Policy Optimization for Search Agents." pith.science (2026). https://pith.science/paper/I627NOJ7

@misc{pith2026260806128,
  author       = {Pith},
  title        = {Pith review of: Contextual Information Policy Optimization for Search Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I627NOJ7}},
  note         = {Machine review of arXiv:2608.06128}
}
read the original abstract

Search agents extend large language models beyond static parametric memory by enabling them to acquire and use ex ternal evidence during multi-step reasoning. For knowledge intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant ev idence but also on using it to guide subsequent reasoning. However, existing methods primarily reward final-answer cor rectness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence. This misalignment encourages prior-driven reason ing: agents form conclusions based on internal knowledge and use retrieval mainly to confirm them, resulting in confirma tion bias and inefficient evidenceuse.Toaddressthisissue, we propose Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework that explicitly aligns policy optimization with external evidence use. CIPO assigns dense, turn-level credit to reasoning ac tions influenced by retrieved information, while combining this evidence-use signal with a global outcome reward to pre serveanswercorrectness.Withthismanner,CIPOdiscourages evidence-detached guesses and promotes reasoning trajecto ries in which retrieved facts can guide or revise subsequent reasoning. Importantly, CIPO requires neither human process annotations nor an additional reward model. Extensive exper iments on seven in-domain and out-of-domain benchmarks show that CIPO reduces the prevalence of prior-driven rea soning and achieves excellent performance on most tasks.

Figures

Figures reproduced from arXiv: 2608.06128 by the authors.

Figure 1
Figure 1. Illustrative case for the same question. Prior-driven [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of CIPO. It evaluates each post-retrieval reasoning step under evidence-visible and evidence [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Representative step-wise validation-set F1 trajectories on two in-domain and two out-of-domain benchmarks. Each [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prior-driven rates during training. CIPO maintains [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Evidence-driven rates on the test sets across seven [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Additional step-wise validation-set F1 trajectories for the benchmarks not shown in the main paper. Each panel [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 35 canonical work pages

  1. [1]

    Ahmadian, A.; Cremer, C.; Gall \'e , M.; Fadaee, M.; Kreutzer, J.; Pietquin, O.; \"U st \"u n, A.; and Hooker, S. 2024. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12248--12267

    2024

  2. [2]

    Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, volume 2024, 9112--9141

    2024

  3. [4]

    Dai, Y.; Yang, S.; Wang, G.; Deng, Y.; Zhang, Z.; Yin, J.; Zeng, P.; Ying, Z.; Meng, C.; Yi, C.; et al. 2026. Careful queries, credible results: Teaching rag models advanced web search tools with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence

    work page 2026

  4. [6]

    Feng, L.; Xue, Z.; Liu, T.; and An, B. 2026. Group-in-group policy optimization for llm agent training. NeurIPS

    work page 2026

  5. [9]

    Goel, H.; Udathu, A.; Jabbireddy, S.; Kalkar, P.; and Parulekar, A. 2026. S ^3 -R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data. arXiv preprint arXiv:2605.01248

    work page Pith review arXiv 2026

  6. [10]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081): 633--638

    2025

  7. [11]

    He, B.; Hu, M.; Xu, Z.; Wang, H.; Zong, L.; Chen, Y.; Ma, C.; Liu, X.; Zhou, P.; and King, I. 2026. Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration. ICML

    work page 2026

  8. [12]

    D.; Sugawara, S.; and Aizawa, A

    Ho, X.; Nguyen, A.-K. D.; Sugawara, S.; and Aizawa, A. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, 6609--6625

    work page 2020

Show all 82 references
  1. [15]

    S.; and Zettlemoyer, L

    Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601--1611

    2017

  2. [16]

    Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics

    2019

  3. [17]

    u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \

    Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; K \"u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \"a schel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS

    2020

  4. [19]

    Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; and Hajishirzi, H. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL

    2023

  5. [20]

    S.; et al

    Ning, L.; Liang, Z.; Jiang, Z.; Qu, H.; Ding, Y.; Fan, W.; Wei, X.-y.; Lin, S.; Liu, H.; Yu, P. S.; et al. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In Proceedings of the 31st ACM SIGKDD Conference on Knowle...

    2025

  6. [21]

    A.; and Lewis, M

    Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, 5687--5711

    2023

  7. [22]

    L.; Lai, H.; Sun, X.; Sun, J.; Yang, X.; Yang, Y.; Yao, S.; Xu, W.; et al

    Qi, Z.; Liu, X.; Iong, I. L.; Lai, H.; Sun, X.; Sun, J.; Yang, X.; Yang, Y.; Yao, S.; Xu, W.; et al. 2025. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, volume 2025, 79791--79821

    2025

  8. [23]

    Qu, C.; Dai, S.; Wei, X.; Cai, H.; Wang, S.; Yin, D.; Xu, J.; and Wen, J.-R. 2025. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8): 198343

    2025

  9. [26]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, 1279--1297

    2025

  10. [28]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10: 539--554

    2022

  11. [29]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long p...

    2023

  12. [31]

    Wang, G.; Dai, S.; Ye, G.; Gan, Z.; Yao, W.; Deng, Y.; Wu, X.; and Ying, Z. 2026 b . Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents. ICLR

    2026

  13. [32]

    Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024 a . Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2024

  14. [33]

    Wang, X.; Wang, Z.; Gao, X.; Zhang, F.; Wu, Y.; Xu, Z.; Shi, T.; Wang, Z.; Li, S.; Qian, Q.; et al. 2024 b . Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 17716--17736

    2024

  15. [36]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS

    2022

  16. [38]

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, 2369--2380

    2018

  17. [39]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2022. React: Synergizing reasoning and acting in language models. ICLR

    2022

  18. [40]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38: 113222--113244

    2026

  19. [43]

    Zhang, W.; Li, X.; Dong, K.; Wang, Y.; Jia, P.; Li, X.; Zhang, Y.; Xu, D.; Du, Z.; Guo, H.; et al. 2026. Process vs. outcome reward: Which is better for agentic rag reinforcement learning. Advances in Neural Information Processing Systems, 38: 58701--58729

    2026

  20. [45]

    Zheng, Y.; Fu, D.; Hu, X.; Cai, X.; Ye, L.; Lu, P.; and Liu, P. 2025. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 414--431

    2025

  21. [46]

    Zheng, Y.; Sun, S.; Qiu, L.; Ru, D.; Jiayang, C.; Li, X.; Lin, J.; Wang, B.; Luo, Y.; Pan, R.; et al. 2024. OpenResearcher: unleashing AI for accelerated scientific research. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demo...

    2024

  22. [47]

    NeurIPS , year=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. NeurIPS , year=

  23. [48]

    arXiv preprint arXiv:2604.17555 , year=

    CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search , author=. arXiv preprint arXiv:2604.17555 , year=

    arXiv

  24. [49]

    Goel, Harsh and Udathu, Akhil and Jabbireddy, Susmija and Kalkar, Pradnesh and Parulekar, Atharva , journal=

  25. [50]

    arXiv preprint arXiv:2606.16409 , year=

    PathRouter: Aligning Rewards with Retrieval Quality in Agentic Graph Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2606.16409 , year=

  26. [51]

    ICLR , year=

    React: Synergizing reasoning and acting in language models , author=. ICLR , year=

  27. [52]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  28. [53]

    International conference on learning representations , volume=

    Self-rag: Learning to retrieve, generate, and critique through self-reflection , author=. International conference on learning representations , volume=

  29. [54]

    arXiv preprint arXiv:2503.09516 , year=

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning , author=. arXiv preprint arXiv:2503.09516 , year=

    arXiv

  30. [55]

    ICML , year=

    Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration , author=. ICML , year=

  31. [56]

    ICLR , year=

    Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents , author=. ICLR , year=

  32. [57]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  33. [58]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  34. [59]

    NeurIPS , year=

    Chain-of-thought prompting elicits reasoning in large language models , author=. NeurIPS , year=

  35. [60]

    EMNLP , year=

    Search-o1: Agentic search-enhanced large reasoning models , author=. EMNLP , year=

  36. [61]

    arXiv preprint arXiv:2503.05592 , year=

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2503.05592 , year=

    arXiv

  37. [62]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  38. [63]

    arXiv preprint arXiv:2505.15107 , year=

    Stepsearch: Igniting llms search ability via step-wise proximal policy optimization , author=. arXiv preprint arXiv:2505.15107 , year=

    arXiv

  39. [64]

    outcome reward: Which is better for agentic rag reinforcement learning , author=

    Process vs. outcome reward: Which is better for agentic rag reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  40. [65]

    NeurIPS , year=

    Group-in-group policy optimization for llm agent training , author=. NeurIPS , year=

  41. [66]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  42. [67]

    Buy 4 reinforce samples, get a baseline for free! , author=

  43. [68]

    arXiv e-prints , pages=

    Reinforce++: A simple and efficient approach for aligning large language models , author=. arXiv e-prints , pages=

  44. [69]

    arXiv preprint arXiv:2507.18071 , year=

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    arXiv

  45. [70]

    Transactions of the Association for Computational Linguistics , year=

    Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , year=

  46. [71]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  47. [72]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  48. [73]

    Proceedings of the 28th International Conference on Computational Linguistics , pages=

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps , author=. Proceedings of the 28th International Conference on Computational Linguistics , pages=

  49. [74]

    Transactions of the Association for Computational Linguistics , volume=

    MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  50. [75]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Measuring and narrowing the compositionality gap in language models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  51. [76]

    ACL , year=

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories , author=. ACL , year=

  52. [77]

    arXiv preprint arXiv:2412.15115 , year =

    Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =

    arXiv

  53. [78]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  54. [79]

    arXiv preprint arXiv:2312.10997 , year=

    Retrieval-augmented generation for large language models: A survey , author=. arXiv preprint arXiv:2312.10997 , year=

    arXiv

  55. [80]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Searching for best practices in retrieval-augmented generation , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  56. [81]

    arXiv preprint arXiv:2509.00877 , year=

    EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes , author=. arXiv preprint arXiv:2509.00877 , year=

  57. [82]

    arXiv preprint arXiv:2411.00744 , year=

    Corag: A cost-constrained retrieval optimization system for retrieval-augmented generation , author=. arXiv preprint arXiv:2411.00744 , year=

  58. [83]

    arXiv preprint arXiv:2503.00223 , year=

    Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning , author=. arXiv preprint arXiv:2503.00223 , year=

    arXiv

  59. [84]

    Advances in Neural Information Processing Systems , year=

    Learning to reason with search for llms via reinforcement learning , author=. Advances in Neural Information Processing Systems , year=

  60. [85]

    arXiv preprint arXiv:2505.17005 , year=

    R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning , author=. arXiv preprint arXiv:2505.17005 , year=

    arXiv

  61. [86]

    International Conference on Learning Representations , volume=

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning , author=. International Conference on Learning Representations , volume=

  62. [87]

    arXiv preprint arXiv:2503.23383 , year=

    Torl: Scaling tool-integrated rl , author=. arXiv preprint arXiv:2503.23383 , year=

    arXiv

  63. [88]

    Proceedings of the AAAI Conference on Artificial Intelligence , year=

    Careful queries, credible results: Teaching rag models advanced web search tools with reinforcement learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , year=

  64. [89]

    arXiv preprint arXiv:2508.12800 , year=

    Atom-searcher: Enhancing agentic deep research via fine-grained atomic thought reward , author=. arXiv preprint arXiv:2508.12800 , year=

    arXiv

  65. [90]

    arXiv preprint arXiv:2506.18959 , year=

    From web search towards agentic deep research: Incentivizing search with reasoning agents , author=. arXiv preprint arXiv:2506.18959 , year=

    arXiv

  66. [91]

    arXiv preprint arXiv:2506.18096 , year=

    Deep research agents: A systematic examination and roadmap , author=. arXiv preprint arXiv:2506.18096 , year=

    arXiv

  67. [92]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    A survey of webagents: Towards next-generation ai agents for web automation with large foundation models , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=

  68. [93]

    Frontiers of Computer Science , volume=

    Tool learning with large language models: A survey , author=. Frontiers of Computer Science , volume=. 2025 , publisher=

    2025

  69. [94]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

    OpenResearcher: unleashing AI for accelerated scientific research , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

    2024

  70. [95]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=

    2025

  71. [96]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  72. [97]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  73. [98]

    arXiv preprint arXiv:2509.02547 , year=

    The landscape of agentic reinforcement learning for llms: A survey , author=. arXiv preprint arXiv:2509.02547 , year=

    arXiv

  74. [99]

    arXiv preprint arXiv:2501.15602 , year=

    Rethinking external slow-thinking: From snowball errors to probability of correct reasoning , author=. arXiv preprint arXiv:2501.15602 , year=

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.