Pith. sign in

REVIEW 4 major objections 4 minor 10 cited by

An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes practical design rules for RL-trained LLM search agents: format rewards help, intermediate retrieval rewards do not, and search engine quality shapes both training and inference robustness.

desk verdict Useful systematic RL-agent comparison, but the headline format-reward claim is contradicted by the paper's own Table 1 and by eval-tuned weights. read the letter →

arxiv 2505.15117 v1 pith:XNHZSR66 submitted 2025-05-21 cs.CL cs.AIcs.IR

Bowen Jin , Jinsung Yoon , Priyanka Kargupta , Sercan O. Arik , Jiawei Han This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords reinforcementlearningLLMagentsreasoning-searchinterleavingrewardshapingformatretrievalsearchenginechoicePPOandGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a systematic empirical study of how to train an LLM that interleaves reasoning with live search-engine calls using reinforcement learning. It claims three things: adding a format reward (a reward for emitting correctly tagged reasoning, search, information, and answer blocks) reliably improves final accuracy, especially when starting from a base rather than instruction-tuned model; intermediate rewards for retrieving relevant passages do not help, and can hurt; and the quality of the search engine used during training changes both the training dynamics and how robustly the agent behaves at inference. The practical upshot, if the paper is right, is a concrete recipe for building search agents: reward the action format, do not reward intermediate retrievals, prefer general-purpose backbones, and spend the budget on a strong retriever. This matters because search agents are a path to grounding LLM answers in up-to-date external information, and the paper's tables translate into choices a practitioner can make without extra annotation.

What carries the argument

The central object is the reasoning-search interleaved LLM agent: a policy $\pi_\theta$ that alternates between <think> reasoning steps and <search> queries, receives retrieved passages inside <information> blocks, and finally emits <answer>. The training mechanism is RL (PPO or GRPO) with a rule-based reward $r_\phi(x,y)$ that is composed of an outcome term (exact match of the final answer), an optional format term $\lambda_f$ for whether the rollout obeys the required tag structure, and an optional intermediate retrieval term $\lambda_r$ for whether the gold answer appears in retrieved passages. The format term works by giving gradient signal for staying in the action-observation structure the search engine requires, which is precisely the skill base LLMs lack; the retrieval term is the candidate that must be tested, and the paper finds it unnecessary. The third moving part is the search engine $R$ itself, which enters the rollout distribution $\pi_\theta(\cdot|x;R)$ and therefore shapes what trajectories the policy sees; higher-quality engines (dense exact, then dense approximate, then BM25, then random) make training stable and search calls strategic rather than avoidant or compensatory.

What would settle it

Train the same RL protocol on a domain where the gold answer rarely appears verbatim in retrieved passages (for example, synthesis-style answers or code retrieval) and compute the intermediate retrieval reward with a learned relevance judge instead of substring match; if the retrieval-reward variant consistently beats outcome-plus-format, the paper's claim that intermediate retrieval rewards are unnecessary is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the outcome reward is the main learning signal and everything else should support it, not compete with it. Under this view, a format reward helps because it removes a failure mode—the base LLM cannot reliably emit the <think>/<search>/<information>/<answer> structure needed to trigger the search engine—so the model explores useful trajectories instead of dying in format errors. Intermediate retrieval rewards, computed by substring exact match between the gold answer and retrieved passages, add nothing because a query that retrieves the right passage is already rewarded indirectly through the final outcome; the extra term constrains the trajectory and degrades performance as its weight grows. The paper also claims that general-purpose LLMs are better RL starting points than reasoning-specialized ones (the latter follow instructions poorly early in training, especially under GRPO, where training collapses), that scaling from 3B to 32B improves performance with diminishing returns, and that the retriever used during training matters: random results teach the agent to stop searching, BM25 teaches it to search more and less efficiently, and a strong dense retriever produces stable, judicious search behavior. At inference time, the trained agent is robust to being paired with a different retriever, and a stronger inference-time retriever consistently improves answers.

Load-bearing premise

The load-bearing premise is that a benchmark built on two training sets (NQ and HotpotQA), one corpus (Wikipedia-18), exact-match short answers, and evaluation-tuned reward weights speaks for search agents in other domains, answer formats, and corpora; if that transfer fails, the headline guidelines would not generalize even though the measurements remain valid.

Editorial extensions

If this is right

  • Practitioners should add a format reward when RL-training a search agent, with a small weight ($\lambda_f$ around 0.2–0.4) and a base LLM; the paper measures consistent gains over outcome-only RL, especially on multi-hop QA.
  • Intermediate retrieval rewards can be omitted; the outcome reward already encourages effective queries, and larger $\lambda_r$ lowers final accuracy in the paper's measurements.
  • General-purpose base LLMs are safer RL starting points than reasoning-specialized ones; the reasoning-specialized model collapses under GRPO in these experiments.
  • Scaling the backbone from 3B to 32B improves final accuracy, but with diminishing returns; the paper attributes the residual gains to better retrieval use rather than parametric knowledge.
  • The agent trained with one retriever transfers to another at inference time, and swapping in a stronger retriever (including an online general search engine) improves accuracy on hard benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The format-reward finding suggests a general recipe for RL over discrete action languages: rewards for syntactic validity are most valuable exactly when the base policy is least instruction-tuned, so the optimal $\lambda_f$ should shrink as backbone instruction-following improves.
  • The failure of substring-EM retrieval rewards may be an artifact of the metric, not of retrieval rewards in general; a learned relevance model or a reward based on downstream utility could behave differently, which is a testable extension the paper does not run.
  • The search-engine training dynamics imply a curriculum: starting RL with a strong retriever and later weakening it might produce agents that search more robustly, but the paper does not test this.
  • Because the agent trained on random noise learns to ignore retrieval entirely, RL is effectively doing credit assignment over tool reliability; this suggests the same framework could predict when agents should abandon any tool whose outputs are uninformative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents an empirical study of three design choices for RL training of LLM-based search agents: reward formulation (outcome, format, and intermediate retrieval rewards), the backbone LLM (general-purpose vs. reasoning-specialized, and scale 3B–32B), and the search engine used during training and inference. The authors report that format rewards improve final performance, that intermediate retrieval rewards are of limited use, that general-purpose backbones are more stable and effective than reasoning-specialized ones, that larger models help with diminishing returns, and that stronger search engines lead to more stable training and better inference-time performance. The experiments use NQ and HotpotQA for training, evaluate on several short-form QA datasets, and include additional studies of long-form QA and dataset scaling in the appendices. Code is released.

Significance. If its claims hold, the paper provides actionable guidance for practitioners building RL-trained search agents: add format rewards, avoid intermediate retrieval rewards, prefer general-purpose base LLMs, and invest in strong retrievers. The study covers a plausible design space with a consistent training setup, evaluates multiple LLM sizes and RL algorithms, and includes useful auxiliary analyses in the appendices. The released code is a concrete asset for reproducibility. However, the strength of the headline claims, especially the format-reward result, exceeds what the reported tables support, and the absence of variance estimates and the evaluation-based selection of reward weights weaken the generality of the recommendations.

major comments (4)
  1. [§4.1, Table 1] The claim in Section 4.1 that 'Adding a format reward consistently improves final model performance' is not supported by Table 1. Across the 12 size/initialization/algorithm configurations, the seven-dataset average decreases in four cases: 3B GRPO Instruct (0.336 to 0.319), 7B PPO Instruct (0.385 to 0.370), 7B GRPO Instruct (0.396 to 0.378), and 14B PPO Base (0.479 to 0.459). Several of the positive cases are also small relative to the likely run-to-run variation, and no seeds or error bars are reported. I recommend restating the result as a conditional or backbone-dependent effect and reporting the per-configuration deltas explicitly.
  2. [Appendix G.1, §4.1] The format-reward comparison is confounded by tuning. Appendix G.1 states that λ_f was chosen by grid search over {0.2, 0.4, 0.6, 0.8} by selecting 'the best-performing value for each model,' with no evidence of a held-out validation split, while the outcome-only baselines receive no such tuned hyperparameter. This biases the comparison in favor of the format-reward condition. Please report performance across all λ_f values and all configurations, and either use a validation split or clearly state that the reported numbers include selection on the test sets.
  3. [§5.1, Table 3; §6.1, Table 4] Several claims about training 'stability' and 'collapse' are based on single runs without variance estimates. For example, Section 5.1 states that GRPO training on DeepSeek-R1-Distill-Qwen-7B leads to 'training collapse,' and Table 3 reports a 0.100 average for that configuration, but no repeated runs or confidence intervals are given. Given the prominent role of this observation, I ask that the authors provide at least two or three seeds for the headline comparisons or explicitly limit the claim to the observed run and note the lack of variance reporting.
  4. [Abstract and §7] The abstract states that 'format rewards are effective in improving final performance,' while the conclusion already qualifies this as 'an important role in certain scenarios.' These statements are in tension. The conclusion's wording is more consistent with the data; I recommend aligning the abstract and Section 4.1 with that more cautious formulation, and also specifying the settings in which format rewards help (e.g., base LLMs with PPO/GRPO 3B and 7B base variants) versus the settings where they do not.
minor comments (4)
  1. [§6.1, Figure 3] The text refers to 'Figure 3(c)' for training reward curves and then to 'Figure 3(d)' for search-call frequency, but the figure has only three panels: (a) retrieval quality, (b) training reward, and (c) number of search calls. Please correct the panel references.
  2. [Appendices G.2–G.6] Several appendix sections state that 'For the results reported in Table 1, we use either the final checkpoint...' even when the section describes experiments whose results appear in Tables 2–5. This copy-paste error should be fixed so each section refers to its own table.
  3. [Throughout] Model names are inconsistently capitalized: 'Deepseek' appears in Section 5.1, Figure 2, and Figure 5 captions, while 'DeepSeek' is used in the table caption of Table 3. Please standardize.
  4. [Appendix H] The case studies are useful, but the BM25 example in Table 10 shows the model outputting 'Fail to answer' as a final answer; it would help to state explicitly whether such outputs are counted as incorrect by the exact-match evaluator and whether the training trajectory contains similar non-answer responses that would receive zero outcome reward yet still shape the policy.

Circularity Check

1 steps flagged · score 2.0 of 10

One minor self-definitional step inflates the training-reward convergence claim; final-performance findings are independent measurements.

  1. self definitional [Section 4.1, Eq. (3), Figure 1(a) caption]
    "Figure 1(a) caption: "Training reward curves with varying format reward scaling factors (λf); larger λf values lead to faster convergence." Results: "Format reward accelerates RL convergence; larger λf values lead to faster convergence by explicitly guiding the model to issue correctly formatted search queries and interpret results effectively.""

    In Eq. (3), the reward is r = λf whenever the answer is wrong but the format is correct, and r = 0 when both are wrong. The training-reward curve in Figure 1(a) is therefore mechanically inflated by λf for every correctly formatted but wrong rollout, which dominates early training. Comparing curves at different λf on the very objective that contains λf means the observed "faster convergence" is partly a restatement of the reward definition rather than independent evidence about task learning. The final EM results in Table 1 are a separate, non-circular measurement; only the convergence claim is affected.

full rationale

There is no load-bearing circularity in the paper's central empirical comparisons. The paper reports direct experiments rather than deriving predictions from fitted values, and the final-performance tables (Tables 1-5) measure exact match and other downstream metrics that are independent of the reward hyperparameters being varied. The dependence on Search-R1 [13], authored by the same first author, supplies the training recipe, datasets, RL objective, and baseline; this is dependence, not circularity, because the paper's conclusions are comparisons against that baseline rather than consequences of citing it. The grid search over λf on evaluation metrics (Appendix G.1) is a methodological risk, but it is not circular in the algebraic sense: Table 1 contains multiple configurations where format reward decreases average performance, so the "consistent improvement" headline is not statistically forced by the fitting procedure. The only genuine self-definitional step is the claim that larger λf accelerates convergence, which is read off training-reward curves that include λf by construction. Because this affects a supporting claim rather than the core final-performance findings, the overall circularity score is low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The study is empirical, so the ledger contains no invented entities. The main free parameters are the reward weights lambda_f and lambda_r, both tuned on evaluation performance. The domain assumptions concern whether exact-match rewards, the NQ/HotpotQA training distribution, the Wikipedia-18 corpus, and checkpoint selection support generalizable guidelines.

free parameters (2)
  • lambda_f (format reward weight) = 0.2 for Qwen2.5-3B and 14B, 0.4 for Qwen2.5-7B
    Selected by grid search over {0.2, 0.4, 0.6, 0.8} based on best final evaluation performance (Appendix G.1). The headline that format rewards help is conditional on this selection.
  • lambda_r (intermediate retrieval reward weight) = 0.1 for 3B and 7B
    Selected by grid search over {0.1, 0.3, 0.5} based on best final performance (Appendix G.2). Only positive values were tested, so the conclusion that intermediate retrieval rewards are unnecessary is inferred from monotone degradation with larger values.
assumptions (4)
  • domain assumption Exact string match and substring match are adequate proxies for answer correctness and retrieval relevance.
    The reward functions in Equations 2-4 define the training signal; the conclusions about reward design depend on these proxies holding for the tested factoid QA setting.
  • domain assumption Results on the seven selected QA benchmarks using the Wikipedia-18 corpus generalize to real-world search-agent deployment.
    The paper frames its findings as general guidelines in the Abstract and Conclusion, but all main experiments use one training distribution (NQ and HotpotQA) and one corpus.
  • ad hoc to paper Reward weights and checkpoints selected using evaluation performance do not invalidate cross-setting comparisons.
    Appendix G.1 and G.2 select lambda_f and lambda_r by grid search on evaluation performance and choose final checkpoints based on reward curves, which assumes no overfitting to the benchmark suite.
  • domain assumption The PPO and GRPO implementations and hyperparameter choices are comparable across model types and sizes.
    Sections 5.1 and 5.2 compare Qwen base models with DeepSeek distilled models under identical training settings, assuming that any remaining implementation differences do not drive the observed gaps.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents." pith.science (2026). https://pith.science/paper/XNHZSR66

@misc{pith2026250515117,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNHZSR66}},
  note         = {Machine review of arXiv:2505.15117}
}
read the original abstract

Reinforcement learning (RL) has demonstrated strong potential in training large language models (LLMs) capable of complex reasoning for real-world problem solving. More recently, RL has been leveraged to create sophisticated LLM-based search agents that adeptly combine reasoning with search engine use. While the use of RL for training search agents is promising, the optimal design of such agents remains not fully understood. In particular, key factors -- such as (1) reward formulation, (2) the choice and characteristics of the underlying LLM, and (3) the role of the search engine in the RL process -- require further investigation. In this work, we conduct comprehensive empirical studies to systematically investigate these and offer actionable insights. We highlight several key findings: format rewards are effective in improving final performance, whereas intermediate retrieval rewards have limited impact; the scale and initialization of the LLM (general-purpose vs. reasoning-specialized) significantly influence RL outcomes; and the choice of search engine plays a critical role in shaping RL training dynamics and the robustness of the trained agent during inference. These establish important guidelines for successfully building and deploying LLM-based search agents in real-world applications. Code is available at https://github.com/PeterGriffinJin/Search-R1.

Figures

Figures reproduced from arXiv: 2505.15117 by the authors.

Figure 1
Figure 1. Empirical analyses on format reward and intermediate retrieval reward. (a) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The study of underlying pretrained LLM for development of LLM-based search agents [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Effect of Search Engine Choice on RL Training Dynamics. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Data scaling effects in RL training for search agents. (a) [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: The study of underlying pretrained LLM for development of search agents with RL. (a) [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TCPO: Turn-Level Credit Policy Optimization

    cs.AI 2026-08 conditional novelty 6.0 of 10

    TCPO improves multi-turn LLM RL by converting verifier scores into turn-level credit via three reference-based comparisons, beating baseline methods on math, code, and agent tasks.

  2. TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework

    cs.IR 2025-11 conditional novelty 6.0 of 10

    TeaRAG shows that hybrid chunk+triplet retrieval with Personalized PageRank and an iterative process-aware DPO reward keeps QA accuracy while cutting reasoning tokens by roughly 60%.

  3. Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search

    cs.CL 2025-10 conditional novelty 6.0 of 10

    SLIM separates search and browse tools and summarizes trajectories every 50 turns, beating several open-source agentic search systems on BrowseComp and HLE with fewer tool calls and lower cost.

  4. Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation

    cs.CL 2025-10 reject novelty 6.0 of 10

    A turn-level faithfulness reward improves a Search-R1-style agent's Information-Think and Think-Answer faithfulness as judged by the same reward model used for training, while task accuracy is roughly unchanged.

  5. SSRL: Self-Search Reinforcement Learning

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    SSRL, a training pipeline that uses an LLM's own repeated sampling as a search environment for RL, improves question answering without external tools and transfers to real search engines.

  6. BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

    cs.CL 2025-08 conditional novelty 6.0 of 10

    BrowseComp-Plus is a fixed, human-verified document corpus with supporting and negative documents that allows controlled, reproducible evaluation of deep-research agents and their retrievers.

  7. MetaAgent: Toward Self-Evolving Agent via Tool Meta-Learning

    cs.AI 2025-08 reject novelty 6.0 of 10

    The paper advertises a self-evolving tool-using agent that beats baselines, but the body contains a different math paper, leaving the performance claims unsupported.

  8. TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Turn-level rewards from TD changes in a frozen reference model's gold-answer log-probability improve long-horizon search-agent RL on closed- and open-web benchmarks.

  9. ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

    cs.AI 2025-12 reject novelty 5.0 of 10

    LLM reasoning benchmark scores vary substantially across repeated runs under the same model, strategy, and task, so single-run evaluation can misrank systems.

  10. A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A research agenda calling for geo-temporal reasoning in deep research systems, with no experiments or system implementation.

Reference graph

Works this paper leans on

59 extracted references · 20 canonical work pages · cited by 10 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 3

    arXiv 2023

  2. [2]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. 2

    arXiv 2024

  3. [3]

    Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023. 1, 3

    work page 2023

  4. [4]

    Research: Learning to reason with search for llms via reinforcement learning

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Research: Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025. 2, 3

    arXiv 2025

  5. [5]

    Eli5: Long form question answering

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190, 2019. 16

    arXiv 1907

  6. [6]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025. 2

    arXiv 2025

  7. [7]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2, 2023. 16

    arXiv 2023

  8. [8]

    A confederacy of models: A comprehensive evaluation of llms on creative writing

    Carlos Gómez-Rodríguez and Paul Williams. A confederacy of models: A comprehensive evaluation of llms on creative writing. arXiv preprint arXiv:2310.08433, 2023. 1 10

    arXiv 2023

Show all 59 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 1, 2, 3, 7, 16

    2025 arXiv

  2. [10]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. 7

    2022 arXiv

  3. [11]

    Long-context llms meet rag: Overcoming challenges for long inputs in rag

    Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag: Overcoming challenges for long inputs in rag. In The Thirteenth International Conference on Learning Representations, 2024. 1, 5

    2024

  4. [12]

    Llm alignment as retriever optimization: An information retrieval perspective

    Bowen Jin, Jinsung Yoon, Zhen Qin, Ziqi Wang, Wei Xiong, Yu Meng, Jiawei Han, and Sercan O Arik. Llm alignment as retriever optimization: An information retrieval perspective. arXiv preprint arXiv:2502.03699, 2025. 2

    2025 arXiv

  5. [13]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. 2, 3, 4, 5, 7, 9, 16, 18, 19

    2025 arXiv

  6. [14]

    Reinforcement learning: A survey

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237–285, 1996. 1, 2

    1996

  7. [15]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 7

    2001 arXiv

  8. [16]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769–6781, 2020. 8

    2020

  9. [17]

    A survey of reinforcement learning from human feedback

    Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 10, 2023. 2

    2023

  10. [18]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  11. [19]

    Med-r1: Reinforce- ment learning for generalizable medical reasoning in vision-language models

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-r1: Reinforce- ment learning for generalizable medical reasoning in vision-language models. arXiv preprint arXiv:2503.13939, 2025. 1

    2025

  12. [20]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. 2

    2024 arXiv

  13. [21]

    Large language models in finance: A survey

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance , pages 374–382, 2023. 3

    2023

  14. [22]

    Rec-r1: Bridging generative large language mod- els and user-centric recommendation systems via reinforcement learning

    Jiacheng Lin, Tian Wang, and Kun Qian. Rec-r1: Bridging generative large language mod- els and user-centric recommendation systems via reinforcement learning. arXiv preprint arXiv:2503.24289, 2025. 2, 5

    2025

  15. [23]

    Ra-dit: Retrieval-augmented dual instruction tuning

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Learning Representations, 2023. 5

    2023

  16. [24]

    Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners

    Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504.14239, 2025. 1 11

    2025 arXiv

  17. [25]

    Fin-r1: A large language model for financial reasoning through reinforcement learning

    Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, et al. Fin-r1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252, 2025. 1

    2025

  18. [26]

    Chatqa: Surpassing gpt-4 on conversational qa and rag

    Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. Advances in Neural Information Processing Systems, 37:15416–15459, 2024. 1

    2024

  19. [27]

    Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs

    Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836, 2018. 8, 9

    2018

  20. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  21. [29]

    A study of generative large language model for medical research and healthcare

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210, 2023. 3

    2023

  22. [30]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. 7, 9, 20

    2022 arXiv

  23. [31]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 3

    2023

  24. [32]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. 9, 20

    2024

  25. [33]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009. 2, 8, 9

    2009

  26. [34]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539–68551, 2...

    2023

  27. [35]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 2

    2024 arXiv

  28. [36]

    Agentic retrieval-augmented generation: A survey on agentic rag

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval-augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136, 2025. 3

    2025 arXiv

  29. [37]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025. 2, 3

    2025 arXiv

  30. [38]

    Asqa: Factoid questions meet long-form answers

    Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092, 2022. 16

    2022 arXiv

  31. [39]

    Reinforcement learning

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1):126–134, 1999. 1, 2

    1999

  32. [40]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 3

    2024 arXiv

  33. [41]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509, 2022. 1, 3 12

    2022 arXiv

  34. [42]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. 8, 9

    2022 arXiv

  35. [43]

    Reinforcement learning for reasoning in large language models with one training example

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025. 16

    2025 arXiv

  36. [44]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025. 1

    2025 arXiv

  37. [45]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 1

    2022

  38. [46]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024. 9, 20

    2024 arXiv

  39. [47]

    Simple statistical gradient-following algorithms for connectionist reinforce- ment learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229–256, 1992. 2

    1992

  40. [48]

    Gui-r1: A generalist r1-style vision-language action model for gui agents

    Xiaobo Xia and Run Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025. 1

    2025 arXiv

  41. [49]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025. 1

    2025 arXiv

  42. [50]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 7

    2024 arXiv

  43. [51]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. 16

    2018 arXiv

  44. [52]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. 3, 4

    2023

  45. [53]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 2

    2025 arXiv

  46. [54]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025. 2

    2025 arXiv

  47. [55]

    Vision-r1: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning

    Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Vision-r1: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning. arXiv preprint arXiv:2503.18013, 2025. 1

    2025 arXiv

  48. [56]

    Benchmarking large language models for news summarization

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57, 2024. 1

    2024

  49. [57]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023. 1, 3

    2023 arXiv

  50. [58]

    <| im_start | > as si sta nt

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025. 2, 3, 4, 5, 16 13 A Limitations In this work, we ...

    2025 arXiv

  51. [1953]

    Delicatessen

    is a French film director, producer, and screenwriter. His films are known to mix elements of fantasy, reality and science fiction either to create idealized realities or to give relevance to mundane situations ... </information> <think> From the search results, I have gathere...

    1991

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.