pith. sign in

arxiv: 2605.22511 · v2 · pith:CF5QLFCRnew · submitted 2026-05-21 · 💻 cs.AI · cs.CL· cs.IR

Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning

Zihan Liang , Yufei Ma , Ben Chen , Zhipeng Qian , Xuxin Zhang , Huangyu Dai , Lingtao Mao This is my paper

Pith reviewed 2026-06-30 17:21 UTC · model grok-4.3

classification 💻 cs.AI cs.CLcs.IR
keywords self-distillationsearch-augmented reasoningGRPOself-evolutionon-policy learningquestion answeringlanguage model post-training
0
0 comments X

The pith

Self-distillation to the model's own efficient sibling trajectories suffices to evolve search-augmented reasoning agents after GRPO.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that elaborate external supervision, auxiliary modules, tree search, and hand-crafted rewards are unnecessary for improving search-augmented language model agents. Instead, interleaving standard GRPO with on-policy self-distillation provides the needed supervision: after each GRPO round the model generates its own rollouts, then a token-level forward KL loss aligns its inference distribution to the distribution obtained when a privileged context reveals a more efficient sibling trajectory. This alignment occurs entirely within the model's own outputs and yields strong exact-match scores on QA benchmarks. A sympathetic reader would care because the result shows that self-generated signals alone can produce measurable self-evolution in a standard post-training pipeline.

Core claim

Search-E1 demonstrates that a search-augmented agent improves through only vanilla GRPO interleaved with on-policy self-distillation. After each GRPO round the policy rolls out on its training questions; a token-level forward KL objective then aligns the policy's inference-time distribution to its own distribution under a privileged context that exposes a more efficient sibling trajectory. Despite its simplicity the procedure supplies dense per-step supervision and reaches 0.440 average exact match on seven QA benchmarks with Qwen2.5-3B, surpassing open-source baselines at both scales.

What carries the argument

On-policy self-distillation (OPSD) via token-level forward KL that aligns the policy distribution at inference time to the distribution produced under privileged context containing a more efficient sibling trajectory.

If this is right

  • Search-augmented agents can improve using only their own rollouts and internal self-distillation signals.
  • No external stronger systems, process reward models, retrospective critics, or custom reward bonuses are required.
  • Dense per-step supervision arises naturally from aligning to the model's own more efficient trajectories.
  • The same simple loop works at both 3B and larger scales on standard QA tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The privileged-context mechanism could be tested as a general way to extract better self-demonstrations in other post-training settings.
  • Repeated rounds of GRPO plus OPSD might produce continued iterative gains without saturation or external data.
  • The approach might extend beyond QA to other domains that already use search-augmented rollouts.

Load-bearing premise

That the model's own more efficient sibling trajectories revealed under privileged context supply a training signal strong enough to drive genuine self-evolution without external help.

What would settle it

Applying the full Search-E1 procedure and observing no gain or a drop in average exact match on the seven QA benchmarks relative to GRPO alone would falsify the claim.

Figures

Figures reproduced from arXiv: 2605.22511 by Ben Chen, Huangyu Dai, Lingtao Mao, Xuxin Zhang, Yufei Ma, Zhipeng Qian, Zihan Liang.

Figure 1
Figure 1. Figure 1: Overview of Search-E1. Top: a GRPO round with exact-match outcome reward. Bottom: an OFSD round in which the student conditions on q +τ stu and the teacher on q +τ ref +τ stu, aligned by a token-level forward KL. supervision from a stronger system, either by distilling sub-question decompositions from a 72B teacher (Xu et al., 2025) or by deriving step-wise rewards from GPT-4o annotations (Wang et al., 202… view at source ↗
Figure 1
Figure 1. Figure 1: Pair mining. After a GRPO round converges, we sample the policy on its training questions to obtain a fresh rollout pool: for each question q, we draw K trajectories {τ (1) q , . . . , τ (K) q } from the converged policy, annotated with its outcome reward R ∈ {0, 1} and the number of retrieval calls nsrch. For each question we then construct a pair (τ ref, τ stu). The reference τ ref is the correct traject… view at source ↗
read the original abstract

Post-training has become the dominant recipe for turning a language model into a competent search-augmented reasoning agent. A line of recent work pushes its performance further by adding elaborate machinery on top of this standard pipeline. These augmentations import external supervision from stronger external systems, attach auxiliary modules such as process reward models or retrospective critics, restructure the rollout itself with tree search or multi-stage curricula, or shape the reward with hand-crafted bonuses and penalties. Each addition delivers a measurable gain, but each also inflates the training pipeline and ties the recipe to resources or designs that may not always be available. We take a step back and ask whether any of this machinery is actually necessary, and propose Search-E1, a self-evolution method that lets a search-augmented agent improve through only vanilla GRPO interleaved with on-policy self-distillation (OPSD). After each GRPO round, the policy rolls out on its own training questions. A token-level forward KL objective then aligns the policy's inference-time distribution to its own distribution under a privileged context that exposes a more efficient sibling trajectory. Despite this simplicity, the procedure naturally provides dense per-step supervision. On seven QA benchmarks, Search-E1 reaches 0.440 average EM with Qwen2.5-3B, surpassing all open-source baselines at both scales. Code and complete version will be made public soon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes Search-E1, a self-evolution method for search-augmented reasoning agents that uses only vanilla GRPO interleaved with on-policy self-distillation (OPSD). After each GRPO round, the policy rolls out on its own training questions and applies a token-level forward KL objective to align its inference-time distribution to its distribution under a privileged context exposing a more efficient sibling trajectory. The abstract claims this yields 0.440 average EM on seven QA benchmarks with Qwen2.5-3B, surpassing all open-source baselines at both scales, without external supervision, auxiliary modules, or hand-crafted rewards.

Significance. If the experimental claims hold, the result would be significant for demonstrating that self-generated trajectories and self-distillation alone can drive measurable improvement in search-augmented agents, potentially simplifying post-training pipelines that currently rely on external systems or elaborate machinery. The approach is credited for avoiding fitted parameters or self-referential predictions in the described method.

major comments (2)
  1. [Abstract] Abstract: the central claim that Search-E1 surpasses all open-source baselines with 0.440 average EM is load-bearing but unsupported, as the abstract provides no benchmark list, no comparison tables, no error bars, and no implementation details on how the privileged sibling trajectory is generated or how OPSD is interleaved with GRPO. This prevents assessment of whether the data supports the self-evolution result.
  2. [Abstract] Abstract: the description of OPSD as providing 'dense per-step supervision' sufficient to drive self-evolution without additional machinery is a key assumption, but the abstract gives no equations, rollout details, or ablation evidence to verify that the forward KL alignment actually produces the claimed gains over standard GRPO.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for highlighting the need for greater self-containment in the abstract. The full manuscript contains the benchmark details, tables, equations, rollout procedures, and ablations referenced in the comments. We address each point below and will make targeted revisions to the abstract to improve clarity while preserving its brevity.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that Search-E1 surpasses all open-source baselines with 0.440 average EM is load-bearing but unsupported, as the abstract provides no benchmark list, no comparison tables, no error bars, and no implementation details on how the privileged sibling trajectory is generated or how OPSD is interleaved with GRPO. This prevents assessment of whether the data supports the self-evolution result.

    Authors: The abstract summarizes the headline result; the full paper explicitly lists the seven QA benchmarks, presents comparison tables against open-source baselines (including at both model scales), reports results with error bars where applicable, and details the privileged sibling trajectory generation (on-policy rollouts exposing more efficient paths) plus the GRPO-OPSD interleaving schedule in the Methods and Experiments sections. We agree the abstract would benefit from a concise addition noting the benchmark count and key method elements, and will revise it accordingly. revision: partial

  2. Referee: [Abstract] Abstract: the description of OPSD as providing 'dense per-step supervision' sufficient to drive self-evolution without additional machinery is a key assumption, but the abstract gives no equations, rollout details, or ablation evidence to verify that the forward KL alignment actually produces the claimed gains over standard GRPO.

    Authors: The abstract emphasizes the method's simplicity and the natural emergence of dense token-level supervision via forward KL. The full manuscript supplies the precise OPSD objective equations, rollout implementation details, and ablation results directly comparing Search-E1 to vanilla GRPO to quantify the gains. We will add a brief clause to the abstract referencing these supporting elements in the paper. revision: partial

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The provided abstract and description outline an empirical training procedure (vanilla GRPO interleaved with on-policy self-distillation using a privileged sibling trajectory) whose central claim is an observed performance gain on QA benchmarks. No mathematical derivation chain, equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the text. The method is presented as self-contained without external machinery, and the result is benchmark-driven rather than a first-principles reduction equivalent to its inputs by construction. Absent any quoted reduction of the form 'Eq. X equals input Y by definition,' the derivation is self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no information on free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5800 in / 1143 out tokens · 39799 ms · 2026-06-30T17:21:28.823261+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages · 15 internal anchors

  1. [1]

    Improving language models by retrieving from trillions of tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, O...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for llms via reinforcement learning.arXiv preprint arXiv:2503.19470,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Treegrpo: Tree-advantage grpo for online rl post-training of diffusion models.arXiv preprint arXiv:2512.08153,

    Zheng Ding and Weirui Ye. Treegrpo: Tree-advantage grpo for online rl post-training of diffusion models.arXiv preprint arXiv:2512.08153,

    work page arXiv

  4. [4]

    Dynasearcher: Dynamic knowl- edge graph augmented search agent via multi-reward reinforcement learning.arXiv preprint arXiv:2507.17365,

    8 Preprint Chuzhan Hao, Wenfeng Feng, Yuewei Zhang, and Hao Wang. Dynasearcher: Dynamic knowl- edge graph augmented search agent via multi-reward reinforcement learning.arXiv preprint arXiv:2507.17365,

    work page arXiv

  5. [5]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt ¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks.arXiv preprint arXiv:2005.11401,

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  6. [6]

    Search-o1: Agentic search-enhanced large reasoning models

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5420–5438, Suzhou, China,

    2025

  7. [7]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Unifying distillation and privileged information

    David Lopez-Paz, L ´eon Bottou, Bernhard Sch ¨olkopf, and Vladimir Vapnik. Unifying distillation and privileged information.arXiv preprint arXiv:1511.03643,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Qarm: Quantitative alignment multi-modal recommenda- tion at kuaishou.arXiv preprint arXiv:2411.11739,

    Xinchen Luo, Jiangxia Cao, Tianyu Sun, Jinkai Yu, Rui Huang, Wei Yuan, Hezheng Lin, Yichen Zheng, Shiyao Wang, Qigen Hu, et al. Qarm: Quantitative alignment multi-modal recommenda- tion at kuaishou.arXiv preprint arXiv:2411.11739,

    work page arXiv

  10. [10]

    GPT-4 Technical Report

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2024a. OpenAI. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024b. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [11]

    Qwen2.5 Technical Report

    Qwen. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Self-Distillation Enables Continual Learning

    Idan Shenfeld, Mehul Damani, Jonas H ¨ubotter, and Pulkit Agrawal. Self-distillation enables con- tinual learning.arXiv preprint arXiv:2601.19897,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning

    Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    ZeroSearch: Incentivize the Search Capability of LLMs without Searching

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Stepsearch: Igniting llms search ability via step-wise proximal policy optimization.arXiv preprint arXiv:2505.15107, 2025

    Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization.arXiv preprint arXiv:2505.15107,

    work page arXiv

  19. [19]

    Search-p1: Path-centric reward shaping for stable and efficient agentic rag training.arXiv preprint arXiv:2602.22576,

    Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun, Wenwei Li, Linfang Shang, Liqun Liu, Peng Shu, Huan Yu, and Jie Jiang. Search-p1: Path-centric reward shaping for stable and efficient agentic rag training.arXiv preprint arXiv:2602.22576,

    work page arXiv

  20. [20]

    Thinker: Train- ing llms in hierarchical thinking for deep search via multi-turn interaction.arXiv preprint arXiv:2511.07943,

    Jun Xu, Xinkai Du, Yu Ao, Peilong Zhao, Yang Li, Ling Zhong, Lin Yuan, Zhongpu Bo, Xiaorui Wang, Mengshu Sun, Zhengke Gui, Dalong Zhang, Zhaoyang Wang, Qiwei Wang, Yangyang Hou, Zhiying Yin, Haofen Wang, Huajun Chen, Lei Liang, and Jun Zhou. Thinker: Train- ing llms in hierarchical thinking for deep search via multi-turn interaction.arXiv preprint arXiv:2...

    work page arXiv

  21. [21]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, Brussels, Belgium,

    2018

  23. [23]

    ReAct: Synergizing Reasoning and Acting in Language Models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Criticsearch: Fine-grained credit assignment for search agents via a retrospective critic.arXiv preprint arXiv:2511.12159, 2025

    Yaocheng Zhang, Haohuan Huang, Zijun Song, Yuanheng Zhu, Qichao Zhang, Zijie Zhao, and Dongbin Zhao. Criticsearch: Fine-grained credit assignment for search agents via a retrospective critic.arXiv preprint arXiv:2511.12159, 2025a. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The ...

    work page arXiv