Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

SEED, a training loop that distills a model's own written hindsight skills into token-level reinforcement-learning signals, claims consistent gains over outcome-only RL across text and vision agentic benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:04 UTC pith:IEZ5OJNE

load-bearing objection SEED's synchronized self-evolving distillation is a genuinely new and sensibly implemented mechanism, but the empirical claims need a table audit, seeded variance, and a skill-content control before I'd take the headline numbers at face value. the 3 major comments →

arxiv 2607.14777 v1 pith:IEZ5OJNE submitted 2026-07-16 cs.CL

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Jinyang Wu , Shuo Yang , Zhengxi Lu , Fan Zhang , Yuhao Shen , Lang Feng , Haoran Luo , Zheng Lian
show 3 more authors
Shuai Zhang Zhengqi Wen Jianhua Tao
This is my paper
classification cs.CL
keywords self-evolving distillationon-policy distillationhindsight skillsagentic reinforcement learninglanguage agentstoken-level credit assignmentsparse rewardssample efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes SEED, a training loop in which an agent learns from the hindsight skills it writes about its own completed trajectories. Outcome-based reinforcement learning only rewards whole episodes, so the authors add a dense token-level signal: each sampled action is re-scored with and without a self-generated skill, and the skill-induced probability shift is distilled back into the ordinary policy. Because the same model both acts and analyzes, the supervision evolves with the policy instead of going stale. The central claim is that this joint objective improves success rates, sample efficiency, and cross-domain generalization on text and vision agentic tasks.

Core claim

The paper's central claim is that on-policy hindsight skills—natural-language rules extracted by the policy from its own completed trajectories—can serve as a dense, self-evolving credit-assignment signal when distilled through a confidence gate. Formally, SEED optimizes L_RL + λ_opd E[m·g·(sg[ℓ_skill] − ℓ_θ)], where ℓ_skill is the log-probability of the sampled action under a skill-augmented context, ℓ_θ is the ordinary log-probability, and g = σ(β_opd Δ) gates tokens by the detached log-probability shift. The gate converts variance in skill support into signed token-level credit, so tokens favored by the hindsight skill are promoted and unsupported tokens suppressed. The paper reports cons

What carries the argument

The mechanism is the gated on-policy distillation (OPD) objective: for each on-policy sampled action token, the model re-scores the same token under the original context and under a context augmented with a self-generated hindsight skill; the detached log-probability shift Δ = sg[ℓ_skill − ℓ_θ] is passed through a sigmoid gate g = σ(β_opd Δ), and the loss E[m·g·(sg[ℓ_skill] − ℓ_θ)] pushes the ordinary policy toward skill-endorsed tokens. The self-evolving loop arises because the same model parameterizes both the actor and the trajectory analyzer, so each policy update refreshes the supervisor.

Load-bearing premise

The central claim depends on self-generated hindsight skills being behaviorally informative—specifically, that skill-induced gates assign systematically more support to actions with higher expected return (positive covariance between gate and action value); the paper's propositions establish only that the signal is non-zero and varies across tokens, not that it is correct.

What would settle it

Decouple the analyzer from the actor by freezing the trajectory analyzer at an early checkpoint and continuing to train only the policy with the joint objective. If performance does not degrade as the policy evolves, the self-evolving component is not the source of the gains. More directly, replace self-generated skills with random or adversarially wrong skills of the same format; if SEED still shows large improvements, the distillation signal is not carrying the claimed behavioral information.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, hindsight supervision from a model's own trajectories can fill the gap between sparse trajectory rewards and token-level credit assignment, improving success rates by 5.5–39.0 points over outcome-only RL across backbones.
  • Skills are internalized into parameters, so inference requires no skill prompts, external memory, or retrieval modules; deployment cost is unchanged.
  • Sample efficiency improves: with only 60% of training data, SEED matches or exceeds full-data outcome-only RL on ALFWorld.
  • Cross-domain generalization improves: on the ALFWorld unseen split, SEED gains +15.3 points macro-average over the outcome-only baseline, improving in five of six task families.
  • The self-evolving loop transfers to vision-based agentic tasks, with gains on Sokoban and EZPoints over outcome-only RL.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the gate is computed from the same model's own probability shift, the method will tend to reinforce behaviors the model already finds plausible; a natural test is whether SEED's ceiling is set by the analyzer's skill quality, so comparing against a fixed oracle-supervised teacher could reveal the gap.
  • The covariance condition in Equation 8 suggests a diagnostic: track the correlation between gate weights and per-token value estimates during training; if it turns negative, the distillation signal is actively harmful and the gate should be downweighted.
  • The staleness bound in Proposition 3 predicts that sensitivity to analyzer lag grows with the gate sharpness β_opd, implying a testable extension: anneal β_opd over training or study its interaction with learning rate.
  • Since skills are parametric and prompt-free, SEED could complement—not replace—external memory systems: a policy that also has retrieval access might distill retrieved skills into parameters, then drop the retrieval at inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SEED presents a two-stage self-distillation method for agentic LLM RL. Stage 1 uses externally annotated offline trajectories to SFT a policy into a hindsight-skill generator; Stage 2 repeatedly collects on-policy rollouts with the frozen checkpoint, analyzes them into natural-language skills with the same checkpoint, and jointly optimizes GRPO with a gate-weighted token-level distillation loss that re-scores sampled actions under ordinary and skill-augmented contexts (Section 3.3, Eq. 1). The paper reports consistent gains over GRPO and several self-distillation baselines on ALFWorld, WebShop, Search-based QA, and, in the appendix, two vision-based benchmarks, across three Qwen backbones. The theoretical appendix proves structural properties (occupancy-matched target, dense signal under tied rewards, analyzer-staleness bound) and explicitly disclaims monotonic improvement without skill informativeness.

Significance. If the empirical claims hold, SEED would be a practical and inexpensive way to provide denser credit in sparse-reward agentic RL using only self-generated natural-language skills, with no additional inference-time cost. Strengths of the submission include a clear algorithm description, release of code, a detailed implementation appendix, an honest theoretical analysis that states its limitations, and multi-backbone/multi-domain experiments. The main open issues are (i) arithmetic inconsistencies in the headline aggregate results and (ii) the absence of any experiment that isolates skill content from other effects of the auxiliary loss; both must be resolved before the central claim is supported.

major comments (3)
  1. [Table 1, Eq. (16)] The ALFWorld 'Avg' column is not the unweighted mean of the six per-category success rates defined in Eq. (16). For Qwen2.5-3B, the GRPO row has per-category values whose mean is 70.7, not the reported 75.0; OPSD computes to 23.3, not 28.1; and Skill-GRPO computes to 59.9, not 60.2. Because the headline gains in §4.2 ('14.9-45.9 points over GRPO') are measured on these aggregates, all reported averages need to be recomputed and corrected or the main results table cannot be trusted.
  2. [Appendix A.1, Eq. (8); Table 2] The central mechanism is not directly tested. Eq. (8) shows that the skill-reweighted target is better than the current policy only when Cov_{v~π_k}(Q_k(c,v), w_k(c,v)) > 0. Proposition 2 (Eq. 11) proves only that the OPD gradient is nonzero when the gate is non-constant; the paper itself calls this 'informativeness, not automatic correctness,' and Appendix E concedes that self-generated skills 'can inherit model errors.' The Table 2 ablations always retain informative skill content, and the w/o-OPD condition (SFT+GRPO, 87.0) already provides a 12-point gain over GRPO (75.0), which is larger than the additional 4.8 points from OPD. A matched control with shuffled, random, or otherwise corrupted skills is necessary to attribute the gains to skill informativeness rather than to added KL regularization or context diversity.
  3. [Section 3.3, Eq. (1); Appendix A.1, Eqs. (4)-(5)] The OPD target is self-referential by construction: the same network acts as rollout actor, trajectory analyzer, and teacher, and Proposition 1 shows that the auxiliary gradient is a KL toward a skill-reweighted version of the model's own policy. Since skill quality is only format-checked (§3.2), and since the teacher log-probabilities are detached but generated from the same model, the auxiliary signal can reinforce existing biases. The paper should report evidence that the value-alignment covariance in Eq. (8) is actually positive during training, or provide an ablation with random/shuffled skills, to establish that the dense signal tracks action quality rather than self-agreement.
minor comments (5)
  1. [Proposition 2, Appendix A.2] The proof refers to 'the clipped reward-driven term in Eq. 3.3,' but no equation number 3.3 exists; it should reference the displayed GRPO loss in Section 3.3.
  2. [All main tables] Results are reported as point estimates without standard deviations, confidence intervals, or number of seeds. This is especially relevant for Search-based QA, where some claimed gains are as small as 1.4 points; please add run-level variance or at least clarify whether one seed was used.
  3. [Abstract and Appendix C.3] The abstract and conclusion emphasize vision-based agentic tasks, but the main experiments are text-only; the vision results appear only in an appendix. Consider moving a compact version of Table 8 into the main text or tempering the claim.
  4. [Table 2, 'w/o On-Policy Skill'] The static-library condition is not fully specified: the number, format, and source of the static skills should be described to ensure it is a matched comparison to the on-policy skill condition.
  5. [Throughout] The manuscript needs proofreading: the title/header has missing spaces ('SELF-EVOLVINGON-POLICYDISTILLATION FORAGENTICREINFORCEMENTLEARNING'), the author block contains a corrupted symbol ('gl⌢be'), and some arXiv preprint references use future IDs without clearly indicating availability.

Circularity Check

0 steps flagged

No significant circularity: theoretical results are explicitly non-performance identities and the central claims are validated against external benchmarks.

full rationale

SEED's formal results (Prop. 1-3, Eqs. 4-11) are algebraic identities and staleness bounds derived from the algorithm's own losses; the paper explicitly disclaims that they imply improvement ('These are local statements ... do not by themselves imply monotonic return improvement'; 'This establishes informativeness, not automatic correctness'). The claimed gains over GRPO/static distillation are empirical comparisons on external benchmarks (ALFWorld, WebShop, Search-QA, Sokoban/EZPoints), not consequences of the identities. Eq. 8 openly states the covariance condition on which any benefit depends, and Appendix E concedes that self-generated supervision 'can inherit model errors and plateau below oracle-supervised training'—a limitation, not a disguised input. The only overlapping-author citation is the gate borrowed from SDAR ('Following SDAR ... confidence gate'), an implementation detail rather than a load-bearing proof. The self-referential OPD objective (same model as teacher and student) is the method by definition, not a circular prediction; its success or failure is an experimental question the paper tests externally. Therefore, aside from one minor non-load-bearing self-citation, no circular step rises to the level of Eq. X = Eq. Y by construction or fitted parameter renamed as prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The method's contribution is a training loop, so most of the ledger consists of design assumptions rather than fitted constants. The three hand-set scalars (λ_opd, β_opd, SFT scale) plus the validity filter are the entry cost of the pipeline. The most important ledger item is the unproven informativeness axiom (positive covariance between skill gates and action values), which the paper itself identifies as the condition connecting the theory to the empirical gains.

free parameters (4)
  • λ_opd (OPD loss coefficient) = 0.01
    Chosen by hand to balance RL and distillation terms; no sensitivity analysis reported (Table 5).
  • β_opd (gate sharpness) = 5.0
    Chosen by hand; the staleness bound in Prop. 3 scales analyzer-mismatch sensitivity by β_opd/4, so this choice interacts with the theoretical guarantees (Table 5, Appendix A.3).
  • SFT data scale (M tasks, K0 rollouts) = M=180, K0=8 → 1,440 trajectories
    Hand-chosen annotation budget; no study of how skill quality scales with annotation volume (§3.2, B.4).
  • skill validity filter v_τ = valid-format annotations only
    Annotations retained only if correctly formatted; this ad-hoc exclusion shapes the SFT distribution and is non-reproducible without the GLM-5.2 prompt output (§3.2).
axioms (5)
  • ad hoc to paper Self-generated hindsight skills are behaviorally informative (positive covariance with action value, Eq. 8)
    The paper's own theory requires Cov(Q_k, w_k) > 0 for the distillation target to be better than the current policy; never proven, explicitly conceded in Appendix A.1 and E. This is the load-bearing unproven premise.
  • domain assumption GRPO provides a valid outcome-based backbone with group-relative advantages
    Imported from Shao et al. 2024; standard practice, but the paper assumes trajectory-level terminal rewards are the only environment signal (§3.1-3.3).
  • standard math Softmax full support and bounded score gradients (∥∇ log π_θ∥ ≤ G)
    Used in Propositions 1-3 to obtain 0 < w_k < 1 and the staleness bounds (Appendix A).
  • domain assumption GLM-5.2 external analyzer produces annotations of sufficient quality to bootstrap skill SFT
    Stage 1 depends on a proprietary model at temperature 0; the quality of the 1,440 trajectory-skill pairs is not audited beyond format validation (§3.2, B.4).
  • ad hoc to paper A single appended skill (function H) is a sufficient representation of trajectory-level hindsight
    The method assumes one natural-language skill per trajectory captures reusable guidance; no comparison against multi-skill, hierarchical, or state-anchored summaries (Appendix E lists these as future work).
invented entities (1)
  • Hindsight skill (natural-language training-time construct) independent evidence
    purpose: Intermediate dense supervision extracted from completed trajectories and distilled into the policy
    Skills are directly observable text (Table 4) and their contribution is tested by ablations (Table 2), giving a falsifiable handle inside the paper; they are not an unobservable theoretical entity.

pith-pipeline@v1.3.0-alltime-deepseek · 37356 in / 19722 out tokens · 154920 ms · 2026-08-02T01:04:23.356101+00:00 · methodology

0 comments
read the original abstract

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.

Figures

Figures reproduced from arXiv: 2607.14777 by Fan Zhang, Haoran Luo, Jianhua Tao, Jinyang Wu, Lang Feng, Shuai Zhang, Shuo Yang, Yuhao Shen, Zheng Lian, Zhengqi Wen, Zhengxi Lu.

Figure 1
Figure 1. Figure 1: Overall performance overview. Compared with powerful baseline methods, SEED achieves the strongest average performance across three representative agentic benchmarks. be reinforced or corrected (Andrychowicz et al., 2017; Arjona-Medina et al., 2019; Uesato et al., 2022; Lightman et al., 2024). This leaves a supervision gap between episode-level outcomes and token-level policy learning. A failed trajectory … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SEED. Stage 1 (Hindsight Skill SFT) equips the policy to extract hindsight skills from completed trajectories. Stage 2 (Self-Evolving On-Policy Distillation) jointly optimizes outcome-based RL and skill-conditioned OPD in a self-evolving agentic loop. traces, feedback, or skills and use them to provide token-level guidance for agentic RL (Zhao et al., 2026; Wang et al., 2026; Lu et al., 2026a; … view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics on ALFWorld. We compare SEED and GRPO using Qwen2.5- 3B-Instruct as the backbone. Translucent curves show raw measurements, while solid curves show 13-point centered moving averages. 4.3 TRAINING DYNAMICS [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cross-domain generalizability on ALFWorld Unseen. SEED generally outper￾forms GRPO across unseen task types, demon￾strating stronger cross-domain generalizability [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison on ALFWorld. For the task “put a candle in toilet,” the GRPO￾trained agent searches the target receptacle before locating the candle, takes an irrelevant toilet-paper item, and then enters an off-task loop. In contrast, SEED systematically checks plausible shelves, finds the candle, and completes the required placement in five steps. icy to internalize reusable guidance without relyi… view at source ↗
Figure 7
Figure 7. Figure 7: A representative trajectory on Sokoban. The sequence shows six consecutive actions executed by the agent. Arrows indicate the temporal progression of the trajectory, and the action taken at each step is displayed below the corresponding observation. C.4 ADDITIONAL TRAINING DYNAMICS [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Success rates across three backbones and three domains. Success rates increase over training in all nine settings, showing consistent learning across model scales and agentic tasks [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: OPD loss dynamics. The loss generally decreases and stabilizes during training, indicat￾ing that the policy progressively internalizes the behavioral guidance provided by hindsight skills. been placed, returns for the second, and completes the remaining subgoal without losing progress. These trajectories demonstrate coherent state tracking and precondition management over extended interactions. The Search-… view at source ↗
Figure 10
Figure 10. Figure 10: Prompt of analyzer. Actor Prompt You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description} Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} You are now at step {current_step} and your current observation is: {current_observation} Y… view at source ↗
Figure 11
Figure 11. Figure 11: Prompt of actor (the policy model) in ALFWorld. [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: ALFWorld Example 1. SEED locates and cleans the ladle, opens the closed drawer, and completes the placement in seven steps, demonstrating task decomposition and correct precondition handling. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: ALFWorld Example 2. SEED places two books through repeated trips between the desk and bed while retaining the completed first placement, demonstrating progress tracking over an extended trajectory. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Search-based QA Example 1. A single query retrieves evidence that both people are mathematicians, allowing SEED to answer without unnecessary searches. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Search-based QA Example 2. SEED first identifies Finding Neverland, then performs a targeted second search to verify that Marc Forster directed the film. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: WebShop Example 1. SEED identifies a green shower cap and hair turban set under $40, verifies the selected variant, and completes the purchase in four steps. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: WebShop Example 2. SEED preserves the requested product constraints and selects mossy oak country and 5x-large big before purchasing a matching long-sleeve shirt under $60. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation

    cs.AI 2026-07 conditional novelty 6.0

    MAGA distills three domain-specific GUI agents into one student by amplifying supervision on wrongly generated action tokens and feeding the teacher a training-only hint of the correct action type, improving mean succ...

  2. EvoReason: Self-Evolving Reasoning Primitive-Guided On-Policy Distillation for Latent Reasoning in Generative Recommendation

    cs.IR 2026-07 conditional novelty 5.0

    A self-evolving, primitive-guided distillation method for latent reasoning in generative recommendation reports consistent ranking gains over existing latent-reasoning baselines.

Reference graph

Works this paper leans on

74 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    Frontiers of Computer Science , year =

    A Survey on Large Language Model based Autonomous Agents , author =. Frontiers of Computer Science , year =

  2. [2]

    Science China Information Sciences , year =

    The Rise and Potential of Large Language Model Based Agents: A Survey , author =. Science China Information Sciences , year =

  3. [3]

    2023 , url =

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url =

    2023

  4. [4]

    Advances in Neural Information Processing Systems , year =

    Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems , year =

  5. [5]

    and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E

    Patil, Shishir G. and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E. , booktitle =. Gorilla: Large Language Model Connected with Massive. 2024 , url =

    2024

  6. [6]

    AgentBench: Evaluating

    Liu, Xiao and Yu, Hao and Zhang, Hanchen and Xu, Yifan and Lei, Xuanyu and Lai, Hanyu and Gu, Yu and Ding, Hangliang and Men, Kaiwen and Yang, Kejuan and Zhang, Shudan and Deng, Xiang and Zeng, Aohan and Du, Zhengxiao and Zhang, Chenhui and Shen, Sheng and Zhang, Tianjun and Su, Yu and Sun, Huan and Huang, Minlie and Dong, Yuxiao and Tang, Jie , year =. A...

    Pith/arXiv arXiv

  7. [7]

    International Conference on Learning Representations , year =

    Shridhar, Mohit and Yuan, Xingdi and C. International Conference on Learning Representations , year =

  8. [8]

    2022 , url =

    Yao, Shunyu and Chen, Howard and Yang, John and Narasimhan, Karthik , booktitle =. 2022 , url =

    2022

  9. [9]

    2023 , url =

    Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Samuel and Wang, Boshi and Sun, Huan and Su, Yu , booktitle =. 2023 , url =

    2023

  10. [10]

    Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle =. 2024 , url =

    2024

  11. [11]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =. 2024 , url =

    2024

  12. [12]

    2024 , url =

    Trivedi, Harsh and Khot, Tushar and Hartmann, Mareike and Manku, Ruskin Raj and Dong, Vinty and Li, Edward and Gupta, Shashank and Sabharwal, Ashish and Balasubramanian, Niranjan , booktitle =. 2024 , url =

    2024

  13. [13]

    2503.09516 , archivePrefix =

    Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Wang, Dong and Zamani, Hamed and Han, Jiawei , year =. 2503.09516 , archivePrefix =

    Pith/arXiv arXiv

  14. [14]

    Transactions of the Association for Computational Linguistics , volume =

    Natural Questions: A Benchmark for Question Answering Research , author =. Transactions of the Association for Computational Linguistics , volume =. 2019 , doi =

    2019

  15. [15]

    2017 , doi =

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel and Zettlemoyer, Luke , booktitle =. 2017 , doi =

    2017

  16. [16]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages =

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages =. 2023 , doi =

    2023

  17. [17]

    , booktitle =

    Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William and Salakhutdinov, Ruslan and Manning, Christopher D. , booktitle =. 2018 , doi =

    2018

  18. [18]

    Constructing A Multi-hop

    Ho, Xanh and Duong Nguyen, Anh-Khoa and Sugawara, Saku and Aizawa, Akiko , booktitle =. Constructing A Multi-hop. 2020 , doi =

    2020

  19. [19]

    2022 , doi =

    Trivedi, Harsh and Balasubramanian, Niranjan and Khot, Tushar and Sabharwal, Ashish , journal =. 2022 , doi =

    2022

  20. [20]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    Measuring and Narrowing the Compositionality Gap in Language Models , author =. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =. 2023 , doi =

    2023

  21. [21]

    Advances in Neural Information Processing Systems , year =

    Training Language Models to Follow Instructions with Human Feedback , author =. Advances in Neural Information Processing Systems , year =

  22. [22]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , year =. 2402.03300 , archivePrefix =

    Pith/arXiv arXiv

  23. [23]

    2504.20073 , archivePrefix =

    Wang, Zihan and Wang, Kangrui and Wang, Qineng and Zhang, Pingyue and Li, Linjie and Yang, Zhengyuan and Jin, Xing and Yu, Kefan and Nguyen, Minh Nhat and Liu, Licheng and Gottlieb, Eli and Lu, Yiping and Cho, Kyunghyun and Wu, Jiajun and Fei-Fei, Li and Wang, Lijuan and Choi, Yejin and Li, Manling , year =. 2504.20073 , archivePrefix =

    Pith/arXiv arXiv

  24. [24]

    and Yang, Yuqing , year =

    Luo, Xufang and Zhang, Yuge and He, Zhiyuan and Wang, Zilong and Zhao, Siyun and Li, Dongsheng and Qiu, Luna K. and Yang, Yuqing , year =. Agent Lightning: Train. 2508.03680 , archivePrefix =

    Pith/arXiv arXiv

  25. [25]

    2017 , eprint =

    Proximal Policy Optimization Algorithms , author =. 2017 , eprint =

    2017

  26. [26]

    International Conference on Learning Representations , year =

    High-Dimensional Continuous Control Using Generalized Advantage Estimation , author =. International Conference on Learning Representations , year =

  27. [27]

    and Gillhofer, Michael and Widrich, Michael and Unterthiner, Thomas and Brandstetter, Johannes and Hochreiter, Sepp , booktitle =

    Arjona-Medina, Jose A. and Gillhofer, Michael and Widrich, Michael and Unterthiner, Thomas and Brandstetter, Johannes and Hochreiter, Sepp , booktitle =. 2019 , url =

    2019

  28. [28]

    The Landscape of Agentic Reinforcement Learning for

    Guibin Zhang and Hejia Geng and Xiaohang Yu and Zhenfei Yin and Zaibin Zhang and Zelin Tan and Heng Zhou and Zhong-Zhi Li and Xiangyuan Xue and Yijiang Li and Yifan Zhou and Yang Chen and Chen Zhang and Yutao Fan and Zihu Wang and Songtao Huang and Francisco Piedrahita Velez and Yue Liao and Hongru WANG and Mengyue Yang and Heng Ji and Jun Wang and Shuich...

    2026

  29. [29]

    2022 , eprint =

    Solving Math Word Problems with Process- and Outcome-Based Feedback , author =. 2022 , eprint =

    2022

  30. [30]

    International Conference on Learning Representations , year =

    Let's Verify Step by Step , author =. International Conference on Learning Representations , year =

  31. [31]

    2021 , eprint =

    Training Verifiers to Solve Math Word Problems , author =. 2021 , eprint =

    2021

  32. [32]

    Advances in Neural Information Processing Systems , year =

    Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , year =

  33. [33]

    2024 , url =

    Zhao, Andrew and Huang, Daniel and Xu, Quentin and Lin, Matthieu and Liu, Yong-Jin and Huang, Gao , booktitle =. 2024 , url =

    2024

  34. [34]

    2023 , eprint =

    Voyager: An Open-Ended Embodied Agent with Large Language Models , author =. 2023 , eprint =

    2023

  35. [35]

    Advances in Neural Information Processing Systems , year =

    Self-Refine: Iterative Refinement with Self-Feedback , author =. Advances in Neural Information Processing Systems , year =

  36. [36]

    2015 , eprint =

    Distilling the Knowledge in a Neural Network , author =. 2015 , eprint =

    2015

  37. [37]

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages =

    Sequence-Level Knowledge Distillation , author =. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages =. 2016 , doi =

    2016

  38. [38]

    Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages =

    A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , author =. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages =. 2011 , url =

    2011

  39. [39]

    International Conference on Learning Representations , year =

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author =. International Conference on Learning Representations , year =

  40. [40]

    2026 , eprint =

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author =. 2026 , eprint =

    2026

  41. [41]

    2026 , eprint =

    Reinforcement Learning via Self-Distillation , author =. 2026 , eprint =

    2026

  42. [42]

    2604.10674 , archivePrefix =

    Wang, Hao and Wang, Guozhi and Xiao, Han and Zhou, Yufeng and Pan, Yue and Wang, Jichao and Xu, Ke and Wen, Yafei and Ruan, Xiaohu and Chen, Xiaoxin and Qi, Honggang , year =. 2604.10674 , archivePrefix =

    Pith/arXiv arXiv

  43. [43]

    Self-Distilled

    Yang, Chenxu and Qin, Chuanyu and Si, Qingyi and Chen, Minghui and Gu, Naibin and Yao, Dingyu and Lin, Zheng and Wang, Weiping and Wang, Jiaqi and Duan, Nan , year =. Self-Distilled. 2604.03128 , archivePrefix =

    Pith/arXiv arXiv

  44. [44]

    2026 , eprint =

    Self-Distilled Agentic Reinforcement Learning , author =. 2026 , eprint =

    2026

  45. [45]

    2605.07725 , archivePrefix =

    Zhong, Qiyong and Zheng, Mao and Song, Mingyang and Lin, Xin and Sun, Jie and Jiang, Houcheng and Wang, Xiang and Fang, Junfeng , year =. 2605.07725 , archivePrefix =

    Pith/arXiv arXiv

  46. [46]

    2026 , eprint =

    Scaling Reasoning Efficiently via Relaxed On-Policy Distillation , author =. 2026 , eprint =

    2026

  47. [47]

    arXiv preprint arXiv:2606.26790 , year=

    OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning , author=. arXiv preprint arXiv:2606.26790 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2411.18478 , year=

    Beyond examples: High-level automated reasoning paradigm in in-context learning via mcts , author=. arXiv preprint arXiv:2411.18478 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2605.22177 , year=

    Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensembles , author=. arXiv preprint arXiv:2605.22177 , year=

    Pith/arXiv arXiv

  50. [50]

    Advances in neural information processing systems , volume=

    Hindsight experience replay , author=. Advances in neural information processing systems , volume=

  51. [51]

    Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism

    Shen, Yuhao and Liu, Tianyu and Shen, Junyi and Wu, Jinyang and Kong, Quan and Li, Huan and Wang, Cong. Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026

    2026

  52. [52]

    arXiv preprint arXiv:2503.21460 , year=

    Large language model agent: A survey on methodology, applications and challenges , author=. arXiv preprint arXiv:2503.21460 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2605.06234 , year=

    RobotEQ: Transitioning from Passive Intelligence to Active Intelligence in Embodied AI , author=. arXiv preprint arXiv:2605.06234 , year=

    Pith/arXiv arXiv

  54. [54]

    SPARK : Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning

    Wu, Jinyang and Yang, Shuo and Shen, Yuhao and Zhang, Shuai and Wen, Zhengqi and Tao, Jianhua. SPARK : Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026

    2026

  55. [55]

    arXiv preprint arXiv:2602.05843 , year=

    OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions , author=. arXiv preprint arXiv:2602.05843 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2607.05147 , year=

    DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation , author=. arXiv preprint arXiv:2607.05147 , year=

    Pith/arXiv arXiv

  57. [57]

    ATLAS : Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning

    Wu, Jinyang and Zhai, Guocheng and Jin, Ruihan and Yuan, Jiahao and Shen, Yuhao and Zhang, Shuai and Wen, Zhengqi and Tao, Jianhua. ATLAS : Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026

    2026

  58. [58]

    2026 , eprint=

    Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading , author=. 2026 , eprint=

    2026

  59. [59]

    arXiv preprint arXiv:2601.18137 , year=

    DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints , author=. arXiv preprint arXiv:2601.18137 , year=

    arXiv

  60. [60]

    arXiv preprint arXiv:2401.13649 , year=

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks , author=. arXiv preprint arXiv:2401.13649 , year=

    Pith/arXiv arXiv

  61. [61]

    arXiv preprint arXiv:2604.02268 , year=

    Skill0: In-context agentic reinforcement learning for skill internalization , author=. arXiv preprint arXiv:2604.02268 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv preprint arXiv:2412.15115 , year =

    Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 Technical Report , author =. arXiv preprint arXiv:2505.09388 , year =

    Pith/arXiv arXiv

  64. [64]

    5-vl technical report , author=

    Qwen2. 5-vl technical report , author=. arXiv preprint arXiv:2502.13923 , year=

    Pith/arXiv arXiv

  65. [65]

    Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning , url =

    Zhai, Yuexiang and Bai, Hao and Lin, Zipeng and Pan, Jiayi and Tong, Shengbang and Zhou, Yifei and Suhr, Alane and Xie, Saining and LeCun, Yann and Ma, Yi and Levine, Sergey , booktitle =. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning , url =. doi:10.52202/079017-3522 , editor =

    doi:10.52202/079017-3522

  66. [66]

    , title =

    Schrader, Max-Philipp B. , title =. GitHub repository , howpublished =. 2018 , publisher =

    2018

  67. [67]

    2026 , month = jun, day =

    2026

  68. [68]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2025 , doi=

    2025

  69. [69]

    Forty-third International Conference on Machine Learning , year=

    On the Generalization Gap in Self-Evolving Language Model Reasoning , author=. Forty-third International Conference on Machine Learning , year=

  70. [70]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Mitigating Self-Preference by Authorship Obfuscation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2026 , doi=

    2026

  71. [71]

    International Conference on Learning Representations , year=

    Training Language Models to Self-Correct via Reinforcement Learning , author=. International Conference on Learning Representations , year=

  72. [72]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Importance Weighting Can Help Large Language Models Self-Improve , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2025 , doi=

    2025

  73. [73]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2026 , doi=

    2026

  74. [74]

    2025 , address=

    Ge, Yubin and Romeo, Salvatore and Cai, Jason and Sunkara, Monica and Zhang, Yi , booktitle=. 2025 , address=. doi:10.18653/v1/2025.emnlp-main.839 , url=

    work page doi:10.18653/v1/2025.emnlp-main.839 2025