Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

ECHO claims that keeping each turn source-addressable — and routing positive outcome credit through the same selected context — makes long-horizon agent RL both more accurate and more efficient.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:32 UTC pith:6GJISB6C

load-bearing objection A genuinely new mechanism (reuse the reconstruction trace as the credit mask) in a well-structured empirical paper, but the headline accuracy gap is within binomial noise on an 83-example held-out set and there are no error bars or seeds, so the central claim is not yet statistically supported. the 3 major comments →

arxiv 2606.31650 v5 pith:6GJISB6C submitted 2026-06-30 cs.LG cs.AI

ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL

Zijun Xie , Binbin Zheng , Enlei Gong , Jihua Liu , Yuyang You , Lingfeng Liu , Jiayao Tang , Guanqun Zhao
show 2 more authors
Aoqi Hu Zeyu Chen
This is my paper
classification cs.LG cs.AI
keywords agentic reinforcement learninglong-horizon language agentscontext managementcredit assignmentsource-indexed memoryselective turn memorytool-use agentsprovenance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ECHO is a training framework for long-horizon language agents that tries to solve a mismatch: agents act on bounded, reconstructed context, but outcome-based reinforcement learning has no way to know which earlier tool observations caused success. The paper's central idea is to keep each completed turn addressable as a compact memory record with a source index, reconstruct the policy's context by selecting useful records, and reuse those same selected indices to route positive outcome credit to the evidence that was actually reused. On BrowseComp-Plus, this reaches 43.4% held-out accuracy versus 28.9% for GRPO and 36.1% for a rolling-summary baseline SUPO, while using fewer turns and lower trajectory volume. The authors argue this shows longer search should be made useful through provenance-preserving context management, not merely longer through summarization.

Core claim

The discovery is that source-indexed reconstruction can serve as the shared trace for both acting and learning. ECHO writes for each completed tool turn a compact finding plus a source-turn pointer; when the context budget binds, the policy outputs a selection of memory records, and the bounded context is rendered from the selected records plus recent turns. The same source indices define a hard mask that attaches the positive group-relative advantage only to final-answer tokens, action tokens from selected source turns, their last-turn findings, and memory-selection spans. The result is that outcome rewards reinforce reused evidence paths while screening off redundant search, and the traine

What carries the argument

The central mechanism is source-indexed turn memory plus provenance-guided credit routing. Each completed turn is compressed into a record e_i = (i, α_i, m_i) holding the source-turn index, a compact rendering of the action, and a last-turn finding; the memory stays a non-collapsing set rather than a folded summary. At compression boundaries, the policy autoregressively selects useful source indices, and the bounded policy context is reconstructed from those selected records. The same selected source indices form the credit mask that routes the positive group-relative advantage to the final segment, selected evidence turns, their findings, and selection actions. Positive-part routing (max(A,

Load-bearing premise

The load-bearing premise is that the turns selected into the final reconstructed context are a sufficient proxy for the causal evidence behind success, with positive-part routing further assuming that mixed-outcome reward groups supply the learning signal.

What would settle it

Train ECHO on a synthetic long-horizon task where the only successful path depends on an early turn that is never selected in the final reconstructed context; if accuracy stays high, the final-trace proxy is sufficient, and if it falls, under-crediting of deep dependency chains is real. Alternatively, record the distribution of group outcomes: if all-correct and all-incorrect groups dominate, the positive-part routing should produce near-zero updates and training should stall.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • ECHO reaches 43.4% held-out accuracy on BrowseComp-Plus versus 28.9% for GRPO and 36.1% for SUPO, with fewer turns (45.3 vs. 62.5) and lower trajectory volume (3.13 vs. 4.18) than SUPO.
  • Ablations show both learned source selection and traceable credit routing are necessary: replacing learned selection with semantic top-k retrieval lowers accuracy, and removing the credit mask reduces both accuracy and stability.
  • The trained policy transfers zero-shot to multi-objective QA, code generation, and deep-information benchmarks, improving average scores from 33.6% (GRPO) and 34.8% (SUPO) to 40.2% on the dense backbone.
  • The same mechanisms transfer to a sparse MoE backbone, where ECHO reaches about 35.0% accuracy on BrowseComp-Plus while the rolling-summary baseline collapses by step 50.
  • If the central claim holds, long-horizon agent RL can be made more accurate without rewarding longer and more redundant rollouts, because credit is attached to reused evidence rather than to all generated tokens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the final-trace approximation makes ECHO's advantage task-dependent; success on tasks with deep nested dependencies should show smaller gains than on shallow-evidence tasks, which can be tested by varying dependency depth while holding horizon fixed.
  • Inference: because positive-part routing zeros updates when all rollouts in a group succeed or fail, ECHO's learning signal is carried entirely by mixed-outcome groups of size 8; training on verifiers that produce mostly homogeneous groups should stall, and one could control group composition to check this.
  • Inference: the (M, Φ) interface implies that any context manager exposing provenance metadata could adopt the same traceable credit routing, not only selective turn memory; one could apply the mask idea to truncation with saved source indices or to retrieval-based agents.
  • Inference: the last-turn-finding compression suggests that the memory record, not the raw observation, is what matters for selection; this points toward learned or lossy memory representations and implies a testable sensitivity to summary quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ECHO, a context-management framework for multi-turn RL agents. ECHO stores each completed tool-use turn as a compact, source-indexed memory record, reconstructs bounded policy contexts by having the policy select useful memory records, and reuses the selected source indices as a token-level mask for routing positive outcome advantages to final-answer tokens, selected historical source turns, their memory findings, and memory-selection actions. The authors formalize this through an (M, Φ) context-managed RL interface that separates history state, bounded-context reconstruction, and source traceability. They evaluate ECHO on BrowseComp-Plus with Qwen3-32B-Instruct and Qwen3-30B-A3B-Instruct backbones, reporting 43.4% held-out accuracy versus 28.9% for GRPO and 36.1% for SUPO, while also reporting zero-shot transfer to multi-objective QA, code generation, and deep information-seeking benchmarks.

Significance. If the empirical claims are reliable, the paper makes a useful conceptual contribution: it identifies a concrete interaction between context management and credit assignment, and proposes a simple, reproducible mechanism—source indices as provenance—that is cleanly integrated into both acting and learning. Strengths include evaluation against external verifier-based benchmarks, ablations that attempt to isolate learned source selection and traceable credit routing, and an honest statement of the final-trace approximation in Section 7. The central limitation is statistical: the primary comparison on BrowseComp-Plus rests on an 83-example held-out set with no seeds or confidence intervals, and the key ECHO-vs-SUPO gap is within sampling noise. The conceptual framework is nevertheless solid enough to merit revision rather than rejection.

major comments (3)
  1. [§5.1, §5.2, Figure 1] The headline claim that ECHO 'outperforms SUPO' is not statistically supported. The held-out set has 83 examples: ECHO at 43.4% corresponds to 36 successes, SUPO at 36.1% to 30 successes. The 7.3-point gap is about 6 examples; the standard error of the difference is roughly 7.6 percentage points, so the 95% confidence interval includes zero. The ECHO-vs-GRPO gap is larger but still borderline, and no seeds, per-example scores, or confidence intervals are reported anywhere. Because Section 5.3's ablations attribute the gain to specific components, this statistical weakness is load-bearing. Please provide multiple seeds, bootstrap confidence intervals, and ideally a larger held-out evaluation; without this, the central empirical claim is not established.
  2. [§4.3, Eq. (14)] The positive-part routing A_+ = max(A, 0) combined with a group-relative advantage over N=8 rollouts and binary verifier rewards means that all-correct or all-incorrect groups produce zero traceable update. Learning therefore depends entirely on mixed-outcome groups. The paper does not report how often such groups occur on BrowseComp-Plus, nor how this fraction evolves during training. If mixed groups are rare, the effective gradient signal is small; if they become rare as the policy improves, training may stall. Additionally, because only positive-advantage rollouts receive updates, the method never directly penalizes long or redundant rollouts except through the group baseline. Please report the distribution of group outcome types, the fraction of mixed groups over training, and sensitivity to group size or to an outcome-based non-group-relative advantage.
  3. [§4.3, Eq. (13); §7] The final-trace approximation is acknowledged in the limitations, but it is load-bearing for the paper's long-horizon motivation. Eq. (13) routes credit only to source turns selected into the final reconstructed context, so earlier turns that enabled those selected turns but were not themselves selected receive no credit. In tasks with deep dependency chains, this systematically under-covers the true causal history, and the method's advantage over dense credit assignment could shrink in exactly the regime the introduction targets. The current experiments do not isolate this failure mode: there is no evaluation stratified by dependency depth, and no comparison against a recursively traced variant or a dense variant on a task where success depends on unselected early evidence. A diagnostic—for example, controlled chain-depth tasks or a recursive-trace variant—would clarify where the approx
minor comments (5)
  1. [Table 2] The column header 'A vg.' should be 'Avg.'.
  2. [§5.2] The term 'trajectory split rate' (85.5% for SUPO, 57.8% for ECHO) is used without definition in the main text. Please define it explicitly, or move the definition from the appendix into Section 5.1.
  3. [Figure 4] The ablation plots have no numeric labels or final values. Since the text says 'substantially lower' and 'reduces accuracy and stability', include final accuracy numbers or a small table for the ablations, ideally with the same uncertainty information requested above.
  4. [Appendix C.2] The CodeGym medium-difficulty filter selects instances solved exactly once by the base policy before evaluating any trained method. This is a reasonable control, but it should be stated in the main text that the subset is fixed in advance and is not selected based on trained-policy performance.
  5. [Eq. (10)] The repair operator ρ_{B,S,K} is described only in prose. A short formal definition would make the reconstruction and credit-mask logic easier to verify.

Circularity Check

0 steps flagged

No significant circularity: ECHO's equations define a constructive method, and all headline claims are measured against external benchmarks with an external verifier.

full rationale

ECHO is an empirical method paper rather than a first-principles derivation. The core equations (8)-(15) define a rollout procedure and a token-level credit mask: Eq. (13) marks final-segment tokens, selected source-turn tokens, their finding tokens, and memory-selection tokens; Eq. (14) restricts updates to positive-advantage rollouts. These masks depend on the policy's own selections, but that is a training-loop property, not a reduction of a prediction to its input: no fitted parameter encodes the 43.4% held-out accuracy or any benchmark number. The headline results come from held-out evaluation on BrowseComp-Plus with an external verifier, and the zero-shot numbers in Table 2 are measured on external benchmarks (GAIA, HLE, Frames, CodeGym, LoCoBench). The reference list contains no prior work by the present authors, so there is no self-citation chain on which the argument rests; GRPO and SUPO are external baselines. The §7 final-trace limitation is an honest coverage caveat, not a circular premise, and the absence of seeds/error bars is a statistical-reliability issue, not a circularity issue. No specific reduction (Eq. X = Eq. Y by construction, or fitted parameter renamed as prediction) can be exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The method is empirical and builds on standard RL; the main hand-set knobs are S=8 and K=3 with no sensitivity analysis, and the credit design rests on two paper-specific assumptions (final-trace proxy and positive-part routing) that the authors themselves flag as approximations. No exotic entities are postulated beyond an inspectable data structure.

free parameters (3)
  • selection cap S = 8 (hand-chosen; no sensitivity analysis)
    Max number of model-selected historical turns; set in §5.1 and Appendix D.2 ('selection max turns = 8'); no ablation over S.
  • recent-turn retention K = 3 (hand-chosen)
    Latest turns retained automatically (§5.1 'retains the latest 3 turns automatically'); no ablation over K.
  • CodeGym medium-difficulty filter = 'solved exactly once' subset, 128 examples
    Evaluation-set construction choice (§C.2) that determines the code benchmark; fixed before evaluating any method but affects what the zero-shot code result means.
axioms (5)
  • standard math Policy-gradient (REINFORCE/GRPO) objective with group-relative advantage (Eqs. 2–3, 15) is a valid learning signal for multi-turn agents
    Background framework from Schulman et al. 2017 / Shao et al. 2024; the paper suppresses clipping/KL terms and assumes this skeleton suffices.
  • domain assumption The BrowseComp-Plus binary verifier reward is an accurate success signal for tool-use quality
    Used as the sole outcome reward in §5.1; if the verifier is noisy or gameable, all compared methods' rankings could change.
  • ad hoc to paper Final-trace approximation: turns selected into the final reconstructed context are a sufficient proxy for the evidence that caused success
    Eq. 13 defines the credit mask via I_src from the final context only; §7 explicitly acknowledges it 'does not recursively follow the full dependency chain.'
  • ad hoc to paper Positive-part routing A_+ = max(A,0) yields useful learning dynamics
    Eq. 14 zeroes traceable credit for below-baseline and tied rollouts (including all-correct/all-incorrect groups of size 8); no learning-dynamics analysis of this sparsity.
  • domain assumption Self-generated last-turn findings and selection indices are reliable enough to serve as both context and credit trace
    Algorithm 1 and Appendix D.2 rely on parse() and repair operator ρ; if the base policy's summaries/selections are noisy, the credit mask inherits the noise.
invented entities (1)
  • source-indexed turn-memory record e_i = (i, alpha_i, m_i) independent evidence
    purpose: Keeps each completed turn addressable after it leaves the context window, enabling selection-based reconstruction and credit routing (Eq. 9).
    Not a physical entity; a software data structure. Independently inspectable in the claimed GitHub repo and via the source-index masks it produces, which are falsifiable in code.

pith-pipeline@v1.3.0-alltime-deepseek · 106 in / 20747 out tokens · 156147 ms · 2026-08-04T04:32:15.743621+00:00 · methodology

0 comments
read the original abstract

Long-horizon language agents must repeatedly interact with tools, accumulate evidence, and make decisions under bounded context windows. Context-management methods make such rollouts feasible by simplifying past interactions through deletion, folding, or memory editing. However, when useful history is collapsed into compressed states, the reconstructed context may no longer reveal which earlier observations support a successful final answer. This creates a mismatch between bounded-context acting and outcome-based reinforcement learning: the policy acts on reconstructed context, while the learner lacks source-level provenance for assigning credit to the evidence that mattered. We propose ECHO, a selective turn-memory framework for traceable context reconstruction in Agentic RL. ECHO compresses each completed environment turn into a compact source-indexed memory record, reconstructs bounded policy contexts by selecting useful records, and reuses the selected source indices to route positive outcome credit to the final trajectory segment, reused evidence turns, memory findings, and memory-selection actions. On BrowseComp-Plus, ECHO reaches 43.4% held-out accuracy, outperforming GRPO at 28.9% and the rolling-summary baseline SUPO at 36.1%, while using fewer turns and lower trajectory volume than SUPO. The trained policy also improves zero-shot generalization across multi-objective QA, code generation, and deep information-seeking benchmarks on both dense and MoE backbones.

Figures

Figures reproduced from arXiv: 2606.31650 by Aoqi Hu, Binbin Zheng, Enlei Gong, Guanqun Zhao, Jiayao Tang, Jihua Liu, Lingfeng Liu, Yuyang You, Zeyu Chen, Zijun Xie.

Figure 1
Figure 1. Figure 1: Held-out accuracy, tool-use turns per rollout, and trajectory volume over training on BrowseComp-Plus [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training diagnostics on long-horizon search. Summarization-based context management enables longer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of ECHO. ECHO stores completed turns as source-indexed memories, selects useful memo￾ries for bounded context reconstruction, and reuses the same source trace for credit assignment. Autoregressive memory selection. Let S cap model-selected turns and K denote the latest turns retained au￾tomatically. At a compression boundary before segment j, let Hbd j be the bounded local state available there. E… view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study on BrowseComp-Plus. (a) Learned source selection in ECHO outperforms semantic [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics on BrowseComp-Plus with the Qwen3-30B-A3B-Instruct MoE backbone. SUPO is [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Existing context reconstruction strategies reduce context length but often lose full traceability. ECHO [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

Review history (5 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Group-Reflective Self-Distillation for Agentic Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.0

    Contrasting a policy’s own success and failure reflections yields turn-level credit that beats GRPO and skill-based self-distillation on agentic tasks.

  2. Group-Reflective Self-Distillation for Agentic Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.0

    GRSD uses a policy's own contrast of success/failure reflections to convert trajectory-level rewards into turn-level credit for agentic RL.

Reference graph

Works this paper leans on

47 extracted references · 32 linked inside Pith · cited by 1 Pith paper

  1. [1]

    BrowseComp-Plus: A more fair and trans- parent evaluation benchmark of deep-research agent.arXiv preprint arXiv:2508.06600,

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. BrowseComp-Plus: A more fair and trans- parent evaluation benchmark of deep-research agent.arXiv...

    Pith/arXiv arXiv

  2. [3]

    Generalizable end-to-end tool-use RL with synthetic CodeGym.arXiv preprint arXiv:2509.17325,

    Weihua Du, Hailei Gong, Zhan Ling, Kang Liu, Lingfeng Shen, Xuesong Yao, Yufei Xu, Dingyuan Shi, Yim- ing Yang, and Jiecao Chen. Generalizable end-to-end tool-use RL with synthetic CodeGym.arXiv preprint arXiv:2509.17325,

    arXiv

  3. [4]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  4. [8]

    Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan

    Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. Acon: Optimizing context compression for long-horizon llm agents.arXiv preprint arXiv:2510.00615,

    Pith/arXiv arXiv

  5. [9]

    Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. InPro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologie...

    2025

  6. [10]

    MemPO: Self-memory policy optimization for long-horizon agents.arXiv preprint arXiv:2603.00680,

    Ruoran Li, Xinghua Zhang, Haiyang Yu, Shitong Duan, Xiang Li, Wenxin Xiang, Chonghua Liao, Xudong Guo, Yongbin Li, and Jinli Suo. MemPO: Self-memory policy optimization for long-horizon agents.arXiv preprint arXiv:2603.00680,

    Pith/arXiv arXiv

  7. [11]

    ToRL: Scaling tool-integrated RL.arXiv preprint arXiv:2503.23383,

    10 Xuefeng Li, Haoyang Zou, and Pengfei Liu. ToRL: Scaling tool-integrated RL.arXiv preprint arXiv:2503.23383,

    Pith/arXiv arXiv

  8. [12]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  9. [14]

    Scaling LLM multi-turn RL with end-to-end summarization-based context management.arXiv preprint arXiv:2510.06727,

    Miao Lu, Weiwei Sun, Weihua Du, Zhan Ling, Xuesong Yao, Kang Liu, and Jiecao Chen. Scaling LLM multi-turn RL with end-to-end summarization-based context management.arXiv preprint arXiv:2510.06727,

    arXiv

  10. [15]

    Gaia: a benchmark for general ai assistants

    Gr´egoire Mialon, Cl´ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InInternational Conference on Learning Representations, volume 2024, pp. 9025–9049,

    2024

  11. [16]

    Agent-omit: Adaptive context omission for efficient llm agents.arXiv preprint arXiv:2602.04284,

    Yansong Ning, Jun Fang, Naiqiang Tan, and Hao Liu. Agent-omit: Adaptive context omission for efficient llm agents.arXiv preprint arXiv:2602.04284,

    Pith/arXiv arXiv

  12. [17]

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G

    doi: 10.48550/arXiv.2602.04284. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems.arXiv preprint arXiv:2310.08560,

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.04284

  13. [18]

    Hiper: Hierarchical reinforcement learning with explicit credit assignment for large language model agents

    Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming, Zhaoran Wang, Alfredo Garcia, and Mingyi Hong. Hiper: Hierarchical reinforcement learning with explicit credit assignment for large language model agents. arXiv preprint arXiv:2602.16165,

    Pith/arXiv arXiv

  14. [19]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Pith/arXiv arXiv

  15. [20]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687–5711,

    2023

  16. [21]

    ToolRL: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958,

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T ¨ur, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958,

    Pith/arXiv arXiv

  17. [22]

    WebResearcher: Unleashing unbounded reasoning capability in long-horizon agents.arXiv preprint arXiv:2509.13309,

    Zile Qiao, Guoxin Chen, Xuanzhong Chen, Donglei Yu, Wenbiao Yin, Xinyu Wang, Zhen Zhang, Baixuan Li, Huifeng Yin, Kuan Li, et al. WebResearcher: Unleashing unbounded reasoning capability in long-horizon agents.arXiv preprint arXiv:2509.13309,

    arXiv

  18. [23]

    Locobench-agent: An interactive benchmark for llm agents in long-context software engineering.arXiv preprint arXiv:2511.13998,

    Jielin Qiu, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Jianguo Zhang, Haolin Chen, Shiyu Wang, Ming Zhu, Liang- wei Yang, Juntao Tan, et al. Locobench-agent: An interactive benchmark for llm agents in long-context software engineering.arXiv preprint arXiv:2511.13998,

    arXiv

  19. [24]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  20. [26]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  21. [27]

    QwenLong-CPRS: Towards∞-LLMs with dynamic context optimization.arXiv preprint arXiv:2505.18092,

    Weizhou Shen, Chenliang Li, Fanqi Wan, Shengyi Liao, Shaopeng Lai, Bo Zhang, Yingcheng Shi, Yuning Wu, Gang Fu, Zhansheng Li, et al. QwenLong-CPRS: Towards∞-LLMs with dynamic context optimization.arXiv preprint arXiv:2505.18092,

    Pith/arXiv arXiv

  22. [28]

    R1-Searcher: Incentivizing the search capability in LLMs via reinforcement learning.arXiv preprint arXiv:2503.05592,

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-Searcher: Incentivizing the search capability in LLMs via reinforcement learning.arXiv preprint arXiv:2503.05592,

    Pith/arXiv arXiv

  23. [31]

    MIRIX: Multi-agent memory system for LLM-based agents.arXiv preprint arXiv:2507.07957,

    Yu Wang and Xi Chen. MIRIX: Multi-agent memory system for LLM-based agents.arXiv preprint arXiv:2507.07957,

    Pith/arXiv arXiv

  24. [32]

    Milestone-guided policy learning for long-horizon language agents.arXiv preprint arXiv:2605.06078,

    Zixuan Wang, Yuchen Yan, Hongxing Li, Teng Pan, Dingming Li, Ruiqing Zhang, Weiming Lu, Jun Xiao, Yuet- ing Zhuang, and Yongliang Shen. Milestone-guided policy learning for long-horizon language agents.arXiv preprint arXiv:2605.06078,

    Pith/arXiv arXiv

  25. [33]

    BrowseComp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516,

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516,

    Pith/arXiv arXiv

  26. [35]

    Con- textweaver: Selective and dependency-structured memory construction for llm agents.arXiv preprint arXiv:2604.23069,

    Yating Wu, Yuhao Zhang, Sayan Ghosh, Sourya Basu, Anoop Deoras, Jun Huan, and Gaurav Gupta. Con- textweaver: Selective and dependency-structured memory construction for llm agents.arXiv preprint arXiv:2604.23069,

    Pith/arXiv arXiv

  27. [36]

    A-MEM: Agentic memory for LLM agents.arXiv preprint arXiv:2502.12110,

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic memory for LLM agents.arXiv preprint arXiv:2502.12110,

    Pith/arXiv arXiv

  28. [37]

    Concise and precise context compression for tool-using language models

    Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, et al. Concise and precise context compression for tool-using language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 16430–16441,

    2024

  29. [38]

    Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828,

    Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Sch¨utze, V olker Tresp, and Yunpu Ma. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828,

    Pith/arXiv arXiv

  30. [39]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380,

    2018

  31. [40]

    DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  32. [41]

    From reasoning to agentic: Credit assignment in reinforcement learning for large language models.arXiv preprint arXiv:2604.09459,

    12 Chenchen Zhang. From reasoning to agentic: Credit assignment in reinforcement learning for large language models.arXiv preprint arXiv:2604.09459,

    Pith/arXiv arXiv

  33. [42]

    Memory as action: Au- tonomous context curation for long-horizon agentic tasks.arXiv preprint arXiv:2510.12635,

    Yuxiang Zhang, Jiangming Shu, Ye Ma, Xueyuan Lin, Shangxi Wu, and Jitao Sang. Memory as action: Au- tonomous context curation for long-horizon agentic tasks.arXiv preprint arXiv:2510.12635,

    Pith/arXiv arXiv

  34. [43]

    Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, and Jianmin Wu

    Haotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S.-T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, and Jianmin Wu. AEM: Adaptive entropy modulation for multi-turn agentic reinforcement learning.arXiv preprint arXiv:2605.00425,

    Pith/arXiv arXiv

  35. [44]

    Deep- Researcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160,

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deep- Researcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160,

    Pith/arXiv arXiv

  36. [45]

    MEM1: Learning to synergize memory and reasoning for efficient long-horizon agents

    Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. MEM1: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841,

    Pith/arXiv arXiv

  37. [46]

    The top pipeline shows the rollout loop, where the policy generates tool calls, receives observations, and triggers reconstruction once history exceeds the budget

    13 A CONTEXTRECONSTRUCTIONSTRATEGIES Figure 6 compares context reconstruction strategies under bounded-context agentic RL. The top pipeline shows the rollout loop, where the policy generates tool calls, receives observations, and triggers reconstruction once history exceeds the budget. The lower panels show how different methods reconstruct the next polic...

    2025

  38. [47]

    CodeGym frames tasks as synthetic interactive environments where agents invoke problem-specific APIs, together withobserve()anddone(), rather than writing raw code

    and LoCoBench-Agent (Qiu et al., 2025). CodeGym frames tasks as synthetic interactive environments where agents invoke problem-specific APIs, together withobserve()anddone(), rather than writing raw code. Since many CodeGym tasks are either too easy or too difficult for meaningful comparison, we construct a medium- difficulty subset using the originalQwen...

    2025

  39. [1992]

    Resum: Unlocking long-horizon search intelligence via context summarization.arXiv preprint arXiv:2509.13313,

    Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou. Resum: Unlocking long-horizon search intelligence via context summarization.arXiv preprint arXiv:2509.13313,

    arXiv

  40. [2017]

    Cognitive memory in large language models

    Lianlei Shan, Shixian Luo, Zezhou Zhu, Yu Yuan, and Yong Wu. Cognitive memory in large language models. arXiv preprint arXiv:2504.02441,

    Pith/arXiv arXiv

  41. [2018]

    Hindsight credit assignment for long-horizon llm agents.arXiv preprint arXiv:2603.08754,

    11 Hui-Ze Tan, Xiao-Wen Yang, Hao Chen, Jie-Jing Shao, Yi Wen, Yuteng Shen, Weihong Luo, Xiku Du, Lan-Zhe Guo, and Yu-Feng Li. Hindsight credit assignment for long-horizon llm agents.arXiv preprint arXiv:2603.08754,

    arXiv

  42. [2020]

    Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model.arXiv preprint arXiv:2408.09559,

    Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model.arXiv preprint arXiv:2408.09559,

    Pith/arXiv arXiv

  43. [2022]

    In-context former: Lightning-fast compressing context for large language model

    Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, and Enhong Chen. In-context former: Lightning-fast compressing context for large language model. InFindings of the Association for Computa- tional Linguistics: EMNLP 2024,

    2024

  44. [2023]

    Prompt compression for large language models: A survey.arXiv preprint arXiv:2410.12388,

    Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey.arXiv preprint arXiv:2410.12388,

    Pith/arXiv arXiv

  45. [2024]

    SAM: State-adaptive memory for long-horizon reasoning agent.arXiv preprint arXiv:2605.24468,

    Yuyang Hu, Hongjin Qian, Shuting Wang, Jiongnan Liu, Ziliang Zhao, Jiejun Tan, Zheng Liu, and Zhicheng Dou. SAM: State-adaptive memory for long-horizon reasoning agent.arXiv preprint arXiv:2605.24468,

    Pith/arXiv arXiv

  46. [2025]

    Mem0: Building production- ready AI agents with scalable long-term memory.arXiv preprint arXiv:2504.19413,

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production- ready AI agents with scalable long-term memory.arXiv preprint arXiv:2504.19413,

    Pith/arXiv arXiv

  47. [2026]

    Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Wang Dong, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Pith/arXiv arXiv