Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

This paper claims that treating the runtime harness around an AI agent as a composable, trace-evolvable object improves agent success by 14.5 percentage points on average (up to 44), with the largest gains going to the weakest models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:29 UTC pith:PBTVU63K

load-bearing objection Useful systems paper with a sound compositional substrate, but the headline +14.5% is peak-score on the evolution set and should be read as an upper bound until held-out evaluation lands. the 4 major comments →

arxiv 2606.14249 v3 pith:PBTVU63K submitted 2026-06-12 cs.AI

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

Tingyang Chen , Shuo Lu , Kang Zhao , Weicheng Meng , Hanlin Teng , Tianhao Li , Chao Li , Xule Liu
show 6 more authors
Jian Liang Zhizhong Zhang Yuan Xie Heng Qu Kun Shao Jian Luan
This is my paper
classification cs.AI
keywords agent harnessharness compositionself-evolving agentstrace-driven evolutioncross-harness GRPOoperational mirrorvariant isolationAI agent benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that an AI agent's runtime harness—the prompts, tools, memory, and control flow wrapped around a model—is not a fixed scaffold but a first-class object that can be composed, adapted, and evolved from execution traces. It builds a typed substitution algebra over hook-attached processors, an evolution engine (AEGIS) that treats harness edits as actions in a symbolic MDP with a deterministic safety gate, and a co-evolution loop in which the same trajectories drive both harness edits and model training. On five benchmarks spanning embodied, web, retrieval, dialogue, and coding tasks, harness evolution improves 14 of 15 model–benchmark configurations by an average of +14.5 percentage points, up to +44.0 for the weakest agent; interleaving model training adds another +4.7 points. A sympathetic reader would take away that agent progress need not come from model scaling alone: improving the runtime interface from execution feedback is an actionable, complementary lever, especially for capability-limited models.

Core claim

The central claim is that the harness is the bottleneck that model scaling leaves untouched, and that it can be optimized like a program rather than hand-written. Concretely, the paper defines a harness as a typed configuration H = (M, C), decomposes C into nine behavioral dimensions instantiated as processors attached to eight lifecycle hooks, and shows that edits to these processors—insertion, replacement, removal—preserve type safety. AEGIS then evolves H over rounds: a Digester compresses raw traces, a Planner builds an adaptation landscape, an Evolver proposes typed candidate edits with change manifests, and a Critic plus deterministic gate rejects regressions, mirroring RL's policy/sta

What carries the argument

The load-bearing object is the harness configuration C=(P,S): a hook-indexed list of typed processors with a shared slot-resource set, composed through a substitution algebra that makes every edit type-safe. Around it, the operational mirror—an explicit mapping from RL concepts (policy, state, action, feedback, update) to symbolic-space duals (harness-update procedure, configuration-plus-trace store, typed edit, verifier score, acceptance gate)—turns known RL pathologies into design defenses: the Critic against reward hacking, the deterministic seesaw gate against catastrophic forgetting, the Planner against under-exploration. The co-evolution loop's key mechanism is cross-harness GRPO on a

Load-bearing premise

The headline numbers are computed on the same task set used for evolution and reported at the best evolution round, so if optimizing on the evaluation set inflates those numbers through selection, the average +14.5-point gain is an upper bound, not a measure of held-out improvement.

What would settle it

Compute the peak-round evolved harness from a run, then score it on a fresh set of tasks sampled from the same benchmark distribution that never appeared in the evolution batch; if the average gain over the static baseline is not clearly positive—or is far below +14.5 points—the reported gains are overfitting to the adaptation set. A cheaper falsifier is to compare final-round accuracy, not peak, against the static baseline on the same 15 configurations.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fixed-model agents can be improved substantially by evolving their harness alone; gain size tracks baseline weakness, so harness evolution most helps small or weak agents, narrowing the gap to strong closed models.
  • Harness evolution is amenable to audit and control: every shipped edit carries a change manifest, is gated by a deterministic regression check, and is trace-attributable, so the failure modes of self-improvement (reward hacking, forgetting, under-exploration) are detectable in practice.
  • Compositional structure makes per-task harness configuration a substitution problem rather than a rewrite, which is what lets a variant-isolation strategy convert a stagnating single-harness run into a non-degrading +13.6-point gain on heterogeneous tasks (GAIA, GPT-5.4).
  • Interleaving harness evolution with model training over one shared replay buffer produces gains beyond either route alone, at no additional rollout cost, because the same trajectories feed both the harness edit and the GRPO update.
  • The evolved harness is deployable as a static artifact; on GAIA the upfront 107.8M-token evolution cost is amortized within roughly 1,300 task invocations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the inverse-scaling pattern generalizes, harness evolution is a low-cost complement to scaling for small open-weight models; the same machinery could be pointed at a deployment distribution rather than a benchmark, using held-out tasks as the gate.
  • The paper's own strictest test is missing: none of the gains is measured on tasks held out from evolution. A natural extension is to evolve on one benchmark split and evaluate on another, which would separate genuine harness improvement from adaptation-set overfitting.
  • The deterministic gate checks only binary per-task flips; an explicit tripwire on estimated success-probability drift could catch the sub-threshold coupling that produced the Telecom regression, suggesting a graded gate as a concrete upgrade.
  • Cross-harness grouping by task identity might also apply to other off-policy RL settings where the 'policy difference' comes from changing tools or interfaces rather than weights—e.g., browser-use and robotic-control agents with discrete action spaces.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces HarnessX, a framework for treating an agent's runtime harness as a composable, typed object and optimizing it automatically. HarnessX defines processors attached to lifecycle hooks, formalizes a nine-dimensional taxonomy of harness behavior, and proposes AEGIS, a trace-driven multi-agent evolution loop whose architecture is motivated by an 'operational mirror' between RL and symbolic adaptation. The paper also proposes harness-model co-evolution via cross-harness GRPO over a shared replay buffer. The headline empirical claim is that across five benchmarks and three task-agent models, HarnessX improves 14 of 15 configurations by an average of +14.5% (up to +44.0%), and that co-evolution adds +4.7%. The paper includes detailed failure case studies, per-round evolution trajectories, and a substantial appendix documenting prompts, manifests, and experimental infrastructure.

Significance. The work is potentially significant: harness optimization from execution traces is an underexplored lever, the typed-composition design is clean and enables useful ablations such as variant isolation, and the detailed trace artifacts and change-manifest schema are valuable for reproducibility and for studying failure modes. The paper is also unusually transparent about its limitations. However, the central quantitative claim is not currently supported as stated: the +14.5% average and +44.0% maximum are peak scores on the same task sets used for evolution, with no held-out evaluation and no confidence intervals. The stress-test concern in the review lands: the reported numbers are upper bounds that conflate genuine harness improvement with selection bias and potential overfitting to the evolution set. If the claims are reframed or the experiments augmented with held-out evaluation and final-round (or confidence-interval-bounded) metrics, the contribution would be solid; as presented, the significance is conditional.

major comments (4)
  1. [§6.1, §7.7, Table 4, Table 5] The abstract's +14.5% average and +44.0% maximum are computed from Table 4, where 'Evolved = peak accuracy achieved' on the same task set used for evolution. Section 6.1 explicitly states: 'All reported gains are measured on the same task set used for evolution; held-out generalization to unseen tasks is not evaluated in this work.' Section 7.7 concedes the numbers 'carry both selection bias and potential overfitting.' This is not merely a caveat: Table 5 shows that the Global strategy on GAIA/GPT-5.4 peaks at 73.8% (R4) and finishes at 49.5% (R5), yet Table 4 reports that cell as Δ=0.0 because it uses the peak. Similarly, SWE-bench GPT-5.4 peaks at 63.6% at R3 and degrades to 50.9% by R5, yet Table 4 reports a +18.2% gain. Reporting the best round from a 15-round trajectory on the evaluation set is equivalent to selecting the best epoch on the training set; the headline numbers are ther
  2. [Appendix D.5 vs. Table 4] There is an internal contradiction that highlights the danger of peak-based reporting. Table 4 reports Qwen3.5-9B on SWE-bench with a +18.2% gain (23.6% → 41.8%, best round 2), a configuration included in the headline average. Appendix D.5 states, for the same configuration, that 'every lever collapses to near-zero (prompt 0.05, config 0.05, processor 0.06)' and that the run 'yields only noise on Qwen3.5 (peak 42%, zero durable gains).' If the appendix is correct, the +18.2% cell is a noise-inflated peak, and the headline average materially overstates the result. This internal inconsistency must be resolved, either by reporting final/durable gains in Table 4 or by removing the 'zero durable gains' language from the appendix.
  3. [§6.5, Figure 5] The co-evolution claim ('co-evolution breaks the scaffolding ceiling' and '+4.7% over harness-only evolution') is subject to the same evaluation limitation. The GAIA and WebShop gains are measured on the same task sets used for evolution, and the text reports peak values (GAIA 37.4% → 41.7%, WebShop 49.0% → 54.0%) rather than final-round or held-out performance. The paper notes the gap persists to the final round, which is encouraging, but no uncertainty estimate is provided, and the selection of the best round over multiple evaluations on GAIA (each round evaluated twice and averaged) still biases the reported gain. A held-out split, or at least final-round values with confidence intervals, is needed before 'breaking the scaffolding ceiling' is supported.
  4. [Table 4, Table 8] Table 8 lists 'seeds random seeds per cell 3', yet Table 4 reports point estimates only, with no standard deviations, confidence intervals, or per-seed ranges across the 15 configurations. Given that the paper's central empirical claim is a numerical average of gains, and given the acknowledged noise (pass@2, small task counts, stochastic LLM behavior), the absence of any variance reporting is a serious gap. Section 6.4 even uses 'one standard error (~3.3% at n=103)' to argue that a 1.0% difference is not significant, implying standard errors exist; they should be reported for all main cells.
minor comments (5)
  1. [Abstract and §6.1] The abstract and introduction state the quantitative gains without the crucial qualifiers that the measurements are peak scores on the evolution set. Please add explicit wording such as 'peak accuracy on the evolution set' to the abstract, or better, replace the headline with a final-round or held-out number.
  2. [§7.3, §6.6] The paper claims the operational mirror's predicted pathologies are 'empirically confirmed' in Section 6.6, but Section 7.3 appropriately concedes that the mirror is 'a design checklist rather than a predictive theory.' The case studies are selected post hoc from runs whose pipeline already embeds defenses against these pathologies. I suggest softening 'confirmed' to 'illustrated' or 'observed,' and making clear that no counterfactual validation of the mirror's predictive power is attempted.
  3. [§6.4] The comparison between AEGIS and CC SDK is informative, but the conclusion that 'the four-stage decomposition does not improve final accuracy at this meta-agent capability level' is based on a single benchmark (GAIA, GPT-5.4) and a single run. Consider clarifying that this is a limited-scope ablation, not a general claim about the four-stage architecture.
  4. [Abstract, Future Work] The codebase is not yet released ('will be open-sourced in a future release'). The detailed artifact layout in Appendix E is valuable, but the inability to inspect the code behind the experiments limits reproducibility. Please state a concrete release plan or provide a public anonymized repository for review.
  5. [Figure 4 and Table 5] Figure 4's caption and inline text contain formatting artifacts (e.g., 'Best-so-farPeakIllustration') and the figure's relationship to Table 5's peak/final distinction is confusing. Please label the peak and final points clearly and explain why Table 4 reports peak while the figure shows trajectories.

Circularity Check

1 steps flagged

The headline +14.5% gain is a best-round in-sample score: Table 4's 'Evolved = peak accuracy achieved' on the evolution task set makes the central empirical claim a selected maximum, not a validated prediction.

specific steps
  1. fitted input called prediction [Section 6.1 (Scope), Table 4 (caption), Section 7.7 (Limitations), Table 5]
    ""All reported gains are measured on the same task set used for evolution; held-out generalization to unseen tasks is not evaluated in this work." Table 4: "Evolved = peak accuracy achieved." Section 7.7: "Since we report peak accuracy and evaluate on the adaptation set itself, the numbers carry both selection bias and potential overfitting." Table 5: "Global (single harness) Final (%) 49.5 Peak (%) 73.8 Final−Peak −24.3.""

    The headline gain is not an independent estimate: AEGIS optimizes harness edits against the very task set that Table 4 scores, and the table defines 'Evolved' as the best round on that set. Choosing the peak of an in-sample optimization trajectory makes a positive gain statistically expected whenever the trajectory has any noise or overfitting, so '+14.5% average, up to +44.0%' is a selected maximum, not a validated improvement. The reduction is explicit: Evolved_score = max_r accuracy_r(task_set), Initial_score = accuracy_0(task_set), and reported gain = Evolved_score − Initial_score. The paper's own Table 5 shows the Global GAIA/GPT-5.4 run peaking at 73.8% and finishing at 49.5% (−24.3), yet Table 4 reports this cell as Δ=0.0 because peak is used. The co-evolution +4.7% and inverse-scal

full rationale

The paper's composition formalism, AEGIS pipeline, and co-evolution equations are not circular: they are constructive design proposals with independent architectural content, and the RL-to-symbolic mapping is explicitly labeled a design heuristic rather than a formal derivation (Section 7.3). The central load-bearing quantitative claim, however, is the abstract's '+14.5% (up to +44.0%)' and Section 5's '+4.7%' co-evolution gain. These are computed on the same task set used for evolution and at the best evolution round, exactly as Table 4's caption 'Evolved = peak accuracy achieved' and Section 6.1's scope statement make explicit. The paper itself concedes in Section 7.7 that the numbers 'carry both selection bias and potential overfitting.' Selecting the peak of an in-sample optimization curve is a statistical selection effect, not a prediction of harness generalization; Table 5's Global strategy peaking at 73.8% and finishing at 49.5% while Table 4 reports Δ=0.0 is a concrete instance of the peak metric masking collapse. This makes the main empirical headline partially circular in the 'fitted input called prediction' sense, while the underlying system and case studies remain informative. No separate self-citation chain is load-bearing; the cited prior work by overlapping authors (MIA, Memento, K2-Agent) is contextual related work, not the basis of the headline result. Score 6 rather than 8 because the central claim is an empirical measurement with disclosed limitations rather than a definitional identity, and internal ablations (e.g., variant isolation's peak=final) provide some independent content.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 3 invented entities

This is an empirical systems paper, not a formal derivation. The central claim depends on hyperparameter choices (T, P, K_t, alpha, pass@2, GRPO settings, buffer size) and on assumptions about verifier reliability, pass@2 sensitivity, bounded off-policy bias, and meta-agent capability. The paper itself documents instances where pass@2 masked regressions and where verifier exploits occurred, so these assumptions are load-bearing rather than cosmetic.

free parameters (8)
  • Evolution rounds T = 15
    Chosen budget; headline gains are selected at the best round within this budget.
  • Early-stopping patience P = 3 consecutive no-op rounds
    Stops evolution after P rounds without a shipped edit; affects both final and peak trajectories.
  • Candidates per round K_t = 4
    Number of Evolver proposals per round; more candidates raise the chance of finding an improving edit.
  • Noise threshold alpha = ±5% pass-count delta
    Single-round pass-count deltas below 5% are ignored; the paper later shows this masks sub-threshold regressions.
  • Pass@2 attempts = 2 rollouts per task
    Metric choice reduces sampling noise but hides sub-threshold success-probability drift, as the paper acknowledges.
  • GRPO learning rate = 1e-6
    Co-evolution result depends on this choice; no sensitivity analysis is reported.
  • GRPO clip epsilon = 0.2
    Standard clipping value but chosen by hand; no ablation is reported.
  • Replay buffer window = 4 rounds (824 GAIA traces / 400 WebShop traces)
    The bounded off-policy bias claim relies on this window; no ablation is reported.
axioms (6)
  • domain assumption Evolution on the evaluation set is a valid proxy for deployment performance.
    Section 6.1 explicitly scopes results to the evolution set; Section 7.7 says held-out generalization is untested.
  • domain assumption Pass@2 binary per-task outcomes provide sufficient signal for the seesaw constraint.
    Sections 6.6 and 7.3 show that sub-threshold coupling can accumulate and regress tasks without triggering the deterministic gate.
  • domain assumption FIFO buffer bound keeps cross-harness GRPO off-policy bias small.
    Section 5.4 argues bounded lag; no direct empirical measurement of the bias magnitude is provided.
  • domain assumption The closed-source meta-agent (Claude Opus 4.6) is capable enough for AEGIS.
    Section 7.7 states open-weight meta-agents are untested; the results may not transfer to weaker evolvers.
  • domain assumption Fixed verifier rewards remain a reliable and comparable optimization target across harness versions.
    The paper's own case studies in Section 6.6 document verifier-exploit pathways, and the fixed-verifier assumption is central to Eq. 3.
  • domain assumption GRPO importance sampling and clipping remain valid when trajectories come from different harness contexts.
    Section 5.3 asserts task-level grouping avoids action-space alignment; no formal or empirical validation of the off-policy correction is provided.
invented entities (3)
  • AEGIS (Digester, Planner, Evolver, Critic) no independent evidence
    purpose: Trace-driven multi-agent engine for evolving harness configurations.
    Described in detail and evaluated in the paper, but no public code or artifacts are released, so there is no falsifiable handle outside the paper.
  • Operational mirror no independent evidence
    purpose: Maps RL concepts onto symbolic harness-editing failure modes.
    Section 7.3 explicitly calls it a design heuristic, not a formal framework; its predicted pathologies are reported retrospectively from the same system.
  • Scaffolding ceiling / training-signal ceiling no independent evidence
    purpose: Explain why co-evolution beats harness-only or model-only optimization.
    Introduced as conceptual limits; no direct measurement and the only support is the co-evolution experiment on two benchmarks.

pith-pipeline@v1.3.0-alltime-deepseek · 36712 in / 17600 out tokens · 180321 ms · 2026-08-02T11:29:38.682577+00:00 · methodology

0 comments
read the original abstract

AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. Project homepage: https://darwin-agent.github.io/HarnessX/.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

    cs.AI 2026-07 conditional novelty 6.0

    A behavior-centered handbook generated from agent-harness code helps LLM planners find the right edit sites and produce better edit plans than direct repository exploration.

  2. Mi-Memory: A Lifecycle Memory Framework for Personal AI

    cs.AI 2026-07 conditional novelty 5.0

    Mi-Memory proposes a four-role lifecycle framework for personal AI memory with an audit contract of typed evidence, traces, strategy artifacts, and rollback records; modules are benchmarked separately, but the contrac...

Reference graph

Works this paper leans on

66 extracted references · 17 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Langchain.https://github.com/langchain-ai/langchain, 2022

    2022

  2. [2]

    Claude code.https://github.com/anthropics/claude-code, 2025

    Anthropic. Claude code.https://github.com/anthropics/claude-code, 2025

    2025

  3. [3]

    Introducing dynamic workflows in claude code

    Anthropic. Introducing dynamic workflows in claude code. https://claude.com/blog/ introducing-dynamic-workflows-in-claude-code, 2026

    2026

  4. [4]

    Cursor.https://www.cursor.com, 2023

    Anysphere. Cursor.https://www.cursor.com, 2023

    2023

  5. [5]

    Deerflow.https://github.com/bytedance/deer-flow, 2025

    ByteDance. Deerflow.https://github.com/bytedance/deer-flow, 2025

    2025

  6. [6]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    2026

  7. [7]

    Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Prompt- breeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompt- breeder: Self-referential self-improvement via prompt evolution. InInternational Conference on Machine Learning, pages 13481–13544. PMLR, 2024

    2024

  9. [9]

    Glm-5: from vibe coding to agentic engineering, 2026

    GLM-5-Team. Glm-5: from vibe coding to agentic engineering, 2026. URLhttps://arxiv.org/abs/2602.15763

    Pith/arXiv arXiv 2026

  10. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, 2024

    2024

  12. [12]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. InInternational Conference on Learning Representations, 2025

    2025

  13. [13]

    Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, 2024

    2024

  14. [14]

    Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    2017

  16. [16]

    Exploration in deep reinforcement learning: A survey.Information Fusion, 85:1–22, 2022

    Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey.Information Fusion, 85:1–22, 2022

    2022

  17. [17]

    Langgraph.https://github.com/langchain-ai/langgraph, 2024

    LangChain AI. Langgraph.https://github.com/langchain-ai/langgraph, 2024

    2024

  18. [18]

    The Darwin Gödel Machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22535, 2025

    Robert Tjarko Lange, Yujin Tang, and Yingtao Tian. The Darwin Gödel Machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22535, 2025

    arXiv 2025

  19. [19]

    Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Agent harness engineering: A survey.arXiv preprint, 2026

    Junjie Li, Xi Xiao, Yunbei Zhang, Chen Liu, Lin Zhao, Xiaoying Liao, Yingrui Ji, Janet Wang, Jianyang Gu, Yingqiang Ge, et al. Agent harness engineering: A survey.arXiv preprint, 2026

    2026

  21. [21]

    Agentswift: Efficient llm agent design via value-guided hierarchical search

    Yu Li, Lehui Li, Zhihao Wu, Qingmin Liao, Jianye Hao, Kun Shao, and Fengli Xu. Agentswift: Efficient llm agent design via value-guided hierarchical search. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31843–31851, 2026. 24

    2026

  22. [22]

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Xuanjing Huang, Hang Yan, Zhenhua Han, and Tao Gui. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    LlamaIndex, 11 2022

    Jerry Liu. LlamaIndex, 11 2022. URLhttps://github.com/jerryjliu/llama_index

    2022

  24. [24]

    Openclaw research: A systematic survey of large language model agents in open deployment

    Shuo Lu, Kecheng Yu, Siru Jiang, Yinuo Xu, Bing Zhan, Yanbo Wang, Changxin Ke, Yuan Xu, Xin Xiong, Xinyun Zhou, et al. Openclaw research: A systematic survey of large language model agents in open deployment. 2026

    2026

  25. [25]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InInternational Conference on Learning Representations, 2024

    2024

  26. [26]

    Crewai: Framework for orchestrating role-playing autonomous ai agents, 2025

    João Moura. Crewai: Framework for orchestrating role-playing autonomous ai agents, 2025

    2025

  27. [27]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9340–9366, 2024

    2024

  28. [28]

    Memgpt: towards llms as operating systems

    Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: towards llms as operating systems. 2023

    2023

  29. [29]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 7957–7968, 2023

    2023

  30. [30]

    Memory intelligence agent.arXiv preprint arXiv:2604.04503, 2026

    Jingyang Qiao, Weicheng Meng, Yu Cheng, Zhihang Lin, Zhizhong Zhang, Xin Tan, Jingyu Gong, Kun Shao, and Yuan Xie. Memory intelligence agent.arXiv preprint arXiv:2604.04503, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    A self-improving coding agent.arXiv preprint arXiv:2504.15228, 2025

    Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent.arXiv preprint arXiv:2504.15228, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    2025

  33. [33]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    From mind to machine: The rise of manus ai as a fully autonomous digital agent.arXiv preprint arXiv:2505.02024, 2025

    Minjie Shen, Yanshu Li, Lulu Chen, Zhichao Fan, Yanhang Li, and Qikai Yang. From mind to machine: The rise of manus ai as a fully autonomous digital agent.arXiv preprint arXiv:2505.02024, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Pith/arXiv arXiv 2010

  36. [36]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Evoagentx: An automated framework for evolving agentic workflows

    Yingxu Wang, Siwei Liu, Jinyuan Fang, and Zaiqiao Meng. Evoagentx: An automated framework for evolving agentic workflows. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 643–655, 2025

    2025

  38. [38]

    Learning beyond gradients.https://trinkle23897.github.io/learning-beyond-gradients/, May

    Jiayi Weng. Learning beyond gradients.https://trinkle23897.github.io/learning-beyond-gradients/, May

  39. [39]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. InFirst conference on language modeling, 2024

    2024

  40. [40]

    Kˆ 2-agent: Co-evolving know-what and know-how for hierarchical mobile device control.arXiv preprint arXiv:2603.00676, 2026

    Zhe Wu, Donglin Mo, Hongjin Lu, Junliang Xing, Jianheng Liu, Yuheng Jing, Kai Li, Kun Shao, Jianye Hao, and Yuanchun Shi. Kˆ 2-agent: Co-evolving know-what and know-how for hierarchical mobile device control.arXiv preprint arXiv:2603.00676, 2026

    arXiv 2026

  41. [41]

    Adapting the interface, not the model: Runtime harness adaptation for deterministic llm agents.arXiv preprint arXiv:2605.22166, 2026

    Tianshi Xu, Huifeng Wen, and Meng Li. Adapting the interface, not the model: Runtime harness adaptation for deterministic llm agents.arXiv preprint arXiv:2605.22166, 2026. 25

    Pith/arXiv arXiv 2026

  42. [42]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. InInternational Conference on Learning Representations, 2024

    2024

  44. [44]

    Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  45. [45]

    tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text.arXiv preprint arXiv:2406.07496, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Hyperagents.arXiv preprint arXiv:2603.19461, 2026

    Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. Hyperagents.arXiv preprint arXiv:2603.19461, 2026

    arXiv 2026

  48. [48]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. InInternational Conference on Learning Representations, 2025

    2025

  49. [49]

    A2flow: Automating agentic workflow generation via self-adaptive abstraction operators

    Mingming Zhao, Xiaokang Wei, Yuanqi Shao, Kaiwen Zhou, Lin Yang, Siwei Rao, Junhui Zhan, and Zhitang Chen. A2flow: Automating agentic workflow generation via self-adaptive abstraction operators. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29930–29938, 2026

    2026

  50. [50]

    Memento: Fine-tuning llm agents without fine-tuning llms.arXiv preprint arXiv:2508.16153, 2025

    Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, et al. Memento: Fine-tuning llm agents without fine-tuning llms.arXiv preprint arXiv:2508.16153, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents

    Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Li Erran Li. Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents. InForty-second International Conference on Machine Learning, 2025

    2025

  52. [52]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  53. [53]

    Resmas: Resilience optimization in llm-based multi-agent systems

    Zhilun Zhou, Zihan Liu, Jiahe Liu, Qingyu Shao, Yihan Wang, Kun Shao, Depeng Jin, and Fengli Xu. Resmas: Resilience optimization in llm-based multi-agent systems. InProceedings of the AAAI Conference on Artificial Intelligence, 2026

    2026

  54. [54]

    Put a cooled apple in the microwave

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. InForty-first International Conference on Machine Learning, 2024. 26 Contributions and Acknowledgments Core Contributors •Tingyang Chen* •Shuo Lu* •Kang Zhao* •Weicheng Meng •Kun Shao † •Jian Luan † Contr...

    2024

  55. [55]

    Write the code to your scratch dir

  56. [56]

    Two levels: - Level 1 -- unit call works: instantiate the processor/tool, drive the async hook, assert the expected state mutation happened

    Verify by actually running it -- not by reasoning about it. Two levels: - Level 1 -- unit call works: instantiate the processor/tool, drive the async hook, assert the expected state mutation happened. - Level 2 -- round-trip reaches the model: a unit call that returns does not prove the agent sees the return. Simulate the path from your code to the model'...

  57. [57]

    Do NOT hide the failure in a try/except

    Iterate if verification fails -- fix the bug, or pivot if the environment does not support what you assumed. Do NOT hide the failure in a try/except

  58. [58]

    I believe this will work

    Attach the verifying output as`capability_evidence`. "I believe this will work" is not acceptable; paste the actual command and its output. A candidate whose new code has not been observed to work will burn a round's ship slot for zero flips. Pure prompt-bucket candidates (no code asset) are exempt -- the counterfactual gate provides the equivalent smoke ...

  59. [59]

    Decompose the goal into ordered sub-goals (locate -> acquire -> transform -> deliver) and complete each before moving on

  60. [60]

    Open closed containers before judging them empty -- the admissible list surfaces`open <recep>`when you arrive at a closed one

    Systematic exploration: search each surface and container at most once before revisiting. Open closed containers before judging them empty -- the admissible list surfaces`open <recep>`when you arrive at a closed one

  61. [61]

    Grab immediately: when a required object appears, take it on the very next step before moving elsewhere

  62. [62]

    Transform before placing: perform any clean/heat/cool state change at the appropriate appliance before heading to the final destination

  63. [63]

    Direct delivery: once holding the goal object, navigate straight to the target receptacle and place it

  64. [64]

    Only stop searching when the count reaches zero

    Track progress: keep an internal count of objects still to find and place. Only stop searching when the count reaches zero

  65. [65]

    If stuck, move to a different unexplored location

    Avoid loops: never repeat the same action more than twice in a row. If stuck, move to a different unexplored location

  66. [66]

    ", evidence:

    Trust the admissible list: if`take X from Y`does not appear, you are not at Y, Y is closed, or X is not visible --`go to`,`open`, or move on rather than guessing. ## Common Mistakes to Avoid - Revisiting searched locations without new evidence. - Ignoring visible objects -- if the target appears, take it immediately. - Skipping the state change -- do not ...