Pith. sign in

REVIEW 4 major objections 6 minor 76 references

LLM agents can improve during evaluation by evolving their executable control programs from unlabeled traces alone, without weight updates or gold labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 12:40 UTC pith:L4MWM4EA

load-bearing objection Useful methods paper: open-ended harness evolution at test time from unlabeled traces works on several domains, but transductive scoring and hard-slice construction leave the strongest claim only partly supported. the 4 major comments →

arxiv 2607.08124 v1 pith:L4MWM4EA submitted 2026-07-09 cs.SE cs.LG

TTHE: Test-Time Harness Evolution

Jun Nie , Yonggang Zhang , Jun Song , Qianshu Cai , Dahai Yu , Yike Guo , Xinmei Tian , Bo Han This is my paper
classification cs.SE cs.LG
keywords test-time adaptationLLM agentsharness evolutionexecutable control programsunlabeled execution tracesagentic systemsReActsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

An LLM agent’s behavior depends as much on its harness—the executable program that builds context, calls tools, checks intermediate results, and recovers from failures—as on the model itself. Most systems freeze that harness after development-time search, so they cannot adapt when test distributions, tools, or failure modes differ. This paper asks whether the harness can instead be optimized during evaluation using only the unlabeled execution traces the agent produces on the test inputs. It introduces Test-Time Harness Evolution: a population of candidate harnesses is refined by proposers that read those traces and execution-derived proxies, then a judge commits one program that persists to later inputs. Solver, proposers, and judge are different roles around the same frozen model, so all adaptation is a code change to the surrounding program. Across text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool-use, the committed harnesses beat fixed ReAct-style baselines and leave inspectable policies for grounding, verification, and repair. A reader who cares about deployable agents would care because this reframes test-time adaptation as evolution over control programs rather than weight updates, labeled development search, or per-query retries—and it surfaces proxy reliability as the central remaining obstacle.

Core claim

During evaluation, a population-based generate-and-judge loop can evolve a persistent executable harness from unlabeled execution traces and execution-derived proxies alone—without gold labels, weight updates, or a separate adaptation model—and the committed harness improves fixed ReAct-style baselines across text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool-use, yielding inspectable policies rather than a pre-searched workflow or one-off retries.

What carries the argument

Test-Time Harness Evolution (TTHE): within each unlabeled batch, G parallel harness branches evolve for R rounds; agentic proposers rewrite each branch’s executable code by reading detailed traces and proxy signals (execution health, round-trip consistency, public-test outcomes); an agentic judge commits one final branch that carries forward. All roles share one frozen LLM; gold is used only after commitment for measurement.

Load-bearing premise

The method assumes that imperfect execution proxies plus an agentic judge reading raw traces are reliable enough to commit harnesses that truly raise gold accuracy, rather than ones that game the proxies or overfit the current batch.

What would settle it

Run a prequential protocol: commit a harness on batch t using only label-free proxies, then score that same harness on batch t+1 before any further adaptation. If it fails to beat the fixed baseline on held-out batches under the same measurement, the reported gains are batch-local specialization rather than durable harness improvement.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents can adapt to new schemas, repositories, tools, and failure modes at deployment without a labeled development set tailored to that workload.
  • Grounding, verification, and recovery become persistent, inspectable source rather than one-off reflections or frozen development-time workflows.
  • Adaptation cost shifts from weight updates and labeled search to test-time execution and program search under incomplete proxies.
  • Proxy quality and judge selection become first-class research targets for unsupervised agent improvement.
  • The same loop can lift both code/query generation and multi-service tool-use agents, not only one domain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If selection regret and proxy gaming can be reduced, continuous self-maintenance of production agents on live traffic becomes a practical research target rather than a development-time craft.
  • The coverage gap—tasks never solved by any candidate—suggests complementary diversity mechanisms or hybrid external verifiers will be needed before this fully replaces human harness engineering.
  • Treating the harness as the adaptation state invites direct comparison of when program edits are more sample-efficient than parameter or prompt-only updates under distribution shift.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Test-Time Harness Evolution (TTHE): during evaluation, a population of executable agent harnesses is refined from unlabeled execution traces and execution-derived proxies (execution health, round-trip consistency, public-test pass rate), then a label-free agentic judge commits one harness that persists to subsequent inputs. Solver, proposers, and judge are roles around the same frozen LLM; gold is used only for post-selection measurement. On hard slices of BIRD, LiveCodeBench, SWE-bench Verified, DS-1000, and claw-eval, TTHE improves fixed ReAct-style baselines (e.g., BIRD 12%→50%, claw-eval 48.9%→69.8%) and yields inspectable grounding/verification/repair policies, with ablations on search budget, batch size, model, and accumulate-vs-reset, plus an audit of selection regret and coverage.

Significance. If the result holds under a stronger evaluation protocol, the contribution is substantial for agent systems: it reframes test-time adaptation as evolution of the executable control program rather than weights, prompts alone, or per-query retries, and does so without gold or a separate adaptation model. Strengths that should be credited include (i) clean isolation of gold from the adaptation loop, (ii) multi-domain empirical gains with inspectable evolved source (Figs. 4–5), (iii) useful ablations (Tables 1–3, Fig. 3) and unusually honest diagnostics of selection regret and coverage (§5.8), and (iv) public code. The work also usefully elevates proxy reliability as a first-class research problem for unsupervised agent improvement.

major comments (4)
  1. [§4.4, Algorithm 1, §6] §4.4, Algorithm 1, and §6: reported accuracy is transductive—each batch X_t is both the unlabeled evidence used to select H_{t+1} and the set on which that harness is scored after commitment (SCORE after JUDGE, no re-execution). The abstract and §5.4 present these numbers as test-time improvement of a persistent harness, but under this protocol gains can reflect batch-local specialization to X_t’s artifacts or proxy structure rather than a harness that improves subsequent unlabeled inputs. The paper correctly flags prequential scoring as future work; for the central claim as stated, at least the main tables (Fig. 2 / BIRD trajectory) need a prequential or held-out next-batch evaluation, or the claims must be narrowed to within-batch adaptation.
  2. [§5.1, Fig. 2] §5.1 and Fig. 2: the BIRD hard slice is built from questions the baseline repeatedly missed, so baseline accuracy is low by construction (12%). The headline +38 point gain therefore measures recovery on an adversarially filtered failure set, not improvement on a fixed, unconditioned evaluation distribution. The paper notes this, but the abstract and cross-domain bar chart still lead with that magnitude. Either report the same protocol on a standard unconditioned BIRD/Mini-Dev slice (or a slice selected without baseline outcomes), or demote the conditioned slice to a recovery analysis and lead with unconditioned numbers.
  3. [§5.2, §6] §5.2 and §6: the only systematic baseline is a fixed ReAct harness. TTHE spends a population of candidates over R rounds with re-execution and probing; without compute-matched controls (e.g., best-of-N / self-consistency under the same execution budget, or a development-time workflow optimizer frozen at test time with matched total calls), it is hard to attribute gains to harness evolution per se versus extra test-time compute and retries. The limitations section lists this as future work, but for a methods paper whose claim is the value of the full test-time procedure, at least one compute-matched comparison on the main slices is load-bearing.
  4. [§5.8, Eq. (2)] §5.8 and Eq. (2): the pool-oracle audit (judge 50% vs candidates-seen 64%, all-generated 70%) shows substantial selection regret and incomplete coverage under the same proxies the judge uses. That is a valuable finding, but it also means the unsupervised selection premise—that execution-derived proxies plus raw traces suffice to commit harnesses whose gold accuracy improves—is only partially supported. The manuscript should either (a) strengthen the judge (e.g., re-execution probes, multi-proxy aggregation, or an information ablation of scores vs full traces as suggested in §6) and re-report committed accuracy, or (b) report primary results as “best generated under label-free proxies” alongside committed accuracy so readers can separate generation from selection.
minor comments (6)
  1. [title page] Author line: “Y onggang Zhang” appears to contain a stray space/line-break artifact.
  2. [§5.7] Ablations in §5.7 are single completed runs with no multi-seed variance; even error bars from 2–3 seeds on the BIRD G/R and batch-size sweeps would make non-monotonic claims more credible.
  3. [Table 3] Table 3 (accumulate vs reset) is still scored transductively on each batch; if retained, clarify that it does not establish forward transfer of the committed harness.
  4. [§5.5, Appendix C.2] Appendix C.2: claw-eval batches are ordered by ascending baseline difficulty, which confounds accumulation; state this next to the +20.9 claim in the main text, not only in the appendix.
  5. [Fig. 2] Figure 2 mixes pass@1 (%) with claw-eval graded score ×100; the caption notes the metric change, but a second axis or separate panel would reduce misreading.
  6. [§2] Related work is thorough on harness optimizers and Meta-Harness; a short explicit comparison table (when gold is available, when the program is frozen, object of adaptation) would help readers place TTHE against DSPy/AFlow/MOSS/Meta-Harness at a glance.

Circularity Check

0 steps flagged

No significant circularity: empirical harness-evolution method with gold isolated from the adaptation loop; gains are measured, not derived by construction.

full rationale

TTHE is an empirical systems paper, not a first-principles derivation. The adaptation loop (Alg. 1; Eq. 2) selects harnesses from unlabeled traces and execution-derived proxies (execution health, round-trip consistency, public-test pass rate; §4.3), while gold enters only the post-selection measurement objective J⋆ (Eq. 1) and SCORE after JUDGE. That separation means reported accuracy is not forced by the selection signal: a harness can pass proxies and still fail gold, which the paper itself quantifies as selection regret (50% committed vs 64% pool oracle on BIRD; §5.8). Hard-slice construction for BIRD (baseline-conditioned failures; §5.1) and the transductive protocol (§4.4, §6) are methodological limitations that can inflate or localize measured gains, but they do not make the claimed improvements equivalent to the inputs by definition—gold is still an external check. Related-work citations (Meta-Harness, Self-Harness, MOSS, STOP, etc.) are used for positioning, not as load-bearing uniqueness theorems that force the method or the numbers. No self-definitional identity, fitted-parameter-as-prediction, or ansatz-via-self-citation chain is present. Circularity score 0 is therefore appropriate; proxy reliability and prequential generalization remain open empirical risks, not circular reductions.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central empirical claim rests on a small set of free search hyperparameters, standard ML/agent assumptions, and the paper-specific premise that incomplete execution proxies plus an LLM judge can select better control programs. No new physical entities are postulated; the 'invented' object is the adaptation loop itself as a method.

free parameters (4)
  • branches G and rounds R (search budget) = G=3, R=3 (main)
    Population size and evolution depth are chosen by hand; best reported setting G=3,R=3 on BIRD. Accuracy is non-monotonic in budget (Table 1).
  • batch size B = B=10 (main BIRD)
    Controls how much unlabeled evidence and how many commit steps the stream provides; ablated, peaks at B=10 on BIRD (Figure 3).
  • proposer role prompts (conservative / exploratory / adversarial)
    Diversity is steered by hand-written role objectives rather than learned; confounded with G in the best setting.
  • resource and wall-clock budgets for candidates
    Bounds on malformed/non-terminating harnesses and compute; necessary but hand-set guardrails that affect which programs survive.
axioms (4)
  • ad hoc to paper Execution-derived proxies (execution health, round-trip consistency, public-test pass rate) plus raw traces provide a usable unsupervised selection signal for harness quality.
    Core premise of §4.3–4.4 and Eq. 2; paper's own audits show it is imperfect (selection regret, shape-mismatch failures).
  • domain assumption A frozen LLM can serve as solver, proposer, and judge under different harnesses without a stronger teacher or trained adaptation model.
    Stated implementation boundary in §4.5; standard capability assumption for agentic coding scaffolds.
  • domain assumption Transductive batch scoring (adapt on Xt then measure on Xt) is a valid primary evaluation of test-time harness evolution.
    Protocol in §4.4 and §5; authors note prequential evaluation is the decisive missing test (§6).
  • domain assumption Hard evaluation slices selected from benchmark metadata/reference artifacts (baseline failures, patch complexity, solution length) fairly represent the method's value.
    §5.1 and Appendix A; conditions absolute baselines and gain magnitudes.
invented entities (1)
  • Test-Time Harness Evolution (TTHE) loop no independent evidence
    purpose: Defines the adaptation state as executable harness code and the unlabeled generate-and-judge procedure that updates it during evaluation.
    Methodological construct, not a physical entity; independent evidence is the empirical multi-domain runs and released code claim.

pith-pipeline@v1.1.0-grok45 · 20169 in / 3475 out tokens · 36165 ms · 2026-07-10T12:40:06.900216+00:00 · methodology

0 comments
read the original abstract

The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures. Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time. This limits adaptation when the test distribution, failure modes, or tool interactions differ from those seen during development. We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs. We introduce Test-Time Harness Evolution (TTHE), which treats the executable harness as the state of test-time adaptation. During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces, without gold labels or task-specific supervision; a judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs. Crucially, TTHE does not update model weights, require gold labels, or train a separate adaptation model: solver, proposers, and judge are different roles and harnesses around the same frozen LLM, so all adaptation occurs through changes to the surrounding program. Across text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool-use tasks, TTHE improves fixed ReAct-style baseline harnesses, yielding persistent, inspectable improvements rather than a pre-searched workflow or per-query retries. These results recast test-time adaptation for LLM agents as evolution over executable control programs and identify execution-derived proxy reliability as a central challenge for robust unsupervised agent improvement.

Figures

Figures reproduced from arXiv: 2607.08124 by Bo Han, Dahai Yu, Jun Nie, Jun Song, Qianshu Cai, Xinmei Tian, Yike Guo, Yonggang Zhang.

Figure 1
Figure 1. Figure 1: Test-Time Harness Evolution within one unlabeled batch [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Baseline harness versus committed TTHE harness across all evaluated domains. Bars show pass@1 accuracy (%) for the code do￾mains and the mean graded task score for claw￾eval (right of the divider, a different metric in [0, 1], shown ×100). 5 10 25 50 Batch size B (log scale) 0 20 40 60 Accuracy (%) Baseline, 12% 44 50 38 44 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Source of an evolved Text-to-SQL harness (excerpt, lightly abridged), discovered from [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An evolved SWE-bench harness (excerpt, abridged), starting from the stock mini-swe [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 76 canonical work pages · 13 internal anchors

  1. [1]

    International Conference on Machine Learning,

    Test-time training with self-supervision for generalization under distribution shifts , author=. International Conference on Machine Learning,

    work page

  2. [2]

    Olshausen and Trevor Darrell , title =

    Dequan Wang and Evan Shelhamer and Shaoteng Liu and Bruno A. Olshausen and Trevor Darrell , title =. International Conference on Learning Representations,

    work page

  3. [3]

    Advances in Neural Information Processing Systems,

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in Neural Information Processing Systems,

    work page

  4. [4]

    Le and Ed H

    Xuezhi Wang and Jason Wei and Dale Schuurmans and Quoc V. Le and Ed H. Chi and Sharan Narang and Aakanksha Chowdhery and Denny Zhou , title =. International Conference on Learning Representations,

    work page

  5. [5]

    Narasimhan and Yuan Cao , title =

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik R. Narasimhan and Yuan Cao , title =. International Conference on Learning Representations,

    work page

  6. [6]

    Teaching Large Language Models to Self-Debug , booktitle =

    Xinyun Chen and Maxwell Lin and Nathanael Sch. Teaching Large Language Models to Self-Debug , booktitle =

    work page

  7. [7]

    Advances in Neural Information Processing Systems,

    Noah Shinn and Federico Cassano and Ashwin Gopinath and Karthik Narasimhan and Shunyu Yao , title =. Advances in Neural Information Processing Systems,

    work page

  8. [8]

    International Conference on Learning Representations,

    Shengran Hu and Cong Lu and Jeff Clune , title =. International Conference on Learning Representations,

    work page

  9. [9]

    International Conference on Learning Representations,

    Jiayi Zhang and Jinyu Xiang and Zhaoyang Yu and Fengwei Teng and Xionghui Chen and Jiaqi Chen and Mingchen Zhuge and Xin Cheng and Sirui Hong and Jinlin Wang and Bingnan Zheng and Bang Liu and Yuyu Luo and Chenglin Wu , title =. International Conference on Learning Representations,

    work page

  10. [10]

    Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T and Moazam, Hanna and others , journal=

    work page

  11. [11]

    Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution , booktitle =

    Chrisantha Fernando and Dylan Banarse and Henryk Michalewski and Simon Osindero and Tim Rockt. Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution , booktitle =

    work page

  12. [12]

    International Conference on Learning Representations,

    Yongchao Zhou and Andrei Ioan Muresanu and Ziwen Han and Keiran Paster and Silviu Pitis and Harris Chan and Jimmy Ba , title =. International Conference on Learning Representations,

    work page

  13. [13]

    Self-taught optimizer (

    Zelikman, Eric and Lorch, Eliana and Mackey, Lester and Kalai, Adam Tauman , journal=. Self-taught optimizer (

    work page

  14. [16]

    Goodman , title =

    Eric Zelikman and Yuhuai Wu and Jesse Mu and Noah D. Goodman , title =. Advances in neural information processing systems,

    work page

  15. [17]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    Jiaxin Huang and Shixiang Gu and Le Hou and Yuexin Wu and Xuezhi Wang and Hongkun Yu and Jiawei Han , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    work page 2023

  16. [18]

    He, Yufei and Liu, Juncheng and Liu, Yue and Li, Yibo and Cao, Tri and Hu, Zhiyuan and Xu, Xinxing and Hooi, Bryan , journal=

    work page

  17. [19]

    Li, Jinyang and Hui, Binyuan and Qu, Ge and Yang, Jiaxi and Li, Binhua and Li, Bowen and Wang, Bailin and Qin, Bowen and Geng, Ruiying and Huo, Nan and others , booktitle=. Can

    work page

  18. [20]

    Pourreza, Mohammadreza and Rafiei, Davood , booktitle=

    work page

  19. [21]

    Text-to-

    Gao, Dawei and Wang, Haibin and Li, Yaliang and Sun, Xiuyu and Qian, Yichen and Ding, Bolin and Zhou, Jingren , journal=. Text-to-

    work page

  20. [22]

    International Conference on Learning Representations,

    Naman Jain and King Han and Alex Gu and Wen. International Conference on Learning Representations,

    work page

  21. [23]

    Chen, Bei and Zhang, Fengji and Nguyen, Anh and Zan, Daoguang and Lin, Zeqi and Lou, Jian-Guang and Chen, Weizhu , journal=

    work page

  22. [24]

    Competition-level code generation with

    Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R. Competition-level code generation with. Science , year=

    work page

  23. [27]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [28]

    OpenAI o1 System Card

    Openai o1 system card , author=. arXiv preprint arXiv:2412.16720 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [29]

    Jimenez, Carlos E and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle=

    work page

  26. [30]

    Yang, John and Jimenez, Carlos and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle=

    work page

  27. [31]

    Lai, Yuhang and Li, Chengxi and Wang, Yiming and Zhang, Tianyi and Zhong, Ruiqi and Zettlemoyer, Luke and Yih, Wen-tau and Fried, Daniel and Wang, Sida and Yu, Tao , booktitle=

    work page

  28. [34]

    International Conference on Learning Representations,

    Large language models as optimizers , author=. International Conference on Learning Representations,

    work page

  29. [35]

    International Conference on Learning Representations,

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers , author=. International Conference on Learning Representations,

    work page

  30. [36]

    Nature , year=

    Mathematical discoveries from program search with large language models , author=. Nature , year=

    work page

  31. [37]

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and others , booktitle=

    work page

  32. [38]

    Toolformer: Language Models Can Teach Themselves to Use Tools , booktitle =

    Timo Schick and Jane Dwivedi. Toolformer: Language Models Can Teach Themselves to Use Tools , booktitle =

    work page

  33. [39]

    Lianmin Zheng and Wei. Judging. Advances in Neural Information Processing Systems,

    work page

  34. [40]

    Guanzhi Wang and Yuqi Xie and Yunfan Jiang and Ajay Mandlekar and Chaowei Xiao and Yuke Zhu and Linxi Fan and Anima Anandkumar , title =. Trans. Mach. Learn. Res. , year =

    work page

  35. [41]

    International Conference on Learning Representations,

    Hunter Lightman and Vineet Kosaraju and Yuri Burda and Harrison Edwards and Bowen Baker and Teddy Lee and Jan Leike and John Schulman and Ilya Sutskever and Karl Cobbe , title =. International Conference on Learning Representations,

    work page

  36. [42]

    Radev , title =

    Tao Yu and Rui Zhang and Kai Yang and Michihiro Yasunaga and Dongxu Wang and Zifan Li and James Ma and Irene Li and Qingning Yao and Shanelle Roman and Zilin Zhang and Dragomir R. Radev , title =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

    work page 2018

  37. [43]

    Advances in Neural Information Processing Systems,

    Marvin Zhang and Sergey Levine and Chelsea Finn , title =. Advances in Neural Information Processing Systems,

    work page

  38. [44]

    Program Synthesis with Large Language Models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  39. [45]

    MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems

    Qianshu Cai, Yonggang Zhang, Xianzhang Jia, Wei Xue, Jun Song, Xinmei Tian, and Yike Guo. Moss: Self-evolution through source-level rewriting in autonomous agent systems. arXiv preprint arXiv:2605.22794, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  40. [46]

    CodeT: Code Generation with Generated Tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT : Code generation with generated tests. arXiv preprint arXiv:2207.10397, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [47]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  42. [48]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Sch \" a rli, and Denny Zhou. Teaching large language models to self-debug. In International Conference on Learning Representations, ICLR , 2024

    work page 2024

  43. [49]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \" a schel. Promptbreeder: Self-referential self-improvement via prompt evolution. In International Conference on Machine Learning, ICML , 2024

    work page 2024

  44. [50]

    Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to- SQL empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [51]

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, ICLR , 2024

    work page 2024

  46. [52]

    EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems

    Yufei He, Juncheng Liu, Yue Liu, Yibo Li, Tri Cao, Zhiyuan Hu, Xinxing Xu, and Bryan Hooi. EvoTest : Evolutionary test-time learning for self-improving agentic systems. arXiv preprint arXiv:2510.13220, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [53]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, ICLR , 2025

    work page 2025

  48. [54]

    LiveCodeBench : Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen - Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar - Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench : Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, ICLR , 2025

    work page 2025

  49. [55]

    SWE-bench : Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, ICLR , 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench : Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, ICLR , 2024

    work page 2024

  50. [56]

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. DSPy : Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  51. [57]

    DS-1000 : A natural and reliable benchmark for data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000 : A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, ICML , 2023

    work page 2023

  52. [58]

    Meta-Harness: End-to-End Optimization of Model Harnesses

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  53. [59]

    Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to- SQL s

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to- SQL s. In Advances in Neural Information Processing Systems, NeurIPS , 2023

    work page 2023

  54. [60]

    Competition-level code generation with AlphaCode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode . Science, 2022

    work page 2022

  55. [61]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations, ICLR , 2024

    work page 2024

  56. [62]

    Self-Refine : Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine : Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, NeurIPS , 2023

    work page 2023

  57. [63]

    DIN-SQL : Decomposed in-context learning of text-to- SQL with self-correction

    Mohammadreza Pourreza and Davood Rafiei. DIN-SQL : Decomposed in-context learning of text-to- SQL with self-correction. In Advances in Neural Information Processing Systems, NeurIPS , 2023

    work page 2023

  58. [64]

    Mathematical discoveries from program search with large language models

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 2024

    work page 2024

  59. [65]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi - Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, NeurIPS , 2023

    work page 2023

  60. [66]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, NeurIPS , 2023

    work page 2023

  61. [67]

    Test-time training with self-supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, ICML , 2020

    work page 2020

  62. [68]

    Olshausen, and Trevor Darrell

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, ICLR , 2021

    work page 2021

  63. [69]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trans. Mach. Learn. Res., 2024

    work page 2024

  64. [70]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, ICLR , 2023

    work page 2023

  65. [71]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, NeurIPS , 2022

    work page 2022

  66. [72]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, ICLR , 2024 a

    work page 2024

  67. [73]

    SWE-agent : Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent : Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, NeurIPS , 2024 b

    work page 2024

  68. [74]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, ICLR , 2023

    work page 2023

  69. [75]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processi...

    work page 2018

  70. [76]

    Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation

    Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer ( STOP ): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  71. [77]

    Self-Harness: Harnesses That Improve Themselves

    Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-harness: Harnesses that improve themselves. arXiv preprint arXiv:2606.09498, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  72. [78]

    Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954, 2025 a

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  73. [79]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation. In International Conference on Learning Representations, ICLR , 2025 b

    work page 2025

  74. [80]

    MEMO: test time robustness via adaptation and augmentation

    Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, NeurIPS , 2022

    work page 2022

  75. [81]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM -as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, NeurIPS , 2023

    work page 2023

  76. [82]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In International Conference on Learning Representations, ICLR , 2023

    work page 2023