Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Steering many coding agents with separate git branches and local context finds stronger open-ended optimizations than one long-running agent polishing a single approach.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:54 UTC pith:A3DWRCR2

load-bearing objection Useful harness idea with a clean diagnosis of agent collapse, but the 13/15 headline is a single-run ranking the authors themselves flag as high-variance. the 4 major comments →

arxiv 2607.02807 v1 pith:A3DWRCR2 submitted 2026-07-02 cs.AI

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

Yuvraj Virk , Zack Edds , Chunqiu Steven Xia , Lingming Zhang This is my paper
classification cs.AI
keywords coding agentsopen-ended discoveryorchestrator-subagenttest-time scalinggit branchingmulti-agent searchcontext managementprogram evolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-running coding agents can keep improving code for open-ended problems, but they often lock onto one high-level idea and then spend hours on small edits, missing better directions. This paper argues that two harness choices cause that trap: one growing conversation history and one shared program to edit. SwarmResearch instead uses a Shepherd Agent with global memory to spawn Search Agents that each work in their own git branch with only local context—fresh for explorers, inherited for optimizers. Across fifteen open-ended math, systems, and heuristics tasks, that design matches or beats strong evolutionary and multi-agent baselines on thirteen tasks, and it does so by making larger, higher-level code changes. An orchestrator that adapts how many agents run in parallel at different depths also beats the best fixed parallel-versus-serial schedule on most tested tasks. The practical claim is that harness design—not just a better model—can keep autonomous code search exploring rather than converging early.

Core claim

The paper claims that an orchestrator-subagent harness, in which a Shepherd Agent uses global summaries to choose parents, agent types, and minimal prompts while Search Agents edit only their own git branch under local context, discovers better or comparable solutions to leading LLM-guided evolution and multi-agent methods on 13 of 15 open-ended optimization tasks, primarily because it sustains higher-level exploration rather than early commitment to one approach.

What carries the argument

SwarmResearch: a Shepherd Agent that steers a population of Search Agents via three controls—parent selection on a new git branch, Explorer versus Optimizer type, and non-prescriptive prompts—while Explorers start with fresh local context and Optimizers fork a parent’s history, so global strategy and local experimentation stay separated.

Load-bearing premise

That one long frontier-model run per task is the fairest representative comparison, even though discovery methods are known to vary a lot from run to run.

What would settle it

Rerun SwarmResearch and the same baselines for several independent frontier-model budgets on the same fifteen tasks and check whether the 13/15 win count and the orchestrator-over-fixed-scaling wins still hold with error bars.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SwarmResearch, an orchestrator–subagent harness for open-ended optimization with coding agents. A Shepherd Agent maintains global context and steers a population of Search Agents (Explorers with fresh context; Optimizers that fork parent history), each on a separate git branch/worktree with lineage-local findings.md. The authors argue that single long-running agents and single-program experimental loops cause premature convergence to one high-level approach. Empirically, on 15 math/systems/heuristics tasks they report better or comparable solutions to EvoX and CORAL on 13/15 tasks under a ~$50 Claude Code budget, larger median LOC changes as a proxy for higher-level exploration, and—on a stripped Pi harness—orchestrator-guided scaling beating optimal fixed (n,k) scaling on 4/5 tasks. A speculative-decoding case study reports ~4.58× tok/s vs vanilla with held-out evaluation.

Significance. If the empirical ranking holds under re-runs, the work is a clear contribution to agent harness design for open-ended discovery: it isolates two concrete failure modes (context accumulation; single editable program state), implements a minimal skills-based fix interoperable with Claude Code, and shows both stronger solutions and more flexible test-time scaling than fixed serial/parallel agents. Strengths include a multi-domain suite (math, ADRS systems, ALE-Bench heuristics), an explicit scaling ablation, a detailed speculative-decoding case study with held-out tasks, public code, and unusually transparent discussion of cost, variance, and Shepherd limitations (Appendix B, §3.5, §4). The architecture is simple enough to be useful as a base for practitioners and for future training on discovery.

major comments (4)
  1. §3.1–3.2, Table 1, Appendix B: The headline claim (better/comparable on 13/15 tasks) rests on one run per method per task with no error bars. Appendix B itself states that discovery methods have high between-run variance and that stable estimates may require multiple independent runs, then justifies one long run as “most representative” under cost. Those positions cannot jointly underwrite a 13/15 win count. Several margins are tiny (Circle Packing 2.635996 vs 2.635985; Erdős 0.381080 vs 0.381099; MMD-14-3 essentially tied; PRISM all equal), so reordering under re-runs is plausible. At minimum the paper should (i) multi-run a subset of tasks (or report bootstrap/seed variance on cheaper models), (ii) report win rates with uncertainty, and (iii) separate “clear wins” from “ties within noise” rather than aggregating 13/15.
  2. §3.1 baseline fairness: EvoX is run with Opus 4.6 and the authors note it is weaker than the original GPT-based results; the CORAL reproduction underperforms the originally reported numbers, yet original CORAL numbers are still cited as SOTA “where applicable.” Table 1 therefore mixes a same-model reproduction with selective original scores and a model-mismatched evolutionary baseline. For the central ranking claim, either all methods should be re-run under a fixed model/API stack with multi-seed estimates, or the paper should demote “beats SOTA multi-agent/evolution” to a same-stack comparison and treat original published numbers only as external reference, not as co-ranked competitors.
  3. §3.4, Table 2: The orchestrator-guided vs fixed-scaling result (4/5 wins) is important but not yet commensurate with the main claim. Fixed scaling uses a minimal Pi harness, Minimax-M2.5, shared git history, and 3 runs; the “minimal SwarmResearch” variant further restricts the orchestrator to branch selection only and uses Sonnet-4.6 for orchestration. This isolates a useful mechanism, but it does not establish that full SwarmResearch’s $50 Claude Code runs beat fixed scaling of full coding agents. Either extend multi-run fixed-scaling controls to the main stack on a few tasks, or clearly scope Table 2 as a mechanism study rather than support for the abstract’s scaling claim.
  4. §3.3, Figure 6: Median LOC changed is used as primary evidence that gains are “driven by higher-level exploration.” LOC is a weak proxy (rewrites can be cosmetic; small diffs can be algorithmically deep), and the paper already notes high-level experiments can be inefficient when strong approaches need polishing (EPLB, AHC026). Strengthen the causal story with qualitative taxonomy of approach families per task (as in the speculative-decoding figure), or ablate Explorer-only vs Optimizer-heavy populations, rather than resting the “driven by” claim mainly on LOC medians.
minor comments (5)
  1. Figure 1 is dense and hard to read in preprint form; a simplified tree plus a short table of top speedups/accuracies would help.
  2. §3.1: Clarify exact evaluator APIs, seeds, and whether any task-specific hyperparameters differ across methods beyond the stated budgets.
  3. Table 1 caption says SwarmResearch exceeds CORAL on 10/15 and matches 2, while the boxed summary later says exceeds on 8 and similar on 5—align these counts.
  4. Skills in Appendix C are valuable; a short main-text box listing the three non-prescriptive steering mechanisms and the “no specific ideas” guardrail would improve accessibility.
  5. Related work could more sharply contrast with island-model / MAP-Elites evolutionary systems that already preserve multiple niches, to clarify what is new beyond agentic packaging of multi-lineage search.

Circularity Check

0 steps flagged

No circular derivation: empirical harness paper evaluated on external task metrics and external baselines.

full rationale

SwarmResearch is a systems/empirical paper, not a first-principles derivation. Its load-bearing claims are comparative performance on open-ended optimization tasks against EvoX and CORAL, plus a fixed-scaling ablation. Task scores come from external evaluators (math constructions, ADRS systems benchmarks, ALE-Bench heuristics, and held-out speculative-decoding throughput/accuracy); baselines are independently specified methods, not quantities defined in terms of SwarmResearch’s outputs. The LOC-changed analysis is an interpretive proxy the authors associate with high-level rewrites, not a fitted parameter renamed as a prediction or a self-definitional identity. There is no uniqueness theorem, self-citation chain that forces the result, ansatz smuggled in as derivation, or renaming of a known closed-form result. Between-run variance and single-run ranking concerns affect reliability of the empirical ranking, not circularity of a derivation chain. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is empirical systems design plus evaluation protocol, not a formal theory. The claim rests on domain assumptions about LLM coding agents, fixed dollar/iteration budgets, chosen baselines and models, and the representativeness of single long runs. Free parameters are operational (budgets, wave sizes, model IDs). Invented entities are the harness roles and artifacts, which are software constructs with independent evidence via the public skills/code rather than postulated physical objects.

free parameters (5)
  • Per-task dollar budget for CORAL/SwarmResearch = $50
    Main comparison stops at $50 of Claude Code spend; ranking can shift under different budgets as authors note diminishing but non-converged returns.
  • Shepherd explorer agent budget (skills) = 25 explorers
    Skill text hard-codes an explorer budget of 25; population dynamics depend on this operational limit.
  • EvoX iteration count = 100 iterations
    Baseline fixed at 100 iterations (~$23.50 average) as in the EvoX paper configuration.
  • Fixed-scaling total iteration budget and (n,k) grid = 60 iterations; (5,12)…(30,2)
    Scaling study uses 60 total iterations and five discrete width×depth configurations; 'optimal fixed scaling' is optimal only within this grid.
  • Primary model choices (Opus 4.6/4.8, Minimax-M2.5, Sonnet-4.6 orchestrator)
    Authors explicitly note EvoX is model-sensitive (weaker under Opus than GPT); results are conditioned on these model IDs.
axioms (5)
  • domain assumption Long-running coding agents' premature convergence is substantially caused by single accumulated context and a single editable program state (harness-level hypothesis).
    Stated in Abstract/Introduction as the motivating hypothesis; experiments support the redesigned harness but do not isolate each cause with a full factorial ablation of context vs git alone.
  • domain assumption Task evaluators and SOTA reference scores are valid external ground for ranking discovery methods.
    Math/systems/heuristic metrics and human/AI SOTA numbers are taken as given; internals of evaluators are hidden from agents by design (Section 3.1).
  • ad hoc to paper Under realistic cost, one long run per technique is more representative than multiple short runs for these methods.
    Appendix B argues this for SwarmResearch and applies the same single-run protocol to baselines; this is a methodological choice, not a standard statistical principle.
  • ad hoc to paper Median lines of code changed per attempt is a useful proxy for high-level vs low-level exploration.
    Section 3.3 equates larger diffs with higher-level approach changes; plausible but not independently validated as a diversity metric.
  • standard math Standard software/git and LLM agent tool semantics (worktrees, commits, session fork/resume) behave as described.
    Background engineering assumptions required for the harness to function as specified in Appendix C.
invented entities (3)
  • Shepherd Agent independent evidence
    purpose: Global-context orchestrator that selects parents, agent type (Explorer/Optimizer), and minimal prompts without prescribing specific solution ideas.
    Core invented role of the architecture; evidence is the skill definition and reported behaviors, not an external natural phenomenon.
  • Explorer vs Optimizer Search Agents independent evidence
    purpose: Subagents with fresh vs forked conversation history to trade diversity against serial refinement.
    Defined operationally in Section 2; independent evidence via runnable skills/code.
  • findings.md lineage-local log independent evidence
    purpose: Lightweight ancestor attempt log without full shared memory across lineages.
    Harness artifact introduced to manage context; present in skills and architecture description.

pith-pipeline@v1.1.0-grok45 · 24473 in / 4110 out tokens · 38273 ms · 2026-07-12T06:54:49.670895+00:00 · methodology

0 comments
read the original abstract

Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems. However, they tend to converge onto a single high-level approach, then proceed with low-level edits while missing other superior approaches to the problem. We hypothesize two harness-level design choices contribute to this behavior: accumulating context in a single long-running agent and only exposing a single program state to edit. We introduce SwarmResearch, an orchestrator-subagent harness in which a Shepherd Agent uses global context to steer a population of Search Agents, each operating with local context in their respective git branch. On open-ended optimization tasks, SwarmResearch discovers better or comparable solutions to state-of-the-art LLM-guided evolution and multi-agent techniques on 13/15 tasks, driven by higher-level exploration. Compared with fixed scaling of serial and parallel agents, SwarmResearch's orchestrator-guided scaling discovers better-performing solutions by adapting parallelism at different search depths.

Figures

Figures reproduced from arXiv: 2607.02807 by Chunqiu Steven Xia, Lingming Zhang, Yuvraj Virk, Zack Edds.

Figure 1
Figure 1. Figure 1: SWARMRESEARCH designs speculative decoding implementations. Nodes represent spawned search agents and their solutions. Edges indicate the lower node builds on the solution above. The top node value shows spawn order; the bottom value shows speedup over vanilla decoding. SWARMRESEARCH explores diverse approaches over ∼11 hours without converging onto any one approach. On 100 held-out reasoning-intensive tas… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SWARMRESEARCH. Implements an orchestrator-subagent loop: (1) Shepherd Agent spawns concurrent search agents with 3 per-agent steering mechanisms: selecting a parent solution by setting up its git branch, writing a prompt to share minimal context, and selecting the Search Agent type. (2) Search Agents ideate, implement, evaluate, and iterate on a solution. Explorer Search Agents have fresh conte… view at source ↗
Figure 3
Figure 3. Figure 3: SWARMRESEARCH context management for search agents. SWARMRESEARCH separates context into two tiers. Search Agents operate with local context: the contents of their worktree and, for Optimizer Agents, the conversation history of their direct parent. The Shepherd Agent operates with global context: summaries of all Search Agent attempts, including their approach and evaluated score. This separation is design… view at source ↗
Figure 4
Figure 4. Figure 4: Git management for Search Agents. The Shepherd Agent assigns every Search Agent to a new git branch and worktree. The main branch contains only the initial task setup: a prompt.md file defining the problem and a minimal working baseline solution. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of median-sized SWARMRESEARCH and CORAL diffs for Signal Processing. sized diff from CORAL involves tuning thresholds while a median size diff from SWARMRESEARCH redesigns the solver into different modes applied on different conditions. On most tasks, higher￾level experiments mean SWARMRESEARCH discovers better solutions. However, within the budget, CORAL better exploited strong approaches and dis… view at source ↗
Figure 6
Figure 6. Figure 6: Median lines of code changed per attempt by S [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 21 linked inside Pith

  1. [1]

    Gepa: Reflective prompt evolution can outperform reinforcement learning

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization

    Henrique Assumpção, Diego Ferreira, Leandro Campos, and Fabricio Murai. Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization. 2025. URL https://arxiv.org/abs/2510.14150

    Pith/arXiv arXiv 2025

  3. [3]

    Entropy-guided dynamic expert selection in mixture-of-experts models, January 2026

    Gabriele Balsamo. Entropy-guided dynamic expert selection in mixture-of-experts models, January 2026. URL https://adaptive-k.vercel.app/paper.html. Preprint, under review

    2026

  4. [4]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Flex: Continuous agent evolution via forward learning from experience

    Zhicheng Cai, Xinyuan Guo, Yu Pei, Jiangtao Feng, Jinsong Su, Jiangjie Chen, Ya-Qin Zhang, Wei-Ying Ma, Mingxuan Wang, and Hao Zhou. Flex: Continuous agent evolution via forward learning from experience. arXiv preprint arXiv:2511.06449, 2025

    arXiv 2025

  6. [6]

    Adaevolve: Adaptive llm driven zeroth-order optimization

    Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, et al. Adaevolve: Adaptive llm driven zeroth-order optimization. arXiv preprint arXiv:2602.20133, 2026

    arXiv 2026

  7. [7]

    Avo: Agentic variation operators for autonomous evolutionary search

    Terry Chen, Zhifan Ye, Bing Xu, Zihao Ye, Timmy Liu, Ali Hassani, Tianqi Chen, Andrew Kerr, Haicheng Wu, Yang Xu, et al. Avo: Agentic variation operators for autonomous evolutionary search. arXiv preprint arXiv:2603.24517, 2026

    arXiv 2026

  8. [8]

    Barbarians at the gate: How ai is upending systems research

    Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. Barbarians at the gate: How ai is upending systems research. arXiv preprint arXiv:2510.06189, 2025

    arXiv 2025

  9. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    Codemonkeys: Scaling test-time compute for software engineering

    Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher R \'e , and Azalia Mirhoseini. Codemonkeys: Scaling test-time compute for software engineering. arXiv preprint arXiv:2501.14723, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Towards an ai co-scientist

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Sam decoding: Speculative decoding via suffix automaton

    Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. Sam decoding: Speculative decoding via suffix automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025

    2025

  13. [13]

    SpecDec++ : Boosting speculative decoding via adaptive candidate lengths

    Kaixuan Huang, Xudong Guo, and Mengdi Wang. SpecDec++ : Boosting speculative decoding via adaptive candidate lengths. In Proceedings of the Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=NnExMNiTHw

    2025

  14. [14]

    Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering

    Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering. In Advances in Neural Information Processing Systems, 2025

    2025

  15. [15]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. URL https://arxiv.org/abs/2403.07974

    Pith/arXiv arXiv 2024

  16. [16]

    autoresearch, 2026

    Andrej Karpathy. autoresearch, 2026. URL https://github.com/karpathy/autoresearch

    2026

  17. [17]

    Mahoney, Amir Gholami, and Kurt Keutzer

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Advances in Neural Information Processing Systems, volume 36, pp.\ 39236--39256. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/7b97adeafa1c51cf652...

    2023

  18. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, 2023

    2023

  19. [19]

    Shinkaevolve: Towards open-ended and sample-efficient program evolution

    Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. Shinkaevolve: Towards open-ended and sample-efficient program evolution. arXiv preprint arXiv:2509.19349, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, 2023

    2023

  21. [21]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840, 2025. URL https://arxiv.org/abs/2503.01840

    Pith/arXiv arXiv 2025

  22. [22]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The twelfth international conference on learning representations, 2023

    2023

  23. [23]

    Evox: Meta-evolution for automated discovery

    Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z Pan, et al. Evox: Meta-evolution for automated discovery. arXiv preprint arXiv:2602.23413, 2026

    arXiv 2026

  24. [24]

    The ai scientist: Towards fully automated open-ended scientific discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    AdaSD : Adaptive speculative decoding for efficient language model inference, 2025

    Kuan-Wei Lu, Ding-Yong Hong, Pangfeng Liu, and Jan-Jan Wu. AdaSD : Adaptive speculative decoding for efficient language model inference, 2025. URL https://arxiv.org/abs/2512.11280

    Pith/arXiv arXiv 2025

  26. [26]

    Performance: 53\ https://github.com/Shopify/liquid/pull/2056, mar 2026

    Tobias L \"u tke. Performance: 53\ https://github.com/Shopify/liquid/pull/2056, mar 2026. Pull request \#2056 to Shopify Liquid

    2056

  27. [27]

    AIME 2026

    MathArena . AIME 2026 . Hugging Face dataset, 2026. URL https://huggingface.co/datasets/MathArena/aime_2026. Dataset of problems from the 2026 American Invitational Mathematics Examination

    2026

  28. [28]

    Illuminating search spaces by mapping elites

    Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015

    Pith/arXiv arXiv 2015

  29. [29]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 20286--20332, 2025

    2025

  30. [30]

    Alphaevolve: A coding agent for scientific and algorithmic discovery

    Alexander Novikov, Ng \^a n V \ u , Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Claudini: Autoresearch discovers state-of-the-art adversarial attack algorithms for llms

    Alexander Panfilov, Peter Romov, Igor Shilov, Yves-Alexandre de Montjoye, Jonas Geiping, and Maksym Andriushchenko. Claudini: Autoresearch discovers state-of-the-art adversarial attack algorithms for llms. arXiv preprint arXiv:2603.24511, 2026

    Pith/arXiv arXiv 2026

  32. [32]

    Humanity's last exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sarah Zhang, Michael Chen, Michael Ong, Aarohi Srivastava, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025. URL https://arxiv.org/abs/2501.14249

    Pith/arXiv arXiv 2025

  33. [33]

    Coral: Towards autonomous multi-agent evolution for open-ended discovery

    Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, Jiacheng Zhu, Xuan Jiang, Sirui Li, Cathy Wu, Bryan Kian Hsiang Low, Jinhua Zhao, and Paul Pu Liang. Coral: Towards autonomous multi-agent evolution for open-ended discovery. arXiv preprint arXiv:2604.01658, 2026. URL https://arxiv....

    Pith/arXiv arXiv 2026

  34. [34]

    Recursive introspection: Teaching language model agents how to self-improve

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37: 0 55249--55285, 2024

    2024

  35. [35]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. URL https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv 2023

  36. [36]

    Autokernel

    RightNow-AI . Autokernel. https://github.com/RightNow-AI/autokernel, 2026. Open-source autoresearch system for GPU kernel optimization

    2026

  37. [37]

    Hive, 2026

    rllm org. Hive, 2026. URL https://github.com/rllm-org/hive

    2026

  38. [38]

    Mathematical discoveries from program search with large language models

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625 0 (7995): 0 468--475, 2024

    2024

  39. [39]

    Openevolve: an open-source evolutionary coding agent, 2025

    Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URL https://github.com/algorithmicsuperintelligence/openevolve

    2025

  40. [40]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Thetaevolve: Test-time learning on open problems

    Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baolin Peng, et al. Thetaevolve: Test-time learning on open problems. arXiv preprint arXiv:2511.23473, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Agentless: Demystifying llm-based software engineering agents

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering, 2 0 (FSE): 0 801--824, 2025. doi:10.1145/3715754

    doi:10.1145/3715754 2025

  43. [43]

    Pacevolve: Enabling long-horizon progress-aware consistent evolution

    Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, et al. Pacevolve: Enabling long-horizon progress-aware consistent evolution. arXiv preprint arXiv:2601.10657, 2026

    arXiv 2026

  44. [44]

    Evaluation-driven scaling for scientific discovery

    Haotian Ye, Haowei Lin, Jingyi Tang, Yizhen Luo, Caiyin Yang, Chang Su, Rahul Thapa, Rui Yang, Ruihua Liu, Zeyu Li, Chong Gao, Dachao Ding, Guangrong He, Miaolei Zhang, Lina Sun, Wenyang Wang, Yuchen Zhong, Zhuohao Shen, Di He, Jianzhu Ma, Stefano Ermon, Tongyang Li, Xiaowen Chu, James Zou, and Yuzhi Xu. Evaluation-driven scaling for scientific discovery....

    Pith/arXiv arXiv 2026

  45. [45]

    Learning to discover at test time

    Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, et al. Learning to discover at test time. arXiv preprint arXiv:2601.16175, 2026

    Pith/arXiv arXiv 2026

  46. [46]

    Pi coding agent

    Mario Zechner. Pi coding agent. https://github.com/earendil-works/pi, 2025. Open-source coding agent CLI

    2025

  47. [47]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang : Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pp.\ 62557--62583. Curran Associates, Inc., 2024...

    2024