Pith. sign in

REVIEW 3 major objections 5 minor 201 references

Knowledge-Centric Self-Improvement

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Self-improvement can be driven primarily by a curated knowledge base rather than by the agent itself.

desk verdict A plausible, transparent, and genuinely different mechanism for self-improvement via curated knowledge, but the main-loop causal claim is not yet supported — the held-out transfer results are the cleanest evidence in the paper. read the letter →

arxiv 2607.19592 v1 pith:HRBEWU5X submitted 2026-07-21 cs.AI cs.CLcs.LGcs.MA

Xuefei Julie Wang , Lauren Hyoseo Yoon , Chengrui Qu , Amanda Zichang Wang , Atharva Sehgal , Eric Mazumdar , Yisong Yue This is my paper
classification cs.AIcs.CLcs.LGcs.MA
keywords knowledge-centricself-improvementknowledgebasecurationmulti-agentforumsdistillationdisposableagentszero-shottransferLLMgeneralizationevidence-groundedclaims
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the object of self-improvement in an AI system can be a shared, curated knowledge base instead of the agent. Its protocol keeps agents generic, stateless, and disposable: each reads distilled guidance, attempts one task, and writes evidence-grounded claims into task-level and cross-task forums, which are then distilled back into the knowledge base. Because every agent starts from a fresh context, any gain across generations is attributed to the knowledge they received. Across abstract-reasoning, coding, and terminal benchmarks, the paper reports higher solve rates and lower dollar costs than agent-centric self-improvement baselines, and it shows the distilled knowledge transfers to held-out tasks and across LLM families. This would move the bottleneck in autonomous reasoning from agent architecture to the quality and structure of the curated information agents consume.

What carries the argument

The central object is the curated knowledge base shaped by a three-stage protocol: task-level forums, where agents post evidence-grounded claims about what worked and failed; cross-task forums, where claims must take explicit agree, disagree, or synthesize stances toward cited posts and survive only when grounded in concrete evidence; and distillation, which compresses the surviving claims into typed, actionable bundles—'when X, do Y'—that the next generation of fresh agents consumes. The mechanism isolates knowledge as the only changing variable: every agent is re-instantiated with fresh context and no private memory, so improvement across generations is attributed to the knowledge base.

What would settle it

Run the same 10-generation loop with a no-knowledge control: fresh agents with identical prompts, tools, and retry budget but an empty or sham seed knowledge base. If this control closes the gap with the knowledge-seeded solve rates on any benchmark, the claimed attribution to curated knowledge fails. On the transfer side, replacing the frozen distilled bundle with task-irrelevant text should recover the no-knowledge baseline if the bundle is the true carrier of value.

Watch

Extended reading notes

Core claim

The paper's central claim is that a system can improve at tasks without ever improving its agents, and that the persistent improving object should be a curated knowledge base. Agents are kept generic, stateless, and disposable: each one starts with a clean context, receives a distilled knowledge bundle, attempts a task, and contributes evidence back. A three-stage protocol consisting of task-level forum discussion, cross-task forum debate, and distillation converts individual attempts into scoped, evidence-grounded guidance. On the evaluated benchmarks, the paper reports that this protocol outperforms agent-centric self-improvement baselines in solve rate while using less money, that the dis

Load-bearing premise

The central claim rests on the assumption that every generation-over-generation gain comes from the knowledge base, because agents are re-instantiated with fresh context; the paper includes no control that retries unsolved tasks for the same number of generations without seed knowledge, so stochastic retrying alone could account for part of the improvement.

Editorial extensions

If this is right

  • Self-improvement no longer requires a persistent, specialized agent; generic disposable agents suffice, shifting the bottleneck from agent architecture to the curated knowledge they consume.
  • Solve-rate gains and lower dollar costs on the evaluated benchmarks imply self-improvement can be made cheaper to operate and easier to maintain.
  • Because distilled knowledge transfers across LLM families, knowledge produced by one model can be consumed by another, making the improvement portable.
  • Frozen knowledge improves zero-shot performance on held-out tasks, so the curated asset has standalone value after the run ends.
  • Disagreement in forums is preserved as evidence rather than resolved, which may prevent premature convergence on wrong hypotheses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension of the paper's design is a no-knowledge retry control that re-attempts unsolved tasks for the same number of generations with empty seed knowledge; the paper does not report one, so the size of the pure retrying component is untested.
  • If the knowledge-centric paradigm scales, the cost of self-improvement shifts from inference-time agent complexity to offline curation, so knowledge can be amortized across many tasks and models.
  • The protocol's practice of preserving unresolved disagreements as falsified-versus-untried hypotheses suggests a general anti-premature-consensus mechanism for multi-agent systems.
  • A natural extension, not studied in the paper, is to fine-tune a base model on the distilled bundles, turning the reusable knowledge artifact into a direct weight update.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes knowledge-centric self-improvement: instead of improving the agent, the system maintains a persistent, curated knowledge base that fresh, stateless agents read from and write to. A three-stage protocol (task-level forums, cross-task forums, and distillation) converts agent attempts into typed, evidence-grounded bundles. The authors evaluate this protocol on ARC-AGI-1/2, Polyglot, SWE-bench Pro, and Terminal-Bench 2, comparing against agent-centric self-improvement baselines (DGM, HyperAgents, Meta-Harness) and prompt-optimization baselines (GEPA, OpenEvolve). They report higher or competitive solve rates at lower dollar cost, cross-LLM effectiveness (Haiku 4.5 and GPT-5.4-mini), and positive transfer of a frozen knowledge asset to held-out tasks and to a different LLM family. The central claim is that progress in self-improving agentic systems can be driven primarily by the curated persistent knowledge rather than by the agent design.

Significance. If the central claim is sustained, the paper makes a useful conceptual and practical contribution: it identifies a design axis (what persists) that is complementary to the usual agent-centric optimization, and it provides a cheap, inspectable, and transferable improvement artifact. The paper is unusually transparent: it ships detailed protocol schemas (Appendix E), full task maps (Appendix J), cost accounting (Appendix G), baseline provenance/deviation documentation (Appendix H), and worked examples of disagreement-driven curation (Appendix C). The held-out transfer experiment is a genuine out-of-sample test with a frozen knowledge asset and disjoint task splits, and the cross-LLM transfer is a concrete falsifiable prediction. These strengths are real. However, the evidence currently underdetermines the central attribution claim because the main self-improvement loop lacks a no-knowledge control, and the baseline comparisons are against modified forks with information-parity gates rather than the original published systems.

major comments (3)
  1. [§3, §4.1, Table 1] The paper's central attribution claim—"any performance gain across generations must come from the accumulated and distilled knowledge" (§3)—is not established by the reported experiments. Section 4.1 runs 10 generations on a 50-task pool, but there is no control condition that repeats the same fresh-agent retry schedule without any knowledge base. Because solved tasks are removed and unsolved tasks are retried by new stochastic agents, cumulative chance success across 10 attempts can be substantial; the paper's own Table 4 shows single-generation no-knowledge solve rates of 13–23% on held-out ARC-AGI-1 tasks. A no-knowledge generation-loop control (same number of generations, same per-task budget, no forums, no distillation, no knowledge injection) on the same 50-task pool is necessary to separate retrying/stochastic effects from knowledge-driven gains. The held-out transfer experiment (
  2. [§4.1, Appendix H] Table 1a compares against DGM and HyperAgents, but the reported baseline numbers come from forks with substantial modifications: LLM substitution, information-parity gates that remove gold answers and hidden tests from the self-improvement loop, egress isolation, and newly authored benchmark adapters. The claim that "all baselines are rerun under our evaluation protocol" is transparent, but the claim in the abstract and Section 4.1 that the method "outperforms agent-centric baselines" should be qualified as outperforming the authors' modified, information-parity-gated re-runs, not the original published systems. Whether the parity gates preserve the "core optimization algorithm" is an assumption, not a demonstrated fact; for example, DGM's diagnosis loop is described in Appendix H as no longer seeing gold patches and hidden tests, which may be integral to its original self-modification m
  3. [§4.4, Appendix L, Appendix J] The transfer evidence is weakened by post-hoc design choices that are disclosed but not tested. First, the task-conditioned adapter's item bounds were relaxed after observing benchmark variation (Appendix L: "we relaxed the constraints and allowed the agent to dynamically determine how much knowledge to transfer"), making the adapter a tuned component rather than a fixed property of the frozen knowledge asset. Second, the transfer evaluation set was selected from tasks that both recipient LLMs fail at seed 0 (Appendix J), which inflates headroom and makes the no-knowledge baseline low by construction; this is disclosed, but it means the transfer numbers are conditional on a selection procedure that depends on the recipients' no-knowledge abilities. Third, for Polyglot the transfer split is task-identifier-disjoint but not exercise-disjoint: 7 of 20 evaluation tasks are the same Exercism
minor comments (5)
  1. [§1] Typographical: "in thisknowledge-centric self-improvementparadigm" is missing spaces around "knowledge-centric" and "paradigm."
  2. [Table 1/Table 3] The table header "OURShaiku 4.5" and "OURSgpt 5.4−mini" are visually cramped; please add spacing or a line break for readability.
  3. [Figure 3b] The example labels contain apparent typos ("Count-Dataset-T okens", "T ask-level Insights"). Please proofread figure text.
  4. [§4.4] The caption for Table 4 says "20 held-out tasks" but the selection procedure in Appendix J says tasks were chosen from the intersection of both recipients' failure sets; please make the selection criterion explicit in the main text so readers do not interpret the held-out set as a random sample of all held-out tasks.
  5. [§4.2] The comparison with GEPA and OpenEvolve uses matched dollar budgets but different iteration counts; it would be helpful to state explicitly that the prompt-optimization baselines were not extended to other benchmarks for budget reasons, and to note that this comparison is therefore narrower than the agent-centric comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark comparisons and held-out transfer are externally grounded; the causal-attribution gap is a missing control, not a self-referential derivation.

full rationale

The solve-rate claims in Table 1 rest on external benchmark outcomes (official ARC exact-match scoring, SWE-bench Pro harness, Polyglot tests, Terminal-Bench 2 verifier), and the reported costs cover all loop phases, so the headline comparison is not a rescaling of the authors' own definitions. The held-out transfer experiment (Section 4.4) is a genuine out-of-sample test: the generation-10 asset is frozen and consumed zero-shot by fresh recipients on disjoint 20-task splits with no recipient-side forum or distillation, and the comparison is against a no-knowledge condition on the same tasks. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity. The paper contains no load-bearing self-citation and no imported uniqueness theorem; its references are independent prior systems, and no argument's force comes from an author-overlapping citation. Two caveats should be weighed as validity concerns rather than circularity. First, Section 3 asserts 'any performance gain across generations must come from the accumulated and distilled knowledge they receive' merely from re-instantiating agents; without a no-knowledge generation control that retries unsolved tasks for the same number of generations, cumulative stochastic retrying is a confound, but this is an omitted control rather than an input-output identity. Second, Appendix J admits the transfer tasks were selected from a seed-0 no-knowledge failure intersection ('the no-knowledge baseline solve rates are low by construction') and that 7 of 20 Polyglot transfer tasks are the same Exercism exercise in a different language; Section L similarly notes the transfer adapter was relaxed after observing benchmark variation. These limit the strength of the transfer claims but do not make the improvement definitionally equal to the knowledge base's own content. I therefore find no significant circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on protocol hyperparameters, a causal attribution assumption, and the validity of modified baseline forks. No new physical or mathematical entities are introduced; the knowledge base, forums, and bundles are software artifacts.

free parameters (5)
  • generations = 10
    Hand-set improvement budget; final solve rates and costs are reported at this stopping point; no ablation shows sensitivity to this choice.
  • forum_rounds = 1 per-task forum round; 2 cross-task forum rounds per generation
    Chosen protocol structure; no ablation varying the number of rounds.
  • task_conditioned_adapter_bounds = 0-3 items per list field
    Transfer adapter limits; Section L says constraints were relaxed after observing benchmark variation, so bounds are post-hoc.
  • transfer_eval_set = 20 tasks that both recipient LLMs failed at seed-0
    Selection rule in Section 4.4/Appendix J deliberately keeps hard tasks; acknowledged by authors and controlled by using the same set for the no-knowledge baseline, but it affects absolute solve rates.
  • baseline_fork_timeouts = 3600s per task vs 1800s in main runs
    Budget-parity choice in Appendix H; changes the cost and success profile of the rerun baselines.
assumptions (6)
  • domain assumption LLM agents produce consistent enough behavior across fresh contexts that generation-over-generation gains can be attributed to the knowledge base.
    Section 3/4.1 assumes 'any performance gain across generations must come from accumulated knowledge'; no no-knowledge generation loop is run to test this.
  • ad hoc to paper The information-parity gates and LLM substitutions in the baseline forks preserve the core optimization algorithms of DGM and HyperAgents.
    Appendix H documents modified forks; if the gates cripple baseline capability, the comparison 'OURS vs baselines' is not a comparison to the original systems.
  • domain assumption Held-out transfer tasks are representative of the self-improvement task distribution.
    Appendix J samples from disjoint pools, but Polyglot transfer includes 7 tasks that reuse the same exercise as a donor task in another language; ARC is disjoint.
  • domain assumption Benchmark exact-match and verification protocols are accepted measures of task success.
    Section 4.1 relies on ARC official exact-match scoring, SWE-bench Pro harness, Polyglot test commands, and Terminal-Bench verification.
  • domain assumption Removing solved tasks from the active pool does not bias comparisons against baselines.
    Section 4.1 removes solved tasks to save tokens; it is not stated whether the rerun baselines use the same removal rule, so cost/solve comparisons may not be apples-to-apples.
  • ad hoc to paper Forum and distillation prompts elicit evidence-grounded, actionable knowledge rather than generic advice.
    The protocol's effectiveness depends on this; the only evidence is worked examples in Appendix C, not a controlled comparison of prompt variants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Knowledge-Centric Self-Improvement." pith.science (2026). https://pith.science/paper/HRBEWU5X

@misc{pith2026260719592,
  author       = {Pith},
  title        = {Pith review of: Knowledge-Centric Self-Improvement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRBEWU5X}},
  note         = {Machine review of arXiv:2607.19592}
}
read the original abstract

Self-improving AI systems typically treat the agent as the object that improves, by optimizing prompts, workflows, harnesses, or even the agent's own code. This agent-centric view can make improvements expensive to maintain and difficult to transfer, because gains become tied to a particular agent design, task distribution, or adaptation run. We study a complementary paradigm: knowledge-centric self-improvement, in which agents remain generic and disposable while the persistent object is a curated knowledge base that agents can leverage for future tasks. We conduct controlled case studies to operationalize this idea via a simple protocol. Agents attempt one task, then contribute evidence-grounded insights to a shared knowledge base via task-level and cross-task forums, followed by knowledge distillation. Because self-improvement is contained in the knowledge rather than the agent, improvement can be more inspectable, transferable, and portable. Across abstract reasoning, coding, and terminal benchmarks, this protocol improves solve rates while reducing dollar cost relative to agent-centric baselines. The resulting distilled knowledge also transfers to held-out tasks and across LLM families, indicating that the improvement is not merely an LLM- or run-specific behavior. These results support a new view of self-improving agentic systems: progress can be driven primarily by the curated persistent knowledge. Code is available at https://github.com/recursive-knowledge/KSI.

Figures

Figures reproduced from arXiv: 2607.19592 by the authors.

Figure 1
Figure 1. From agent-centric to knowledge-centric self-improvement. (Left) Conventional self￾improving agentic systems treat the agent as the persistent substrate, evolving prompts, workflows, or agent code. (Middle) We invert this design. Agents are kept generic and disposable, while the persistent improving substrate is a shared knowledge base, curated through a simple protocol to generate and utilize distilled insights. (R… view at source ↗
Figure 2
Figure 2. Knowledge curation protocol consisting of three stages. Task-level forum (agents propose task-level posts summarizing local outcomes and evi￾dence on the current task), Cross-task forum (agents discuss transferable pat￾terns and review peer posts with support￾ing or challenging evidence), and Distil￾lation (forum posts are consolidated into typed task-level and cross-task bundles written back to the shared knowledge… view at source ↗
Figure 3
Figure 3. Examples of knowledge curation on two task families. Local task evidence is first organized [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Simplified attempt runtime lifecycle. Solid arrows are forward calls, dashed arrows are [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: In-depth overview and example of knowledge curation: full curation framework and actual [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: In-depth overview and example of knowledge curation: full curation framework and actual [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

201 extracted references · 30 linked inside Pith

  1. [1]

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    arXiv 2025

  2. [2]

    Akiba, M

    T. Akiba, M. Shing, Y . Tang, Q. Sun, and D. Ha. Evolutionary Optimization of Model Merging Recipes.Nature Machine Intelligence, 7(2):195–204, 2025. ISSN 2522-5839. doi: 10.1038/s42256-024-00975-8

    doi:10.1038/s42256-024-00975-8 2025

  3. [3]

    Assumpção, D

    H. Assumpção, D. Ferreira, L. Campos, and F. Murai. CodeEvolve: An open source evolutionary coding agent for algorithmic discovery and optimization.arXiv preprint arXiv:2510.14150, 2025

    arXiv 2025

  4. [4]

    goose: An open source, extensible ai agent, 2025

    Block, Inc. goose: An open source, extensible ai agent, 2025. URL https://github.com/ block/goose

    2025

  5. [5]

    Borgeaud, A

    S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre. Improving langua...

    arXiv 2022

  6. [6]

    Chhikara, D

    P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    arXiv 2025

  7. [7]

    F. Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    arXiv 1911

  8. [8]

    W. L. de Oliveira, M. Bobokhonov, M. Caorsi, A. Podestà, G. Beltramo, L. Crosato, M. Bonotto, F. Cecchetto, H. Espic, D. T. Salajan, S. Taga, L. Pana, and J. Carthy. Arc-agi-2 technical report. arXiv preprint arXiv:2603.06590, 2026

    arXiv 2026

Show all 201 references
  1. [9]

    X. Deng, J. Da, E. Pan, Y . Y . He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V . Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler. Swe-bench pro: Can ai agents solve ...

    2025 arXiv

  2. [10]

    R. Fang, Y . Liang, X. Wang, J. Wu, S. Qiao, P. Xie, F. Huang, H. Chen, and N. Zhang. Memp: Exploring agent procedural memory.arXiv preprint arXiv:2508.06433, 2025

    2025 arXiv

  3. [11]

    Fernando, D

    C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofICML’24, pages 13481–13544. JMLR.org, 2024

    2024

  4. [12]

    Fu, D.-K

    Y . Fu, D.-K. Kim, J. Kim, S. Sohn, L. Logeswaran, K. Bae, and H. Lee. Autoguide: Automated generation and selection of context-aware guidelines for large language model agents.arXiv preprint arXiv:2403.08978, 2024

    2024 arXiv

  5. [13]

    Gauthier

    P. Gauthier. o1 tops aider’s new polyglot leaderboard.https://aider.chat/2024/12/21/ polyglot.html, Dec. 2024. Accessed: 2026-01-28

    2024

  6. [14]

    S. Hu, C. Lu, and J. Clune. Automated design of agentic systems.arXiv preprint arXiv:2408.08435, 2025

    2025 arXiv

  7. [15]

    Terminus-KIRA: Boosting frontier model performance on terminal-bench with minimal harness, 2026

    KRAFTON AI and Ludo Robotics. Terminus-KIRA: Boosting frontier model performance on terminal-bench with minimal harness, 2026. URL https://github.com/krafton-ai/ KIRA

    2026

  8. [16]

    Y . Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn. Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026. 11

    2026 arXiv

  9. [17]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive nlp tasks.arXiv preprint arXiv:2005.11401, 2020

    2005 arXiv

  10. [18]

    Maharana, D.-H

    A. Maharana, D.-H. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y . Fang. Evaluating very long-term conversational memory of llm agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 13851–13870. Asso...

    2024

  11. [19]

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y . Liu, R. Zhang, L. L. Chen, A. Kashyap, J....

    2026 arXiv

  12. [20]

    Novikov, N

    A. Novikov, N. V˜u, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog. Alphaevolve: A coding agent for scientific and alg...

    2025 arXiv

  13. [21]

    Packer, S

    C. Packer, S. Wooders, K. Lin, V . Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2024

    2024 arXiv

  14. [22]

    Y . Pu, T. Lin, and H. Chen. PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaboration.arXiv preprint arXiv:2505.15047, May 2025

    2025

  15. [23]

    A. Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URL https: //github.com/algorithmicsuperintelligence/openevolve

    2025

  16. [24]

    X. Tang, T. Qin, T. Peng, Z. Zhou, D. Shao, T. Du, X. Wei, P. Xia, F. Wu, H. Zhu, G. Zhang, J. Liu, X. Wang, S. Hong, C. Wu, H. Cheng, C. Wang, and W. Zhou. Agent kb: Leveraging cross-domain experience for agentic problem solving.arXiv preprint arXiv:2507.06229, 2025

    2025

  17. [25]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An Open-Ended Embodied Agent with Large Language Models.arXiv preprint arXiv:2305.16291, Oct. 2023

    2023 arXiv

  18. [26]

    J. Wang, Z. Hu, and L. Bing. Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective. InF orty-Second International Conference on Machine Learning, June 2025

    2025

  19. [27]

    W. Wang, P. Pi˛ ekos, L. Nanbo, F. Laakom, Y . Chen, M. Ostaszewski, M. Zhuge, and J. Schmid- huber. Huxley-gödel machine: Human-level coding agent development by an approximation of the optimal self-improving machine.arXiv preprint arXiv:2510.21614, 2025

    2025

  20. [28]

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. Openhands: An open platform for ai software de...

    2025 arXiv

  21. [29]

    Z. Z. Wang, J. Mao, D. Fried, and G. Neubig. Agent workflow memory.arXiv preprint arXiv:2409.07429, 2024

    2024 arXiv

  22. [30]

    D. Wu, H. Wang, W. Yu, Y . Zhang, K.-W. Chang, and D. Yu. Longmemeval: Benchmarking chat assistants on long-term interactive memory.arXiv preprint arXiv:2410.10813, 2024

    2024 arXiv

  23. [31]

    R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y . Shen, Y . Wang, and B. Shi. Evolver: Self-evolving llm agents through an experience-driven lifecycle.arXiv preprint arXiv:2510.16079, 2026. 12

    2026 arXiv

  24. [32]

    Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. OS-Copilot: Towards Generalist Computer Agents with Self-Improvement, Feb. 2024

    2024

  25. [33]

    C. S. Xia, Z. Wang, Y . Yang, Y . Wei, and L. Zhang. Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?, Nov. 2025

    2025

  26. [34]

    W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y . Zhang. A-mem: Agentic memory for llm agents.arXiv preprint arXiv:2502.12110, 2025

    2025 arXiv

  27. [35]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable automated software engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  28. [36]

    L. Yang, Z. Yu, T. Zhang, S. Cao, M. Xu, W. Zhang, J. E. Gonzalez, and B. Cui. Buffer of thoughts: Thought-augmented reasoning with large language models.arXiv preprint arXiv:2406.04271, 2024

    2024 arXiv

  29. [37]

    H. Ye, X. He, V . Arak, H. Dong, and G. Song. Meta Context Engineering via Agentic Skill Evolution.arXiv preprint arXiv:2601.21557, Feb. 2026

    2026

  30. [38]

    X. Yin, X. Wang, L. Pan, L. Lin, X. Wan, and W. Y . Wang. Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computationa...

  31. [39]

    Zelikman, Y

    E. Zelikman, Y . Wu, J. Mu, and N. Goodman. STaR: Bootstrapping Reasoning With Reasoning. InAdvances in Neural Information Processing Systems, volume 35, pages 15476–15488, Oct. 2022

    2022

  32. [40]

    Zhang, M

    G. Zhang, M. Fu, G. Wan, M. Yu, K. Wang, and S. Yan. G-memory: Tracing hierarchical memory for multi-agent systems.arXiv preprint arXiv:2506.07398, 2025

    2025 arXiv

  33. [41]

    Zhang, H

    G. Zhang, H. Ren, C. Zhan, Z. Zhou, J. Wang, H. Zhu, W. Zhou, and S. Yan. MemEvolve: Meta-Evolution of Agent Memory Systems.arXiv preprint arXiv:2512.18746, Dec. 2025

    2025 arXiv

  34. [42]

    Zhang, S

    J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune. Darwin godel machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954, 2025

    2025 arXiv

  35. [43]

    Zhang, B

    J. Zhang, B. Zhao, W. Yang, J. Foerster, J. Clune, M. Jiang, S. Devlin, and T. Shavrina. Hyperagents.arXiv preprint arXiv:2603.19461, 2026

    2026

  36. [44]

    Zhang, C

    Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V . Kamanuru, J. Rainton, C. Wu, M. Ji, H. Li, U. Thakker, J. Zou, and K. Olukotun. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.arXiv preprint arXiv:2510.04618, Mar. 2026

    2026 arXiv

  37. [45]

    Zhang, J

    S. Zhang, J. Wang, R. Zhou, J. Liao, Y . Feng, Z. Li, Y . Zheng, W. Zhang, Y . Wen, Z. Li, F. Xiong, Y . Qi, B. Tang, and M. Wen. Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    2026 arXiv

  38. [46]

    Zhang, P

    Y . Zhang, P. Ye, X. Yang, S. Feng, S. Zhang, L. Bai, W. Ouyang, and S. Hu. Nature-Inspired Population-Based Evolution of Large Language Models.arXiv preprint arXiv:2503.01155, Mar. 2025

    2025 arXiv

  39. [47]

    A. Zhao, D. Huang, Q. Xu, M. Lin, Y .-J. Liu, and G. Huang. Expel: Llm agents are experiential learners.arXiv preprint arXiv:2308.10144, 2024

    2024 arXiv

  40. [48]

    Zheng, M

    B. Zheng, M. Y . Fatemi, X. Jin, Z. Z. Wang, A. Gandhi, Y . Song, Y . Gu, J. Srinivasa, G. Liu, G. Neubig, and Y . Su. Skillweaver: Web agents can self-improve by discovering and honing skills.arXiv preprint arXiv:2504.07079, 2025

    2025 arXiv

  41. [49]

    Zhong, L

    W. Zhong, L. Guo, Q. Gao, H. Ye, and Y . Wang. Memorybank: Enhancing large language models with long-term memory.arXiv preprint arXiv:2305.10250, 2023

    2023 arXiv

  42. [50]

    adaptive rejection sampling

    M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber. GPTSwarm: Language Agents as Optimizable Graphs. InF orty-First International Conference on Machine Learning, June 2024. 13 Appendix Contents A. Attempt Runtime Lifecycle . . . . . . . . . . . . . . . ...

    2024

  43. [52]

    Read each training pair; note dimensions, palette, objects, and what changed input -> output

  44. [53]

    Within your first few turns, write a first-pass guess to BOTH attempt_1.txt and attempt_2.txt (a copy of the test input is fine as a placeholder) so the workspace always contains a submission

  45. [54]

    Form one transformation rule that explains every training pair and verify it pair-by-pair before applying

  46. [55]

    Apply the rule to the test input and overwrite attempt_1.txt and attempt_2.txt with the refined answer (submit two structurally different hypotheses if you can construct them; otherwise submit the same grid into both files)

  47. [56]

    Read ‘TASK.md‘ in the active task workspace

    Validate before exit: python3 /workspace/task/workspace/validate_prediction.py \\ /workspace/task/workspace/attempt_1.txt \\ /workspace/task/workspace/attempt_2.txt """ Listing 7: Illustrative ARC prompt excerpt; abstracted from _build_arc_no_mcp_execution_promptinprompts/__in...

  48. [57]

    25d487eb

    83302e8f 26. 25d487eb

  49. [58]

    ea32f347

    f76d97a5 27. ea32f347

  50. [59]

    1fad071e

    8efcae92 28. 1fad071e

  51. [60]

    dc0a314f

    1190e5a7 29. dc0a314f

  52. [61]

    6ecd11f4

    57aa92db 30. 6ecd11f4

  53. [62]

    9ecd008a

    a85d4709 31. 9ecd008a

  54. [63]

    b8825c91

    a416b8f3 32. b8825c91

  55. [64]

    25d8a9c8

    890034e9 33. 25d8a9c8

  56. [65]

    776ffc46 27

    694f12f3 34. 776ffc46 27

  57. [66]

    9172f3a0

    a1570a43 35. 9172f3a0

  58. [67]

    6cf79266

    780d0b14 36. 6cf79266

  59. [68]

    caa06a1f

    bdad9b1f 37. caa06a1f

  60. [69]

    d22278a0

    4938f0c2 38. d22278a0

  61. [70]

    4612dd53

    a740d043 39. 4612dd53

  62. [71]

    b6afb2da

    3428a4f5 40. b6afb2da

  63. [72]

    94f9d214

    6455b5f5 41. 94f9d214

  64. [73]

    aabf363d

    f35d900a 42. aabf363d

  65. [74]

    5ad4f10b

    23b5c85d 43. 5ad4f10b

  66. [75]

    1e0a9b12

    ce22a75a 44. 1e0a9b12

  67. [76]

    b548a754

    5582e5ca 45. b548a754

  68. [77]

    05f2a901

    af902bf9 46. 05f2a901

  69. [78]

    239be575

    e48d4e1a 47. 239be575

  70. [79]

    e8593010

    c8f0f002 48. e8593010

  71. [80]

    8731374e

    36d67576 49. 8731374e

  72. [81]

    e509e548 ARC-AGI-1 Knowledge-Transfer Evaluation Tasks

    6b9890af 50. e509e548 ARC-AGI-1 Knowledge-Transfer Evaluation Tasks

  73. [82]

    9def23fe

    15113be4 11. 9def23fe

  74. [83]

    ac2e8ecf

    18419cfa 12. ac2e8ecf

  75. [84]

    af22c60d

    1c56ad9f 13. af22c60d

  76. [85]

    b20f7c8b

    4aab4007 14. b20f7c8b

  77. [86]

    b7cb93ac

    4b6b68e5 15. b7cb93ac

  78. [87]

    b9630600

    626c0bcc 16. b9630600

  79. [88]

    ce039d91

    64a7c07e 17. ce039d91

  80. [89]

    e2092e0c

    79369cc6 18. e2092e0c

  81. [90]

    e7b06bea

    94be5b80 19. e7b06bea

  82. [91]

    ecaa0ec1 ARC-AGI-2 Train/Self-improvement Tasks

    96a8c0cd 20. ecaa0ec1 ARC-AGI-2 Train/Self-improvement Tasks

  83. [92]

    27a28665

    de493100 26. 27a28665

  84. [93]

    c909285e

    694f12f3 27. c909285e

  85. [94]

    1efba499

    c9680e90 28. 1efba499

  86. [95]

    a5f85a15

    e7a25a18 29. a5f85a15

  87. [96]

    d43fd935

    72a961c9 30. d43fd935

  88. [97]

    4612dd53

    11dc524f 31. 4612dd53

  89. [98]

    ea9794b1

    484b58aa 32. ea9794b1

  90. [99]

    91413438

    fcc82909 33. 91413438

  91. [100]

    b9b7f026

    8be77c9e 34. b9b7f026

  92. [101]

    d5d6de2d

    84f2aca1 35. d5d6de2d

  93. [102]

    9f8de559

    6d75e8bb 36. 9f8de559

  94. [103]

    e9afcf9a

    ed98d772 37. e9afcf9a

  95. [104]

    29623171

    d06dbe63 38. 29623171

  96. [105]

    5521c0d9

    db118e2a 39. 5521c0d9

  97. [106]

    1f876c06

    543a7ed5 40. 1f876c06

  98. [107]

    bf699163

    fd4b2b02 41. bf699163

  99. [108]

    1a6449f1

    834ec97d 42. 1a6449f1

  100. [109]

    e98196ab

    63613498 43. e98196ab

  101. [110]

    e0fb7511

    9b365c51 44. e0fb7511

  102. [111]

    b548a754

    e7dd8335 45. b548a754

  103. [112]

    5ad8a7c0

    ea786f4a 46. 5ad8a7c0

  104. [113]

    825aa9e9

    3d6c6e23 47. 825aa9e9

  105. [114]

    973e499e

    896d5239 48. 973e499e

  106. [115]

    2013d3e2

    278e5215 49. 2013d3e2

  107. [116]

    62ab2642 SWE-bench Pro Train/Self-improvement Tasks

    5034a0b5 50. 62ab2642 SWE-bench Pro Train/Self-improvement Tasks

  108. [117]

    instance_gravitational__teleport-af5e2517de7d18406b614e413aca61c319312171- ,→vee9b09fb20c43af7e520f57e9239bbcf46b7113d

  109. [118]

    instance_internetarchive__openlibrary-1894cb48d6e7fb498295a5d3ed0596f6f603b784- ,→v0f5aece3601a5b4419f7ccec1dbda2071be28ee4

  110. [119]

    instance_NodeBB__NodeBB-f2082d7de85eb62a70819f4f3396dd85626a0c0a- ,→vd59a5728dfc977f44533186ace531248c2917516

  111. [120]

    instance_flipt-io__flipt-db1c3b100e231c62f0c90c2ab037614f20a2a63b

  112. [121]

    instance_navidrome__navidrome-3972616585e82305eaf26aa25697b3f5f3082288

  113. [122]

    instance_internetarchive__openlibrary-d8162c226a9d576f094dc1830c4c1ffd0be2dd17- ,→v76304ecdb3a5954fcf13feb710e8c40fcf24b73c

  114. [123]

    instance_gravitational__teleport-eda668c30d9d3b56d9c69197b120b01013611186

  115. [124]

    instance_future-architect__vuls-86b60e1478e44d28b1aff6b9ac7e95ceb05bc5fc

  116. [125]

    instance_internetarchive__openlibrary-bb152d23c004f3d68986877143bb0f83531fe401- ,→ve8c8d62a2b60610a3c4631f5f23ed866bada9818 28

  117. [126]

    instance_gravitational__teleport-3ff19cf7c41f396ae468797d3aeb61515517edc9- ,→vee9b09fb20c43af7e520f57e9239bbcf46b7113d

  118. [127]

    instance_protonmail__webclients-715dbd4e6999499cd2a576a532d8214f75189116

  119. [128]

    instance_flipt-io__flipt-5c7037ececb0bead0a8eb56054e224bcd7ac5922

  120. [129]

    instance_navidrome__navidrome-23bebe4e06124becf1000e88472ae71a6ca7de4c

  121. [130]

    instance_element-hq__element-web-18c03daa865d3c5b10e52b669cd50be34c67b2e5-vnan

  122. [131]

    instance_future-architect__vuls-2c84be80b65d022c262956cd26fc79d8bb2f7010

  123. [132]

    instance_element-hq__element-web-27139ca68eb075a4438c18fca184887002a4ffbc-vnan

  124. [133]

    instance_ansible__ansible-a20a52701402a12f91396549df04ac55809f68e9- ,→v1055803c3a812189a1133297f7f5468579283f86

  125. [134]

    instance_qutebrowser__qutebrowser-0833b5f6f140d04200ec91605f88704dd18e2970- ,→v059c6fdc75567943479b23ebca7c07b5e9a7f34c

  126. [135]

    instance_flipt-io__flipt-c188284ff0c094a4ee281afebebd849555ebee59

  127. [136]

    instance_navidrome__navidrome-89b12b34bea5687c70e4de2109fd1e7330bb2ba2

  128. [137]

    instance_tutao__tutanota-d1aa0ecec288bfc800cfb9133b087c4f81ad8b38- ,→vbc0d9ba8f0071fbe982809910959a6ff8884dbbf

  129. [138]

    instance_protonmail__webclients-caf10ba9ab2677761c88522d1ba8ad025779c492

  130. [139]

    instance_element-hq__element-web-459df4583e01e4744a52d45446e34183385442d6-vnan

  131. [140]

    instance_future-architect__vuls-aaea15e516ece43978cf98e09e52080478b1d39f

  132. [141]

    instance_ansible__ansible-b2a289dcbb702003377221e25f62c8a3608f0e89- ,→v173091e2e36d38c978002990795f66cfc0af30ad

  133. [142]

    instance_ansible__ansible-5e369604e1930b1a2e071fecd7ec5276ebd12cb1- ,→v0f01c69f1e2528b935359cfe578530722bca2c59

  134. [143]

    instance_qutebrowser__qutebrowser-ed19d7f58b2664bb310c7cb6b52c5b9a06ea60b2- ,→v059c6fdc75567943479b23ebca7c07b5e9a7f34c

  135. [144]

    instance_future-architect__vuls-f0b3a8b1db98eb1bd32685f1c36c41a99c3452ed

  136. [145]

    instance_internetarchive__openlibrary-acdddc590d0b3688f8f6386f43709049622a6e19- ,→vfa6ff903cb27f336e17654595dd900fa943dcd91

  137. [146]

    instance_protonmail__webclients-0ec14e36ceb01ba45602a563e12352af8171ed39

  138. [147]

    instance_ansible__ansible-b6290e1d156af608bd79118d209a64a051c55001- ,→v390e508d27db7a51eece36bb6d9698b63a5b638a

  139. [148]

    instance_gravitational__teleport-3587cca7840f636489449113969a5066025dd5bf

  140. [149]

    instance_internetarchive__openlibrary-43f9e7e0d56a4f1d487533543c17040a029ac501- ,→v0f5aece3601a5b4419f7ccec1dbda2071be28ee4

  141. [150]

    instance_future-architect__vuls-c11ba27509f733d7d280bdf661cbbe2e7a99df4c

  142. [151]

    instance_protonmail__webclients-dfe5604193d63bfcb91ce60d62db2f805c43bf11

  143. [152]

    instance_qutebrowser__qutebrowser-44e64199ed38003253f0296badd4a447645067b6- ,→v2ef375ac784985212b1805e1d0431dc8f1b3c171

  144. [153]

    instance_flipt-io__flipt-36e62baffae2132f78f9d34dc300a9baa2d7ae0e

  145. [154]

    instance_navidrome__navidrome-f7d4fcdcc1a59d1b4f835519efb402897757e371

  146. [155]

    instance_internetarchive__openlibrary-5fb312632097be7e9ac6ab657964af115224d15d- ,→v0f5aece3601a5b4419f7ccec1dbda2071be28ee4

  147. [156]

    instance_navidrome__navidrome-669c8f4c49a7ef51ac9a53c725097943f67219eb

  148. [157]

    instance_flipt-io__flipt-dbe263961b187e1c5d7fe34c65b000985a2da5a0

  149. [158]

    instance_ansible__ansible-3b823d908e8a5d17674f8c26d337d3114b7493b1- ,→v0f01c69f1e2528b935359cfe578530722bca2c59

  150. [159]

    instance_navidrome__navidrome-e12a14a87d392ac70ee4cc8079e3c3e0103dbcb2

  151. [160]

    instance_NodeBB__NodeBB-445b70deda20201b7d9a68f7224da751b3db728c- ,→v4fbcfae8b15e4ce5d132c408bca69ebb9cf146ed

  152. [161]

    instance_ansible__ansible-a02e22e902a69aeb465f16bf03f7f5a91b2cb828- ,→vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5

  153. [162]

    instance_gravitational__teleport-d873ea4fa67d3132eccba39213c1ca2f52064dcc- ,→vce94f93ad1030e3136852817f2423c1b3ac37bc4

  154. [163]

    instance_tutao__tutanota-f3ffe17af6e8ab007e8d461355057ad237846d9d- ,→vbc0d9ba8f0071fbe982809910959a6ff8884dbbf

  155. [164]

    instance_qutebrowser__qutebrowser-bf045f7ec7c27709ea3ef61cf41a24e8fdd2e7da- ,→v059c6fdc75567943479b23ebca7c07b5e9a7f34c

  156. [165]

    instance_qutebrowser__qutebrowser-1a9e74bfaf9a9db2a510dc14572d33ded6040a57- ,→v2ef375ac784985212b1805e1d0431dc8f1b3c171

  157. [166]

    instance_NodeBB__NodeBB-04998908ba6721d64eba79ae3b65a351dcfbc5b5-vnan Polyglot Train/Self-improvement Tasks

  158. [167]

    java__forth

    javascript__queen-attack 26. java__forth

  159. [168]

    python__dominoes

    rust__wordy 27. python__dominoes

  160. [169]

    go__word-search

    python__dot-dsl 28. go__word-search

  161. [170]

    javascript__simple-linked-list

    java__satellite 29. javascript__simple-linked-list

  162. [171]

    go__counter

    cpp__diamond 30. go__counter

  163. [172]

    java__react

    rust__accumulate 31. java__react

  164. [173]

    javascript__ocr-numbers

    go__error-handling 32. javascript__ocr-numbers

  165. [174]

    python__scale-generator

    cpp__queen-attack 33. python__scale-generator

  166. [175]

    java__go-counting

    rust__poker 34. java__go-counting

  167. [176]

    rust__doubly-linked-list

    python__sgf-parsing 35. rust__doubly-linked-list

  168. [177]

    python__grade-school

    rust__react 36. python__grade-school

  169. [178]

    javascript__forth

    java__ledger 37. javascript__forth

  170. [179]

    python__wordy

    go__connect 38. python__wordy

  171. [180]

    java__mazy-mice

    rust__macros 39. java__mazy-mice

  172. [181]

    cpp__bank-account

    javascript__triangle 40. cpp__bank-account

  173. [182]

    python__zipper 29

    java__zipper 41. python__zipper 29

  174. [183]

    java__custom-set

    java__bowling 42. java__custom-set

  175. [184]

    java__rest-api

    python__tree-building 43. java__rest-api

  176. [185]

    go__transpose

    javascript__say 44. go__transpose

  177. [186]

    rust__gigasecond

    java__wordy 45. rust__gigasecond

  178. [187]

    rust__say

    python__food-chain 46. rust__say

  179. [188]

    go__food-chain

    javascript__wordy 47. go__food-chain

  180. [189]

    rust__pig-latin

    python__poker 48. rust__pig-latin

  181. [190]

    go__markdown

    javascript__grade-school 49. go__markdown

  182. [191]

    go__crypto-square Polyglot Knowledge-Transfer Evaluation Tasks

    cpp__gigasecond 50. go__crypto-square Polyglot Knowledge-Transfer Evaluation Tasks

  183. [192]

    java__protein-translation

    cpp__circular-buffer 11. java__protein-translation

  184. [193]

    java__sgf-parsing

    go__beer-song 12. java__sgf-parsing

  185. [194]

    java__state-of-tic-tac-toe

    go__ledger 13. java__state-of-tic-tac-toe

  186. [195]

    java__twelve-days

    java__affine-cipher 14. java__twelve-days

  187. [196]

    javascript__go-counting

    java__change 15. javascript__go-counting

  188. [197]

    javascript__phone-number

    java__dominoes 16. javascript__phone-number

  189. [198]

    python__forth

    java__hangman 17. python__forth

  190. [199]

    python__paasio

    java__palindrome-products 18. python__paasio

  191. [200]

    python__phone-number

    java__pig-latin 19. python__phone-number

  192. [201]

    You are a knowledge-transfer adapter for Polyglot coding tasks

    java__pov 20. python__simple-linked-list K Benchmark Details We evaluate on five benchmarks spanning coding, terminal skills, and abstract reasoning: • Polyglot[ 13]: The Polyglot benchmark (Aider Polyglot) is an evaluation framework designed to assess the code generation and ...

  193. [2025]

    doi: 10.18653/v1/2025.acl-long.1354

    Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1354

    2025 doi

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.