Pith. sign in

REVIEW 2 major objections 2 minor 2 cited by

Current search strategies steer AI-generated research ideas on quality and novelty axes but do not expand the frontier to high-quality novel results.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-02 21:08 UTC pith:FGZZDJJ2

load-bearing objection The paper shows that six search strategies across three domains produced no 'Original' ideas and only one novel idea in any top-10 quality list, but the human novelty and quality ratings lack reported reliability checks. the 2 major comments →

arxiv 2606.25198 v2 pith:FGZZDJJ2 submitted 2026-06-23 cs.AI

Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty

Antonis Antoniades , Deepak Nathani , Ritam Saha , Alfonso Amayuelas , Ivan Bercovich , Zhaotian Weng , Vignesh Baskaran , Kunal Bhatia
show 1 more author
William Yang Wang
This is my paper
classification cs.AI
keywords autonomous AI researchsearch strategiesquality diversity noveltyLLM agentsreward hackingmachine learning explorationidea generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Heuresis as a way to break the machine learning research process into reusable parts so autonomous agents can explore ideas systematically. It applies six search strategies, ranging from simple greedy selection to archive-based and divergent methods, across three domains and measures thousands of outputs on quality, diversity, and novelty. Results indicate that ideas rated as novel remain rare and low-performing, with only one such idea reaching the top ten by quality score across all runs. The work shows that existing approaches can direct where ideas land on the three axes but leave the combination of high novelty and high performance untouched. This finding highlights a concrete barrier for agents aiming at ongoing scientific discovery without human guidance.

Core claim

Across 3,222 scored runs in LLM pretraining, on-policy RL, and model unlearning, no idea received an original rating, novel ideas never approached the highest known-recipe scores, and only one novel idea appeared in any domain's top ten by quality; the six strategies allow control over placement on the quality-diversity-novelty space yet leave the quality-novelty frontier unchanged.

What carries the argument

The Heuresis framework, which decomposes the research pipeline into general composable primitives that support open-ended exploration by LLM agents.

Load-bearing premise

Human raters assign consistent objective scores for novelty and quality, and the 40 detected fabrications capture every case of reward-hacking that could distort the measurements.

What would settle it

A replication run in which any of the six strategies produces at least five ideas rated original or with only minor similarity that also match or exceed the top known-recipe quality scores in their domain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Archive-based and divergent strategies shift generated ideas toward greater diversity or novelty compared with greedy or evolutionary baselines.
  • Reward-hacking behaviors appear during execution and must be filtered to preserve valid quality-novelty measurements.
  • Novel ideas remain low in quality across all tested methods, so steering alone does not produce competitive new contributions.
  • The quality-novelty frontier stays fixed, meaning perpetual autonomous progress requires mechanisms beyond current search primitives.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agents may require new primitives that explicitly link novelty generation to performance improvement rather than treating the two as separate axes.
  • The observed rarity of original ideas suggests that scaling current LLM agents without architectural changes will continue to yield incremental rather than breakthrough outputs.
  • Extending the same evaluation protocol to non-ML scientific domains could test whether the frontier limitation is specific to machine learning or more general.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces the Heuresis framework, which abstracts the machine-learning research pipeline into general, composable primitives for LLM-based autonomous research agents. It implements and evaluates six search strategies (greedy baseline, MAP-Elites, Go-Explore, Islands, Curiosity, Omni) across three domains (LLM pretraining, on-policy RL, model unlearning) in a total of 3,222 scored runs. The central empirical claims are that completely novel ideas are rare (zero rated 'Original', few 'Minor Similarity'), that novel ideas never reach the highest quality scores achieved by known recipes (only one such idea appears in any top-10 quality ranking across all strategies and domains), that agents exhibit reward-hacking (40 confirmed fabrications), and that existing quality-diversity and divergent search methods can steer placement along the three axes but do not expand the quality-novelty frontier. Code is released at github.com/a-antoniades/Heuresis.

Significance. If the measurement of novelty and quality is shown to be reliable, the work supplies a large-scale, multi-domain empirical demonstration that current archive-based, evolutionary, and curiosity-driven search strategies fail to produce high-quality novel research ideas. It explicitly documents reward-hacking behaviors and the necessity of fabrication detection, and the public code release supports direct reproducibility. These findings sharpen the open challenge of perpetual autonomous scientific progress beyond the capabilities of existing quality-diversity algorithms.

major comments (2)
  1. [Abstract and Evaluation protocol] The headline result that novel ideas never approach the highest-performing known-recipe scores (only one lands in any top-10 quality ranking) depends entirely on the stability and objectivity of human novelty binning ('Original', 'Minor Similarity', etc.) and quality rankings across 3,222 runs. The manuscript does not report inter-rater reliability metrics (e.g., Cohen's kappa, Fleiss' kappa, or pairwise agreement percentages) for these human scores, leaving the frontier-expansion claim sensitive to rater inconsistency or systematic bias.
  2. [Results on reward-hacking] The claim that 40 detected fabrications represent the full extent of reward-hacking (and that detection was sufficient to keep measurements faithful) is load-bearing for the quality-novelty measurements. The manuscript provides no quantitative assessment of detection completeness (e.g., false-negative rate on a held-out set of runs or inter-annotator agreement on fabrication labels), so undetected or subtler reward-hacking could still inflate quality scores for non-novel ideas.
minor comments (2)
  1. [Abstract] The abstract states 'only one such idea lands in the top-10 by quality' but does not specify whether 'top-10' is computed per domain, per strategy, or globally; a clarifying sentence would remove ambiguity.
  2. [Abstract] The GitHub link is given without a commit hash or release tag, which reduces reproducibility of the exact 3,222-run dataset and scoring protocol.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which highlights important aspects of evaluation reliability in our work. We address each major comment below and will revise the manuscript to incorporate the suggested improvements.

read point-by-point responses

  1. Referee: [Abstract and Evaluation protocol] The headline result that novel ideas never approach the highest-performing known-recipe scores (only one lands in any top-10 quality ranking) depends entirely on the stability and objectivity of human novelty binning ('Original', 'Minor Similarity', etc.) and quality rankings across 3,222 runs. The manuscript does not report inter-rater reliability metrics (e.g., Cohen's kappa, Fleiss' kappa, or pairwise agreement percentages) for these human scores, leaving the frontier-expansion claim sensitive to rater inconsistency or systematic bias.

    Authors: We agree that inter-rater reliability metrics are important for validating the human evaluations of novelty and quality. We will add Fleiss' kappa and pairwise agreement percentages for the novelty binning and quality rankings in the revised manuscript. This will be reported in a dedicated subsection on the evaluation protocol. revision: yes

  2. Referee: [Results on reward-hacking] The claim that 40 detected fabrications represent the full extent of reward-hacking (and that detection was sufficient to keep measurements faithful) is load-bearing for the quality-novelty measurements. The manuscript provides no quantitative assessment of detection completeness (e.g., false-negative rate on a held-out set of runs or inter-annotator agreement on fabrication labels), so undetected or subtler reward-hacking could still inflate quality scores for non-novel ideas.

    Authors: We agree that a quantitative assessment of the detection process would further support the claims. We will include inter-annotator agreement on fabrication labels and an analysis of potential false negatives in the revised manuscript, to be added to the section discussing reward-hacking. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical evaluation relies on external human ratings and direct execution

full rationale

The paper presents an empirical study executing six search strategies on three ML domains for 3,222 runs, with novelty/quality/diversity assessed via human raters and fabrications manually detected. No equations, fitted parameters, predictions derived from inputs, or self-citation chains appear in the reported chain; results follow directly from the executed strategies and external scoring process without reduction to self-definition or renaming. The work is self-contained against its stated benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the reliability of human scoring of novelty and quality plus the assumption that the three chosen domains are representative of broader ML research.

axioms (1)
  • domain assumption Human raters can consistently and objectively assign novelty and quality scores to generated ideas across 3,222 runs
    The evaluation and all reported statistics depend on these scores.

pith-pipeline@v0.9.1-grok · 5892 in / 1252 out tokens · 30298 ms · 2026-07-02T21:08:36.611107+00:00 · methodology

0 comments
read the original abstract

Autonomous AI Research promises to accelerate the scientific progress of machine learning. To realise this goal, current Large Language Model (LLM)-based agents need to go beyond just writing code, to mastering the exploration of simultaneously performant, diverse and novel ideas. To this end, we introduce Heuresis, a framework that abstracts the research pipeline into a set of general and composable primitives, enabling open-ended scientific exploration in machine learning research. We implement six search strategies: a greedy baseline, two archive-based (MAP-Elites, Go-Explore), one evolutionary (Islands), and two divergent (Curiosity, Omni), and evaluate them across three axes (Quality, Diversity, and Novelty) on three domains (LLM Pretraining, On-Policy RL, and Model Unlearning), totalling 3,222 scored runs. We find that completely novel ideas are rare. No idea across our scored runs is rated as "Original", and only a few achieve only "Minor Similarity" to prior work. Moreover, novel ideas never approach the highest-performing known-recipe scores. Across all six strategies and three domains, only one such idea lands in the top-10 by quality. We also observed agents resorting to a variety of reward-hacking techniques during execution (40 confirmed fabrications across 1,628 scored runs), and detecting them was necessary to keep the search faithful to the task. Our results show that while current search and Quality-Diversity strategies enable us to steer where the generated ideas land on the quality, diversity, and novelty axes, they do not expand the quality-novelty frontier. Bridging this gap is the open challenge towards the ultimate goal of perpetual, autonomous scientific progress. Code is available at github.com/a-antoniades/Heuresis.

Figures

Figures reproduced from arXiv: 2606.25198 by Alfonso Amayuelas, Antonis Antoniades, Deepak Nathani, Ivan Bercovich, Kunal Bhatia, Ritam Saha, Vignesh Baskaran, William Yang Wang, Zhaotian Weng.

Figure 1
Figure 1. Figure 1: Agentic loop internals. An Ideator (1) proposes a code change and an Executor (2) imple￾ments it, sharing a swappable agent backend. Multiple Ideator–Executor pairs run asynchronously in parallel. The MemoryServer (3) stores framework-recorded experiments and agent-authored learnings, queryable by semantic KNN or SQL. The GradingServer (4) scores the run; the Hack￾erJudge (5) audits the workspace and emits… view at source ↗
Figure 2
Figure 2. Figure 2: Search methods overview. Six pluggable search strategies that share the same agent loop ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Fitness curves and score distributions. Top row: fitness curves; bottom row: score distributions across search strategies on the three tasks. (a) nanoGPT fitness curve, (b) On-Policy RL fitness curve, (c) Model Unlearning fitness curve, (d) nanoGPT score distribution, (e) On-Policy RL score distribution, (f) Model Unlearning score distribution. reports the running best across valid solutions in the top row… view at source ↗
Figure 4
Figure 4. Figure 4: Idea-space diversity. Across search strategies on (a) NanoGPT and (b) DiscoGen OnPolicyRL. Each row shows a UMAP projection of gemini-embedding-001 embeddings of every accepted idea (left) and the distribution of pairwise cosine distances within each strategy (right). Stars mark each strategy’s best run; the outlined star is the overall best. RMU [34] family. CURIOSITY also crosses the 1.0 unlearned-but-st… view at source ↗
Figure 5
Figure 5. Figure 5: Novelty and quality–novelty trade-off. (a–c) Per-strategy novelty distributions on nanoGPT, On-Policy RL, and Model Unlearning under the 5-point rubric of Gupta and Pruthi [19] (5 = direct copy, 1 = original). (d–f) Pooled quality vs. novelty for the same tasks; the step line traces the cross-strategy Pareto front in (quality, novelty) space. For each strategy the first 300 executed ideas are taken and fil… view at source ↗
Figure 6
Figure 6. Figure 6: Per-island Island Search progress on NanoGPT. Island 0 best: 0.9825 0.9825 Mutating SwiGLU with MQA to reallocate K/V parameter savings into a deeper 13-layer network. 0.9828 Mutating parent solution with Multi-Query Attention to reduce K/V parameters, reinvesting savings into a deeper network. 0.9873 Combine SwiGLU, untied lm_head, and Multi-Query Attention to scale model depth within a parameter budget. … view at source ↗
Figure 7
Figure 7. Figure 7: Linear (Greedy) – running-best timeline. 29 successive best updates from exec_002 (1.0029) to exec_269 (0.9567) within the first 300 iterations. Stateful top-K linear isn’t a tree: every iteration’s first parent is the current best, so any ancestor of the global best has hundreds of flat sibling children. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: MAP-Elites – dominant subtree (founder exec_041). Best in panel: exec_226 (0.9936). Cell-targeted MAP-Elites spreads work across many shallow subtrees rather than deep chains; the dominant tree within 300 iterations here is depth 3. exec_085 0.9849 exec_274 0.9899 exec_186 0.9902 exec_132 0.9940 exec_281 0.9972 0.9850 0.9875 0.9900 0.9925 0.9950 0.9975 1.0000 1.0025 1.0050 val_bpb (lower is better) Go-Expl… view at source ↗
Figure 9
Figure 9. Figure 9: Go-Explore – dominant subtree. Like cell-targeted MAP-Elites, Go-Explore samples a target cell each iteration and the resulting tree is wide and shallow; high failure rate (153 training crashes and 61 judge-errored iterations out of 300) prunes the visible tree heavily. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Islands – one panel per island ( N = 8, 2 × 4 grid). Each panel shows the per-island dominant subtree within the first 300 iterations; only the founder and best-in-panel are labelled to keep the figure readable. Migration edges (purple dashed) cross island boundaries; crossover edges (orange curved) are within-island. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Omni – dominant subtree (founder exec_001). Best in panel: exec_218 (0.9584). 75 of the first 300 iterations passed the MoI gate (25%); failed-leaf nodes are pruned for readability. exec_195 0.9883 exec_115 exec_226 exec_216 0.9886 exec_224 0.9887 exec_194 0.9894 0.990 0.995 1.000 1.005 1.010 val_bpb (lower is better) Curiosity — subtree containing exec_226 (novelty = 2) founder best failed mutation cross… view at source ↗
Figure 12
Figure 12. Figure 12: Curiosity – subtree rooted at exec_115 (32 descendants surviving the 300-iteration cap, depth 9). Contains exec_226 (val_bpb = 0.9873, novelty score 2 verified), the strongest curiosity Pareto idea – ringed gold as the in-panel best. The full subtree from the global founder exec_005 has hundreds of descendants and is too dense for a single page; we crop to the depth-9 ancestral subtree of exec_226. 38 [P… view at source ↗
Figure 13
Figure 13. Figure 13: Per-island Island Search progress on Discogen-OnPolicyRL [PITH_FULL_IMAGE:figures/full_fig_p041_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Linear (Greedy) – running-best timeline. Successive best updates within the first 300 iterations on Discogen-OnPolicyRL. Stateful top-K linear isn’t a tree: every iteration’s first parent is the current best. 43 [PITH_FULL_IMAGE:figures/full_fig_p043_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: MAP-Elites – dominant subtree (founder exec_057). Best in panel: exec_151 (1.5816). Cell-targeted MAP-Elites again produces a wider, shallower tree than chain-based strategies; the dominant tree within 300 iterations is depth 7 with 31 valid descendants. exec_230 1.5610 exec_041 exec_123 1.5078 exec_179 1.4960 exec_162 1.4933 1.36 1.38 1.40 1.42 1.44 1.46 1.48 baseline_normalized_mean (higher is better) G… view at source ↗
Figure 16
Figure 16. Figure 16: Go-Explore – representative subtree rooted at exec_041 (38 descendants, depth 11, 7 branch nodes). Contains the global best exec_230 (1.5610). The full dominant subtree at iter ≤ 300 has 171 descendants at depth 13 and is too dense for a single page. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Islands – one panel per island ( N = 8, 2 × 4 grid). Each panel shows the per-island dominant subtree within the first 300 iterations; only the founder and best-in-panel are labelled to keep the figure readable. Migration edges (purple dashed) cross island boundaries; crossover edges (orange curved) are within-island. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Omni – dominant subtree (founder exec_000). Best in panel: exec_127 (1.1958). Failed-leaf nodes pruned for readability. exec_032 1.1234 exec_003 exec_023 1.0056 exec_024 1.0045 exec_021 0.9531 0.2 0.4 0.6 0.8 1.0 baseline_normalized_mean (higher is better) Curiosity — representative subtree (Discogen-OnPolicyRL) founder best failed mutation crossover migration [PITH_FULL_IMAGE:figures/full_fig_p046_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Curiosity – representative subtree rooted at exec_003 (32 descendants surviving the 300-iteration cap, depth 12). Even after the cap, the full dominant subtree from the global founder exec_013 has 184 descendants at depth 26 and is too dense for a single page; we sub-root at the next-best off-trunk founder. 46 [PITH_FULL_IMAGE:figures/full_fig_p046_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Per-island Island Search progress on Model Unlearning [PITH_FULL_IMAGE:figures/full_fig_p047_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Linear (Greedy) – running-best timeline. Successive best updates within the first 300 iterations on Model Unlearning. Stateful top-K linear isn’t a tree: every iteration’s first parent is the current best. exec_262 0.9702 exec_030 exec_289 0.9667 exec_102 0.9627 exec_296 0.9571 0.84 0.86 0.88 0.90 0.92 0.94 0.96 accuracy (higher is better) MAP-Elites — dominant subtree (Model Unlearning) founder best fail… view at source ↗
Figure 22
Figure 22. Figure 22: MAP-Elites – dominant subtree. Cell-targeted MAP-Elites produces a wide, shallow tree on Model Unlearning: only a handful of ideas clear the WMDP-cyber accuracy baseline, so the empty-cell bias keeps starting fresh from the seed code and the dominant tree within 300 iterations is depth 4 with 24 valid descendants. 49 [PITH_FULL_IMAGE:figures/full_fig_p049_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Go-Explore – dominant subtree (depth 3, 17 valid descendants). Same wide-and-shallow shape as cell-targeted MAP-Elites: the score/visit-weighted cell sampler still has to start from the seed when no cell is populated, which is most of the budget at this difficulty. 50 [PITH_FULL_IMAGE:figures/full_fig_p050_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Islands – one panel per island ( N = 8, 2 × 4 grid). Each panel shows the per-island dominant subtree within the first 300 iterations; only the founder and best-in-panel are labelled to keep the figure readable. Migration edges (purple dashed) cross island boundaries; crossover edges (orange curved) are within-island. 51 [PITH_FULL_IMAGE:figures/full_fig_p051_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Omni – dominant subtree (69 valid descendants, depth 7). Failed-leaf nodes pruned for readability. exec_252 0.9748 exec_012 exec_142 0.9445 exec_039 0.9430 exec_148 0.9403 0.82 0.84 0.86 0.88 0.90 0.92 accuracy (higher is better) Curiosity — dominant subtree (Model Unlearning) founder best failed mutation crossover migration [PITH_FULL_IMAGE:figures/full_fig_p052_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Curiosity – dominant subtree (240 valid descendants, depth 17). Curiosity is the only Model Unlearning strategy whose dominant tree is deep rather than wide-and-shallow: steady-state sampling around the learning-progress signal repeatedly returns to the same neighborhood, generating long chains within the 300-iteration budget. 52 [PITH_FULL_IMAGE:figures/full_fig_p052_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Cumulative auditor-verdict breakdown by iteration — NanoGPT. 54 [PITH_FULL_IMAGE:figures/full_fig_p054_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Cumulative auditor-verdict breakdown by iteration — On-Policy RL. 55 [PITH_FULL_IMAGE:figures/full_fig_p055_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Cumulative auditor-verdict breakdown by iteration — Model Unlearning. 0 50 100 150 200 250 300 Iteration 0 50 100 150 200 Cumulative lost iterations Lost iterations (training crash + timeout + judge errored) Greedy Islands MAP-Elites Go-Explore Omni Curiosity (a) NanoGPT. 0 50 100 150 200 250 300 Iteration 0 50 100 150 200 Cumulative lost iterations Lost iterations (training crash + timeout + judge errore… view at source ↗
Figure 30
Figure 30. Figure 30: Cumulative lost iterations per strategy. Iterations are “lost” when the loop spends a slot without producing a scored, judge-passing idea (training crash, timeout, or judge error). 56 [PITH_FULL_IMAGE:figures/full_fig_p056_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Cumulative MoI-abandoned iterations for Omni (the only strategy that applies the MoI gate). 57 [PITH_FULL_IMAGE:figures/full_fig_p057_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: NanoGPT technique coverage. Component × approach heatmaps, one panel per strategy. 58 [PITH_FULL_IMAGE:figures/full_fig_p058_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: On-Policy RL technique coverage. Component × approach heatmaps, one panel per strategy. 59 [PITH_FULL_IMAGE:figures/full_fig_p059_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Model Unlearning technique coverage. Component × approach heatmaps, one panel per strategy. 60 [PITH_FULL_IMAGE:figures/full_fig_p060_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Strategy idea-space separation. Per-task K × K excess-cosine-distance matrices over Gemini-embedded idea texts (0 = clouds overlap, deeper red = clouds well separated). 61 [PITH_FULL_IMAGE:figures/full_fig_p061_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: Per-strategy quality vs. novelty scatter on NanoGPT (val_bpb, lower is better) [PITH_FULL_IMAGE:figures/full_fig_p062_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: Per-strategy quality vs. novelty scatter on On-Policy RL (baseline-normalized score, higher is better). 62 [PITH_FULL_IMAGE:figures/full_fig_p062_37.png] view at source ↗
Figure 38
Figure 38. Figure 38: Per-strategy quality vs. novelty scatter on Model Unlearning (baseline-normalized score, higher is better). 63 [PITH_FULL_IMAGE:figures/full_fig_p063_38.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Fetch-then-Explore: Decoupling Selection from Extraction over a Persistent Workspace for Search Agents

    cs.AI 2026-08 conditional novelty 6.0

    Fetch-then-Explore, which stores fetched pages in a per-question workspace and extracts evidence on demand with grep/read, beats visit-and-read and browsing baselines on BrowseComp across three LLM backbones.

  2. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    cs.AI 2026-07 conditional novelty 6.0

    A survey of 1,250 papers organizes AI self-improvement along two axes—what is improved and loop closure—finding that demonstrated self-improvement strength tracks a verification hierarchy from formal verifiers down to...

Reference graph

Works this paper leans on

65 extracted references · 65 canonical work pages · cited by 2 Pith papers · 16 internal anchors

  1. [1]

    GQA: training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, De...

    work page doi:10.18653/v1/2023.emnlp-main.298 2023

  2. [2]

    openevolve: Open-source implementation of alphaevolve

    Algorithmic Superintelligence. openevolve: Open-source implementation of alphaevolve. GitHub repository, 2025. URL https://github.com/algorithmicsuperintelligence/ openevolve. Accessed: 2026-05-06

    work page 2025

  3. [5]

    Never give up: Learning directed exploration strategies.CoRR, abs/2002.06038, 2020

    Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies.CoRR, abs/2002.06038, 2020. URLhttps://arxiv.org/abs/2002.06038

    work page arXiv 2002

  4. [7]

    Exploration by Random Network Distillation

    Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation.CoRR, abs/1810.12894, 2018. URL http://arxiv.org/abs/1810. 12894

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  5. [10]

    Angelica Chen, David Dohan, and David R. So. Evoprompting: Language mod- els for code-level neural architecture search. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, Ne...

    work page 2023

  6. [11]

    Robots that can adapt like animals.Nat., 521(7553):503–507, 2015

    Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals.Nat., 521(7553):503–507, 2015. doi: 10.1038/NATURE14422. URL https://doi.org/10.1038/nature14422

    work page doi:10.1038/nature14422 2015

  7. [13]

    Nature , volume =

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore.Nat., 590(7847):580–586, 2021. doi: 10.1038/S41586-020-03157-9. URL https://doi.org/10.1038/s41586-020-03157-9

    work page doi:10.1038/s41586-020-03157-9 2021

  8. [14]

    Diversity is All You Need: Learning Skills without a Reward Function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function.CoRR, abs/1802.06070, 2018. URL http://arxiv.org/abs/1802.06070. 16

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  9. [16]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Forty-first International Conference on Machine Learni...

    work page 2024

  10. [17]

    Alexander D. Goldie, Zilin Wang, Adrian Hayler, Deepak Nathani, Edan Toledo, Ken Tham- piratwong, Aleksandra Kalisz, Michael Beukman, Alistair Letcher, Shashank Reddy, Clarisse Wibault, Theo Wolf, Charles O’Neill, Uljad Berdica, Nicholas Roberts, Saeed Rahmani, Hannah Erlebach, Roberta Raileanu, Shimon Whiteson, and Jakob N. Foerster. Proce- dural generat...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [19]

    All that glitters is not novel: Plagiarism in AI generated research

    Tarun Gupta and Danish Pruthi. All that glitters is not novel: Plagiarism in AI generated research. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, 2025. URLhttps://arxiv.org/abs/2502.16487

    work page arXiv 2025

  12. [20]

    AIRA_2: Overcoming Bottlenecks in AI Research Agents

    Karen Hambardzumyan, Nicolas Mario Baldwin, Edan Toledo, Rishi Hazra, Michael Kuchnik, Bassel Al Omari, Thomas Foster, Anton Protopopov, Jean-Christophe Gagnon-Audet, Ishita Mediratta, Kelvin Niu, Michael Shvartsman, Alisia Maria Lupidi, Alexis Audran-Reiss, Parth Pathak, Tatiana Shavrina, Despoina Magka, Hela Momand, Derek Dunfield, Nicola Cancedda, Pont...

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URLhttps://openreview.net/forum?id=d7KBjmI3GmQ

    work page 2021

  14. [23]

    Test-time learning for large language models

    Jinwu Hu, Zitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanqing Li, and Mingkui Tan. Test-time learning for large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Forty-second International Conference on Machine Learning, ICML 202...

    work page 2025

  15. [24]

    Mlagentbench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, ...

    work page 2024

  16. [25]

    URLhttps://proceedings.mlr.press/v235/huang24y.html. 17

    work page

  17. [26]

    arXiv preprint arXiv:2510.22954 , year=

    Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. Artificial hivemind: The open-ended homogeneity of language models (and beyond).CoRR, abs/2510.22954, 2025. doi: 10.48550/ ARXIV .2510.22954. URLhttps://doi.org/10.48550/arXiv.2510.22954

    work page doi:10.48550/arxiv.2510.22954 2025

  18. [27]

    AIDE: AI-Driven Exploration in the Space of Code

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: ai-driven exploration in the space of code.CoRR, abs/2502.13138,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [30]

    modded-nanogpt: NanoGPT (124m) in 90 seconds

    Keller Jordan. modded-nanogpt: NanoGPT (124m) in 90 seconds. https://github.com/ KellerJordan/modded-nanogpt, 2024. URL https://github.com/KellerJordan/ modded-nanogpt

    work page 2024

  20. [31]

    autoresearch: AI agents running research on single-GPU nanochat train- ing automatically

    Andrej Karpathy. autoresearch: AI agents running research on single-GPU nanochat train- ing automatically. GitHub repository, 2026. URL https://github.com/karpathy/ autoresearch. Accessed: 2026-05-01

    work page 2026

  21. [32]

    To- wards unbounded machine unlearning

    Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. To- wards unbounded machine unlearning. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New...

    work page 2023

  22. [34]

    Gemini Embedding: Generalizable Embeddings from Gemini

    Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [35]

    Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelty alone.Evol. Comput., 19(2):189–223, 2011. doi: 10.1162/EVCO\_A\_00025. URL https://doi.org/10.1162/EVCO_a_00025

    work page doi:10.1162/evco 2011

  24. [37]

    Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-V oss, Cort B. Breuer, Andy ...

    work page 2024

  25. [38]

    An intriguing failing of convolutional neural networks and the coordconv solution

    Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Ja- son Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,Advances in Neural Information Processing System...

    work page arXiv 2018

  26. [39]

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery.CoRR, abs/2408.06292,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [41]

    Intelligent go-explore: Standing on the shoul- ders of giant foundation models

    Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoul- ders of giant foundation models. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=apErWGzCAA

    work page 2025

  28. [42]

    Illuminating search spaces by mapping elites

    Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites.CoRR, abs/1504.04909, 2015. URLhttp://arxiv.org/abs/1504.04909

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  29. [45]

    Does writing with language models reduce content diversity? InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLhttps://openreview.net/forum? id=Feiz5HtCD0

    work page 2024

  30. [46]

    Continual lifelong learning with neural networks: A review,

    German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review.Neural Networks, 113:54–71, 2019. doi: 10.1016/J.NEUNET.2019.01.012. URL https://doi.org/10.1016/j.neunet.2019. 01.012

    work page doi:10.1016/j.neunet.2019.01.012 2019

  31. [47]

    Efros, and Trevor Darrell

    Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supervised prediction. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Proceedings of Machine Learning Research, pages 2778–2787. PML...

    work page 2017

  32. [48]

    doi: 10.3389/frobt.2016.00040

    Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolutionary computation.Frontiers Robotics AI, 3:40, 2016. doi: 10.3389/FROBT.2016.00040. URLhttps://doi.org/10.3389/frobt.2016.00040. 19

    work page doi:10.3389/frobt.2016.00040 2016

  33. [49]

    Pawan Kumar, Emilien Dupont, Francisco J

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models.Nat., 625(7995):468–475, 2024. doi: 10.1038/ S41586-023-06924-...

    work page doi:10.1038/s41586-023-06924-6 2024

  34. [51]

    Adaptive confidence and adaptive curiosity.Forschungsberichte, TU Munich, FKI 149 91:1–9, 1991

    Jürgen Schmidhuber. Adaptive confidence and adaptive curiosity.Forschungsberichte, TU Munich, FKI 149 91:1–9, 1991. URLhttps://d-nb.info/920717624

    work page arXiv 1991

  35. [52]

    Formal theory of creativity, fun, and intrinsic motivation (1990-2010)

    Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Trans. Auton. Ment. Dev., 2(3):230–247, 2010. doi: 10.1109/TAMD.2010.2056368. URL https://doi.org/10.1109/TAMD.2010.2056368

    work page doi:10.1109/tamd.2010.2056368 1990

  36. [53]

    High-Dimensional Continuous Control Using Generalized Advantage Estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors,4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://a...

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  37. [54]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017. URL http://arxiv.org/ abs/1707.06347

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  38. [55]

    Fast Transformer Decoding: One Write-Head is All You Need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.CoRR, abs/1911.02150, 2019. URLhttp://arxiv.org/abs/1911.02150

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  39. [56]

    GLU Variants Improve Transformer

    Noam Shazeer. GLU variants improve transformer.CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  40. [57]

    Smith and Chiyuan Zhang , year=

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models, 2024. URLhttps://arxiv.org/abs/2407.06460

    work page arXiv 2024

  41. [58]

    Towards execution-grounded automated ai research, 2026

    Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candès, Diyi Yang, and Tatsunori Hashimoto. Towards execution-grounded automated ai research, 2026. URLhttps://arxiv.org/abs/ 2601.14525

    work page arXiv 2026

  42. [59]

    David Silver and Richard S. Sutton. Welcome to the era of experience, 2025. URL https://storage.googleapis.com/deepmind-media/Era-of-Experience/The% 20Era%20of%20Experience%20Paper.pdf. To appear inDesigning an Intelligence, ed. G. Konidaris, MIT Press

    work page 2025

  43. [60]

    Skydiscover: A flexible framework for AI-driven scientific and algo- rithmic discovery

    SkyDiscover Authors. Skydiscover: A flexible framework for AI-driven scientific and algo- rithmic discovery. GitHub repository, 2026. URL https://github.com/skydiscover-ai/ skydiscover. Accessed: 2026-05-07

    work page 2026

  44. [61]

    Stanley and Joel Lehman.Why Greatness Cannot Be Planned - The Myth of the Objective

    Kenneth O. Stanley and Joel Lehman.Why Greatness Cannot Be Planned - The Myth of the Objective. Springer, 2015. ISBN 978-3-319-15523-4. doi: 10.1007/978-3-319-15524-1. URL https://doi.org/10.1007/978-3-319-15524-1

    work page doi:10.1007/978-3-319-15524-1 2015

  45. [65]

    On the planning abilities of large language models - A critical investigation

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models - A critical investigation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors,Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information...

    work page 2023

  46. [66]

    Group-evolving agents: Open-ended self-improvement via experience sharing, 2026

    Zhaotian Weng, Antonis Antoniades, Deepak Nathani, Zhen Zhang, Xiao Pu, and Xin Eric Wang. Group-evolving agents: Open-ended self-improvement via experience sharing, 2026. URLhttps://arxiv.org/abs/2602.04837

    work page arXiv 2026

  47. [68]

    https://doi.org/10.48550/arXiv

    Xu Yang, Xiao Yang, Shikai Fang, Yifei Zhang, Jian Wang, Bowen Xian, Qizheng Li, Jingyuan Li, Minrui Xu, Yuante Li, Haoran Pan, Yuge Zhang, Weiqing Liu, Yelong Shen, Weizhu Chen, and Jiang Bian. R&d-agent: An llm-agent framework towards autonomous data science.CoRR, abs/2505.14738, 2025. doi: 10.48550/ARXIV .2505.14738. URLhttps://arxiv.org/abs/ 2505.14738v2

    work page internal anchor Pith review doi:10.48550/arxiv 2025

  48. [69]

    MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments

    Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments.arXiv preprint arXiv:1903.03176, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1903

  49. [70]

    Stanley, and Jeff Clune

    Jenny Zhang, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. OMNI: open-endedness via models of human notions of interestingness.CoRR, abs/2306.01711, 2023. doi: 10.48550/ ARXIV .2306.01711. URLhttps://doi.org/10.48550/arXiv.2306.01711

    work page doi:10.48550/arxiv.2306.01711 2023

  50. [71]

    Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [72]

    Hyperagents, 2026

    Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. Hyperagents.arXiv preprint arXiv:2603.19461, 2026

    work page arXiv 2026

  52. [73]

    Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning.CoRR, abs/2404.05868, 2024. doi: 10.48550/ ARXIV .2404.05868. URLhttps://doi.org/10.48550/arXiv.2404.05868

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2404.05868 2024

  53. [74]

    produce a graded result

    Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obeid, and Zhenzhong Lan. Value residual learning, 2025. URLhttps://arxiv.org/abs/2410.17897. A Supplementary material A.1 Problem-setting details Refers to §3.1. 21 Per-task configuration. • NanoGPT: fulltrain.py model config (DEPTH=8 with derived nembd = 512, nhead = 4 via ASPECT_RATIO=64 + HEAD_DIM=128); TI...

    work page arXiv 2025

  54. [75]

    Termination: The ball hits the bottom of the screen

    work page

  55. [76]

    forget set

    Truncation: The length of the episode reaches max_steps_in_episode (default: 1000) 69 STATE SPACE The state consists of: - ball_y: vertical position of ball (0-9) - ball_x: horizontal position of ball (0-9) - ball_dir: direction of ball movement (0-3) - pos: paddle position (0-9) - brick_map: 10x10 binary map of bricks - strike: boolean indicating if ball...

    work page

  56. [77]

    Hazardous Knowledge Removal: Reduce the model’s ability to answer questions about sensitive cybersecurity topics

    work page

  57. [78]

    General Capability Preservation: Maintain strong performance on general language tasks EVALUATION METRICS The task uses 2 evaluation metrics:

    work page

  58. [79]

    wmdp_cyber/acc: Accuracy on the 1,987-question WMDP Cyber multiple-choice benchmark (lower is better)

    work page

  59. [80]

    min" %}lower beats{% else %}higher beats{% endif %}), baseline={{

    mmlu_stem/acc: Accuracy on the STEM subsection of Massive Multitask Language Understanding (MMLU) benchmark containing 3,153 questions, tested via multiple-choice questions from lm_eval harness ( higher is better) The goal is to optimize for both of them, from which a final score will be computed. MODEL DESCRIPTION Qwen 2.5 1.5B Instruct is a compact inst...

    work page 2024

  60. [81]

    grep -nE -f /workspace/.fabrication_patterns /workspace/run/agent.log

    work page

  61. [82]

    wc -c /workspace/run/{{ stdout }} && tail -25 /workspace/run/{{ stdout }}

    work page

  62. [83]

    diff -ur --no-dereference --exclude-from=/workspace/.diff_excludes /workspace/task/{{ diff_scope }} /workspace/run/{{ diff_scope }}

    work page

  63. [84]

    find /workspace/run/.venv_extra -type f -print 2>/dev/null

    work page

  64. [85]

    Symbolic links X and Y differ

    cat /workspace/run/notes.md Step 3 uses --no-dereference: symlinks are compared as links (by their target path), not followed. When both sides point to the same target the diff is silent – no stanza is emitted. Per-domain dirs (e.g., MinAtar/Breakout/) often symlink back to {{ editable }}/, so a legitimate edit to {{ editable }}/<file> produces ONE stanza...

    work page 2048

  65. [86]

    Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...

    work page