Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

EXP-Bench: Can AI Conduct AI Research Experiments?

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current AI agents complete only 0.5% of end-to-end research experiments, EXP-Bench finds.

desk verdict A genuinely useful benchmark with a real internal inconsistency in its headline 0.5% number; the qualitative conclusion survives, but the quantitative claims need to be cleaned up before they can be trusted. read the letter →

arxiv 2505.24785 v2 pith:F56VSANH submitted 2025-05-30 cs.AI

Patrick Tser Jern Kon , Jiachen Liu , Xinyi Zhu , Qiuyi Ding , Jingjia Peng , Jiarong Xing , Yibo Huang , Yiming Qiu
show 5 more authors
Jayanth Srinivasa Myungjin Lee Mosharaf Chowdhury Matei Zaharia Ang Chen
This is my paper
classification cs.AI
keywords EXP-BenchAIresearchagentsend-to-endexperimentationLLM-as-a-judgebenchmarkconstructionexperimentaldesignevaluationautonomousmachinelearningconjunctivemetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EXP-Bench makes a measurement claim: that complete AI research experiments, not just isolated coding or reasoning subtasks, can be reconstructed from published papers and their codebases into 461 graded tasks, and that under this measurement the best current agents almost always fail. Given a research question, a high-level method, and starter code with key files masked, an agent must design the experiment, implement it, execute it, and write a valid conclusion. Individual phases reach roughly 20-35% for the stronger agent-model pairs, but the strict all-criteria score is 0.5% for the strongest configuration. A sympathetic reader should care because the benchmark converts 'can AI do AI research?' into a concrete, per-phase measurement and shows exactly where current agents break down.

What carries the argument

The load-bearing mechanism is the semi-automated curation pipeline combined with a conjunctive evaluation judge. Curation filters influential NeurIPS/ICLR 2024 papers with active codebases; a multimodal LLM extraction pass turns each paper into research questions and expected outcomes, a tool-augmented implementation agent locates and validates the script chain that realizes each task, and AST tracing converts that chain into step-by-step implementation ground truth, with lightweight human review and containerized execution closing the loop. Evaluation then combines an o3-mini LLM judge, which scores design, implementation, and conclusion against the ground truth and runs an integrity monitor over agent logs, with a Code Execution Validator that reruns the agent's code in a clean environment. The key measurement identity is the conjunctive metric $M\cdot C\cdot D\cdot I\cdot E$: every component must be true for a task to count, which is what makes the 0.5% figure the honest summary rather than the higher partial scores.

What would settle it

Sample roughly 50 EXP-Bench tasks and have two independent expert researchers, blind to the ground truth, score the best agent's outputs on design, implementation, execution, and conclusion; if their scores agree poorly with EXP-Bench's LLM judge, or if their all-criteria pass rate is far above 0.5% on the same runs, the benchmark is not yet measuring experiment quality as intended.

Watch

Extended reading notes

Core claim

The paper's central claim is that today's LLM-based agents cannot yet conduct end-to-end research experiments as EXP-Bench defines them. EXP-Bench comprises 461 tasks extracted from 51 NeurIPS and ICLR 2024 papers, each task supplying a research question, a high-level method, and a code repository with task-relevant scripts masked; the ground truth for each task pairs a structured experimental design with the code changes needed to realize it and a conclusion matching the original paper's findings. Agents are scored on design correctness (D), implementation correctness (I), executable code (E), and conclusion correctness (C). Across OpenHands and IterativeAgent paired with several frontier LLMs, partial scores on D, I, and C reach about 20-35% for the strongest configurations, but the conjunctive metric requiring D, I, C, and executability at once is 0.5% for the best agent and near zero for most others. The paper interprets this as evidence that agents can produce plausible pieces of an experiment while failing to assemble, run, and interpret a complete one, and it uses the failure traces to quantify the bottlenecks: missing implementation components, environment and dependency misconfiguration, design-variable errors, and missing or incorrect conclusions.

Load-bearing premise

The entire measurement rests on the assumption that the LLM-extracted task formulations and ground truths, checked only by lightweight human review, faithfully capture what the source papers actually did, and that the o3-mini judge scores agent outputs accurately; if either fails, the reported scores measure agreement with the pipeline rather than ability to conduct research.

Editorial extensions

If this is right

  • If the benchmark is a faithful measurement, any claim that an agent 'does AI research' should be reported with all-criteria success rates, not with partial phase scores alone.
  • The failure taxonomy points to the places to concentrate effort: implementation completeness (about 39.7% of failures), environment and dependency setup (about 29.4%), script-level errors (about 23.8%), and missing or incorrect conclusions (about 26.2%).
  • Because conjunctive metrics score much lower and vary less than individual metrics, future evaluations should require every phase to pass before crediting an agent.
  • EXP-Bench's fine-grained subtask labels can be used as step-by-step supervision for training research agents, not only as a final exam.
  • The public release of the dataset lets the field track progress on a fixed, realistic task set rather than on ad-hoc agent demos.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 0.5% figure is a joint statement about the agents and about the benchmark's LLM-extracted ground truth; a human expert rescoring of a random sample would reveal which half of that statement is driving the result.
  • Because the tasks come from public papers and codebases, agents with strong memorization of those repositories could be scored higher than their reasoning would justify; a version that mutates experimental details would test generalization instead of recall.
  • The paper does not run human researchers on the same 461 tasks; adding that human baseline would calibrate whether 0.5% means 'agents are far from human' or 'the tasks are extremely difficult for anyone.'
  • The per-subtask scores suggest an immediate extension the paper only names as future work: use them as dense rewards for reinforcement learning, so agents improve one experimental phase at a time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. EXP-Bench introduces a benchmark of 461 AI-research experiment tasks extracted from 51 NeurIPS/ICLR 2024 papers and their codebases, together with a semi-automated curation pipeline and a multi-metric evaluation framework (design D, implementation I, execution E, conclusion C, plus conjunctive metrics including All·E✓). The paper reports that state-of-the-art agents such as OpenHands and IterativeAgent achieve only 0.5% success on the complete, executable-experiment criterion for the best configuration (OpenHands + o3-mini), and it characterizes prevalent failure modes across design, implementation, execution, and conclusion phases. The central claim is that current LLM agents are far from capable of end-to-end AI research experimentation, and that EXP-Bench provides a valid instrument for measuring that gap.

Significance. If the benchmark construction and evaluation are sound, EXP-Bench would be a valuable community asset: it is open-sourced, it addresses a real gap in end-to-end experimentation rather than isolated coding or reasoning subtasks, and it ships with a detailed error taxonomy grounded in 3,238 raw insights. The design of using published papers plus their code as external ground truth is a genuine strength, and the execution-based E metric provides an objective component that is absent from many LLM-judge-only benchmarks. However, the quantitative headline (0.5%) is currently not robustly interpretable because the paper's own reporting is internally inconsistent about the evaluation denominator, and because both task extraction and judging rest on unvalidated LLM components with only lightweight human review. These issues are load-bearing for the main claim, although the qualitative direction — agents perform poorly on end-to-end experiments — is likely robust.

major comments (5)
  1. [§4.2, Fig. 6b vs. Table 1] The reported numbers for the top configuration are internally inconsistent. Table 1 gives OpenHands + o3-mini All·E✓ = 0.5% with #E = 420 tasks execution-checked, implying about 2 successful tasks. Fig. 6b and §4.2 state that on the subset of tasks for which execution was run, only 20.6% pass the initial monitor check M, and M·C·D·I·E is 0.2%. If M has a 20.6% pass rate, at most ~95 of the 461 tasks could be execution-checked, not 420. Moreover, M·C·D·I·E = 0.2% would correspond to less than one task on a 461-task denominator, while Table 1 reports 0.5% for the same configuration. The paper must specify the exact denominator and subset for each metric; as written, the reader cannot tell whether All·E✓ is computed over all 461 tasks, over the #E tasks, or over a monitor-gated subset. This must be fixed before the headline number can be interpreted.
  2. [§3.2, Stage 3 and §4.1, Appendix H] The ground-truth extraction and the judge are both LLM-based, with only 'lightweight human review' and no validation of the o3-mini judge against human experts. The design and implementation ground truths are extracted by an LLM pipeline, and the D, I, and C scores are produced by an o3-mini judge from the prompts in Appendix H. No inter-annotator agreement, human-judge agreement, or error analysis of the judge itself is reported. Since every reported score, including the 0.5% headline, passes through these components, the paper should report (a) the fraction of tasks whose ground truth was manually corrected, (b) a sample of judge decisions reviewed by human experts with agreement rates, and (c) an analysis of judge sensitivity to prompt and model choice. Without this, the scores may measure agreement with an LLM-consensus ground truth rather than experimentation ability.
  3. [§4.1, Table 1, #E column] The execution evaluation is applied to a non-uniform subset whose size varies by model (#E ranges from 56 to 420), and the paper does not define how missing E scores enter All·E✓. If All·E✓ is averaged over all 461 tasks with non-executed tasks treated as failures, then comparing models with different #E values conflates execution coverage with agent ability; if it is averaged only over executed tasks, then the 0.5% and 0.4% figures are not directly comparable across rows. The paper must state the exact aggregation rule and, ideally, report results on a common denominator of tasks that were executed for all models, or explicitly model the missingness.
  4. [§4.2, Fig. 5 and §4.1, conclusion evaluation] The conclusion metric C is scored as a binary 'correct/incorrect' semantic match against a ground-truth string, and the paper itself notes in Fig. 5 that C exhibits high variance because agents 'can produce plausible but unfounded conclusions without a valid experimental foundation.' Since All·E✓ requires C to be correct but does not require the conclusion to be derived from the agent's own executed outputs, a task can pass All·E✓ with a correct-sounding but unsupported conclusion. The evaluation should either condition C on the executed results or add an explicit check that the conclusion is entailed by, or consistent with, the execution outputs.
  5. [§4.1, Monitor M and execution subset] The monitor check M is an integrity filter (paper access, git operations, fake data), not a correctness metric, yet it is used to gate which traces receive execution evaluation. The paper should report how many tasks were excluded by each monitor criterion, separately from correctness failures. This matters because the discrepancy between the 20.6% M pass rate in Fig. 6b and the 420 execution-checked tasks in Table 1 suggests that the monitor was either applied differently in the two analyses or that the subset definition changed, and the current text does not allow the reader to reconcile them.
minor comments (5)
  1. [Abstract and §1] The abstract and introduction say 'leading AI agents' were evaluated, but only two agent frameworks (OpenHands and IterativeAgent) are tested; the wording should be narrowed to reflect the actual scope.
  2. [Table 3 vs. Table 1 and Appendix D] The column header in Table 3 reads 'All✓·E' while Table 1 uses 'All·E✓'; the notation should be unified. In addition, Appendix D's addendum (Table 6) updates the IA+3.5 Haiku Applications I score from 18.0 to 32.3 and other values, but the main-text Table 3 still shows the old values; the paper should reconcile these or clearly mark which table is authoritative.
  3. [§3.2, Stage 1] The paper-selection criteria are described qualitatively (citation counts, GitHub stars/forks) without reporting the actual thresholds or the number of papers passing each filtering stage; providing these numbers would improve reproducibility of the dataset construction.
  4. [§4.2 and Fig. 6a] The cost-time analysis reports time in minutes and notes that the soft 40-minute timeout was sometimes exceeded, but Appendix I.2 shows maximum times over 400 minutes for IterativeAgent; the paper should clarify why the timeout was not enforced and whether these outliers affect the 'maximum of 40 minutes per task' statement in §4.2.
  5. [Appendix G] The failure-category table contains a duplicated entry ('Missing Visualization Impl.' appears twice under setup) and the note that classification was performed by an LLM; the duplicate should be removed and the classification method should be described in more detail, including any human verification of the taxonomy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ground truth is external (papers/code), All·E✓ includes objective execution, and the only self-citation is non-load-bearing.

full rationale

EXP-Bench's evaluation pipeline is not circular by the paper's own construction. The ground truth for each task is anchored in external artifacts: 'Each research task is grounded on an influential AI research paper and its corresponding codebase' (§3), and the implementation ground truth is 'fully based on existing scripts' (§3.2). The headline 0.5% is the All·E✓ metric, which requires D, I, C, and E; E is checked by a separate Code Execution Validator that 'verifies whether the code is executable and produces expected outputs' (§4.1), so the headline is not defined as whatever the LLM judge says. The sole self-citation, Curie [47], appears only as an example in Related Work and is not load-bearing. The paper's use of o3-mini for both task extraction (App. F) and LLM judging (§4.1) is a measurement-validity caveat: D/I/C scores may partly reflect agreement with o3-mini's reading of the papers, but this is not a definitional reduction, because the judge compares against an external ground truth extracted from published papers and code, not against the evaluated agents' outputs. The inconsistent denominators between Fig. 6b and Table 1 are a reporting/reproducibility issue, not a circularity. Accordingly, no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central quantitative claims rest on the fidelity of the LLM-extracted ground truth, the validity of the LLM judge, the representativeness of the selected papers, and the absence of pretraining contamination. None of these are externally verified in the paper beyond execution checks and lightweight human review, so they are listed as domain assumptions that a reader must accept.

free parameters (4)
  • Paper selection thresholds (citation counts, GitHub stars/forks) = not disclosed
    Stage 1 (Section 3.2) filters papers by citation counts and repository activity; the specific thresholds are not given, and they determine which 51 papers and therefore which 461 tasks enter the benchmark.
  • Execution evaluation subset = 56 to 420 tasks per model
    Table 1 reports that only a subset of agent traces was execution-checked (#E), and the execution-based metrics E and All·E✓ are computed over this non-uniform subset rather than all 461 tasks.
  • Per-task timeout = 40 minutes (soft)
    Section 4.1 sets a maximum of 40 minutes per task; the paper notes agents occasionally exceeded this limit, and the timeout affects task completion and therefore all scores.
  • Judge model choice = o3-mini-2025-01-01-preview
    All design, implementation, and conclusion scores are produced by this LLM judge (Section 4.1, Appendix H); scores would likely shift with a different judge, and no human agreement study is provided.
assumptions (4)
  • domain assumption The o3-mini LLM judge provides valid, unbiased scoring of agent design, implementation, and conclusion outputs
    All D, I, C metrics and the final All·E✓ are computed with this judge; no human agreement or calibration study is provided in the paper.
  • domain assumption LLM-extracted task formulations and ground truths, after lightweight human consistency checks, faithfully represent the source papers' experiments
    The fidelity of the benchmark's ground truth is load-bearing for every reported score; human review in Stage 3 is described as 'lightweight' or 'mostly lightweight consistency checks'.
  • domain assumption The selected 51 NeurIPS/ICLR 2024 papers with open-source code and high citation or star counts are representative of influential AI research experimentation
    All 461 tasks derive from this filtered set, so the scope of the paper's conclusions about AI research experiments depends on this representativeness.
  • domain assumption Agents do not exploit parametric memorization of the source papers; the monitor only detects direct file access, not pretraining leakage
    The monitor in Section 4.1 checks for reading the paper PDF, git operations, and fake data, but it cannot detect whether an agent recalls code or results from pretraining; contamination could bias scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EXP-Bench: Can AI Conduct AI Research Experiments?." pith.science (2026). https://pith.science/paper/F56VSANH

@misc{pith2026250524785,
  author       = {Pith},
  title        = {Pith review of: EXP-Bench: Can AI Conduct AI Research Experiments?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F56VSANH}},
  note         = {Machine review of arXiv:2505.24785}
}
read the original abstract

Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments. EXP-Bench is open-sourced at https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.

Figures

Figures reproduced from arXiv: 2505.24785 by the authors.

Figure 1
Figure 1. EXP-Bench evaluates AI agents on research experiment tasks extracted semi-autonomously [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. One AI research task example from ICLR 2024 MogaNet [51]. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. EXP-Bench semi-automated dataset construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Stability Analysis. Metric Stability Analysis. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation of agent performance along cost–time and evaluation metrics. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Task Fields for Example 1. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Evaluation of the design and setup for the Extended Task in Example 1. [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Example 1’s Git diff comparing the masked source file and the agent-reconstructed source [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Error Analysis and Comprehensive Explanation of the agent’s failure to complete the task [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Task fields for Example 2. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: The design and setup evaluation of the extended task in Example 2. [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Example 2’s Git diff of the masked source file and the agent reconstructed source code. In [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]
Figure 14
Figure 14. Figure 14: Stricter metrics reveal lower true correctness. [PITH_FULL_IMAGE:figures/full_fig_p045_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FARS: A Fully Automated Research System Deployed at Scale

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    FARS deployed at scale produced 166 AI/ML papers across 67 topics that received 282 structured human reviews indicating some review-worthy outputs alongside recurring failure modes.

  2. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LLM judges of scientific ideas are measurably swayed by writing style; a style-detecting module reduces but does not remove the bias.

  3. BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?

    cs.CR 2025-10 conditional novelty 6.0 of 10

    An LLM agent generating fabricated papers without experiments gets acceptance-level scores from LLM reviewers up to 82% of the time, and simple integrity-checking mitigations barely beat random.

Reference graph

Works this paper leans on

118 extracted references · 53 canonical work pages · cited by 3 Pith papers

  1. [1]

    https://lmsys

    Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys. org/blog/2023-03-30-vicuna/

    2023

  2. [2]

    F. Alet, J. Lopez-Contreras, J. Koppel, M. Nye, A. Solar-Lezama, T. Lozano-Perez, L. Kaelbling, and J. Tenenbaum. A large-scale benchmark for few-shot program induction and synthesis. InInternational Conference on Machine Learning, pages 175–186. PMLR, 2021

    2021

  3. [3]

    Ashkboos, A

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman. Quarot: Outlier-free 4-bit inference in rotated llms, 2024

    2024

  4. [4]

    J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models.arXiv preprint arXiv:2404.07738, 2024

    arXiv 2024

  5. [5]

    Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V . Do, Y . Xu, and P. Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023

    2023

  6. [6]

    "a is b"

    L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: Llms trained on ""a is b"" fail to learn ""b is a"", 2024

    2024

  7. [7]

    Bettini, A

    M. Bettini, A. Prorok, and V . Moens. Benchmarl: Benchmarking multi-agent reinforcement learning, 2024

    2024

  8. [8]

    Blasiok and P

    J. Blasiok and P. Nakkiran. Smooth ece: Principled reliability diagrams via kernel smoothing, 2023

    2023

Show all 118 references
  1. [9]

    D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    2023

  2. [10]

    Boisvert, M

    L. Boisvert, M. Thakkar, M. Gasse, M. Caccia, T. L. S. D. Chezelles, Q. Cappart, N. Chapados, A. Lacoste, and A. Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks, 2025

    2025

  3. [11]

    A. Bou, M. Bettini, S. Dittert, V . Kumar, S. Sodhani, X. Yang, G. D. Fabritiis, and V . Moens. Torchrl: A data-driven decision-making library for pytorch, 2023

    2023

  4. [12]

    J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. M ˛ adry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2024

    2024

  5. [13]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  6. [14]

    Z. Chen, S. Chen, Y . Ning, Q. Zhang, B. Wang, B. Yu, Y . Li, Z. Liao, C. Wei, Z. Lu, et al. Scienceagent- bench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080, 2024

    arXiv

  7. [15]

    Cheng, A

    C.-A. Cheng, A. Nie, and A. Swaminathan. Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms, 2024

    2024

  8. [16]

    Cheng, T

    P. Cheng, T. Hu, H. Xu, Z. Zhang, Z. Yuan, Y . Dai, L. Han, N. Du, and X. Li. Self-playing adversarial language game enhances llm reasoning, 2025

    2025

  9. [17]

    Chevalier, J

    A. Chevalier, J. Geng, A. Wettig, H. Chen, S. Mizera, T. Annala, M. J. Aragon, A. R. Fanlo, S. Frieder, S. Machado, A. Prabhakar, E. Thieu, J. T. Wang, Z. Wang, X. Wu, M. Xia, W. Xia, J. Yu, J.-J. Zhu, Z. J. Ren, S. Arora, and D. Chen. Language models as science tutors, 2024

    2024

  10. [18]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021

    2021

  11. [19]

    T. M. U. Collaboration, J. Audenaert, M. Bowles, B. M. Boyd, D. Chemaly, B. Cherinka, I. Ciuc ˘a, M. Cranmer, A. Do, M. Grayling, E. E. Hayes, T. Hehir, S. Ho, M. Huertas-Company, K. G. Iyer, M. Jablonska, F. Lanusse, H. W. Leung, K. Mandel, J. R. Martínez-Galarza, P. Melchior...

    2024

  12. [20]

    J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023. 10

    2023

  13. [21]

    T. Dai, B. Wu, P. Liu, N. Li, J. Bao, Y . Jiang, and S.-T. Xia. Periodicity decoupling framework for long-term series forecasting. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  14. [22]

    Y . Du, F. Bai, T. Huang, and B. Zhao. Segvol: Universal and interactive volumetric medical image segmentation, 2025

    2025

  15. [23]

    Elicit: Analyze research papers at superhuman speed, 2025

    Elicit. Elicit: Analyze research papers at superhuman speed, 2025. Accessed: 2025-05-12

    2025

  16. [24]

    Elrefaie, F

    M. Elrefaie, F. Morar, A. Dai, and F. Ahmed. Drivaernet++: A large-scale multimodal car dataset with computational fluid dynamics simulations and deep learning benchmarks, 2025

    2025

  17. [25]

    Y . Fang, N. Zhang, Z. Chen, L. Guo, X. Fan, and H. Chen. Domain-agnostic molecular generation with chemical feedback, 2024

    2024

  18. [26]

    Frieder, L

    S. Frieder, L. Pinchetti, , R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. Petersen, and J. Berner. Mathe- matical capabilities of chatgpt. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, vol...

    2023

  19. [27]

    Gandhi, M

    K. Gandhi, M. Y . Li, L. Goodyear, L. Li, A. Bhaskar, M. Zaman, and N. D. Goodman. Boxinggym: Benchmarking progress in automated experimental design and model discovery, 2025

    2025

  20. [28]

    Gendron, Q

    G. Gendron, Q. Bao, M. Witbrock, and G. Dobbie. Large language models are not strong abstract reasoners.arXiv preprint arXiv:2305.19555, 2023

    2023 arXiv

  21. [29]

    Ghafarollahi and M

    A. Ghafarollahi and M. J. Buehler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning, 2024

    2024

  22. [30]

    Grosnit, A

    A. Grosnit, A. Maraval, J. Doran, G. Paolo, A. Thomas, R. S. H. N. Beevi, J. Gonzalez, K. Khandelwal, I. Iacobacci, A. Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level.arXiv preprint arXiv:2411.03562, 2024

    2024

  23. [31]

    K. Gu, R. Shang, R. Jiang, K. Kuang, R.-J. Lin, D. Lyu, Y . Mao, Y . Pan, T. Wu, J. Yu, et al. Blade: Benchmarking language model agents for data-driven science.arXiv preprint arXiv:2408.09667, 2024

    2024

  24. [32]

    S. Guo, C. Deng, Y . Wen, H. Chen, Y . Chang, and J. Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning.arXiv preprint arXiv:2402.17453, 2024

    2024 arXiv

  25. [33]

    Y . Guo, C. Yang, A. Rao, Z. Liang, Y . Wang, Y . Qiao, M. Agrawala, D. Lin, and B. Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, 2024

    2024

  26. [34]

    S. J. Han, K. J. Ransom, A. Perfors, and C. Kemp. Inductive reasoning in humans and large language models.Cognitive Systems Research, 83:101155, 2024

    2024

  27. [35]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  28. [36]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021

    2021

  29. [37]

    X. Hu, Z. Zhao, S. Wei, Z. Chai, Q. Ma, G. Wang, X. Wang, J. Su, J. Xu, M. Zhu, et al. Infiagent-dabench: Evaluating agents on data analysis tasks.arXiv preprint arXiv:2401.05507, 2024

    2024 arXiv

  30. [38]

    Huang, W

    J.-t. Huang, W. Wang, E. J. Li, M. H. Lam, S. Ren, Y . Yuan, W. Jiao, Z. Tu, and M. Lyu. On the humanity of conversational ai: Evaluating the psychological portrayal of llms. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  31. [39]

    Huang, J

    Q. Huang, J. V ora, P. Liang, and J. Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2023

    2023

  32. [40]

    Huang, J

    Q. Huang, J. V ora, P. Liang, and J. Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2024

    2024

  33. [41]

    Huang, Q

    Z. Huang, Q. Ye, B. Kang, J. Feng, and H. Fan. Classification done right for vision-language pre-training, 2024

    2024

  34. [42]

    Ifargan, L

    T. Ifargan, L. Hafner, M. Kern, O. Alcalay, and R. Kishony. Autonomous llm-driven research—from data to human-verifiable research papers.NEJM AI, 2(1):AIoa2400555, 2025. 11

    2025

  35. [43]

    Jaume, P

    G. Jaume, P. Doucet, A. H. Song, M. Y . Lu, C. Almagro-Pérez, S. J. Wagner, A. J. Vaidya, R. J. Chen, D. F. K. Williamson, A. Kim, and F. Mahmood. Hest-1k: A dataset for spatial transcriptomics and histology image analysis, 2024

    2024

  36. [44]

    B. Jing, H. Stärk, T. Jaakkola, and B. Berger. Generative modeling of molecular dynamics trajectories, 2024

    2024

  37. [45]

    L. Jing, Z. Huang, X. Wang, W. Yao, W. Yu, K. Ma, H. Zhang, X. Du, and D. Yu. Dsbench: How far are data science agents from becoming data science experts?, 2024

    2024

  38. [46]

    Jumper, R

    J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold.nature, 596(7873):583–589, 2021

    2021

  39. [47]

    P. T. J. Kon, J. Liu, Q. Ding, Y . Qiu, Z. Yang, Y . Huang, J. Srinivasa, M. Lee, M. Chowdhury, and A. Chen. Curie: Toward rigorous and automated scientific experimentation with ai agents, 2025

    2025

  40. [48]

    J. Lála, O. O’Donoghue, A. Shtedritski, S. Cox, S. G. Rodriques, and A. D. White. Paperqa: Retrieval- augmented generative agent for scientific research.arXiv preprint arXiv:2312.07559, 2023

    2023 arXiv

  41. [49]

    J. M. Laurent, J. D. Janizek, N. Thakkar, M. Ruzo, M. S. Yao, M. S. Levine, S. G. Rodriques, and A. White. Lab-bench: Measuring capabilities of language models for biology research, 2024

    2024

  42. [50]

    R. Li, T. Patel, Q. Wang, and X. Du. Mlr-copilot: Autonomous machine learning research based on large language models agents.arXiv preprint arXiv:2408.14033, 2024

    2024

  43. [51]

    S. Li, Z. Wang, Z. Liu, C. Tan, H. Lin, D. Wu, Z. Chen, J. Zheng, and S. Z. Li. Moganet: Multi-order gated aggregation network, 2024

    2024

  44. [52]

    X. Li, Z. Huang, F. Xue, and Y . Zhou. Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images, 2024

    2024

  45. [53]

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    2022

  46. [54]

    S. Lin, W. Lin, X. Hu, W. Wu, R. Mo, and H. Zhong. Cyclenet: Enhancing time series forecasting through modeling periodic patterns, 2024

    2024

  47. [55]

    H. Liu, L. Chen, Y . Qiao, C. Lv, and H. Li. Reasoning multi-agent behavioral topology for interactive autonomous driving, 2024

    2024

  48. [56]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  49. [57]

    T. Liu, C. Xu, and J. McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023

    2023

  50. [58]

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y . Su, H. Sun, M. Huang, Y . Dong, and J. Tang. Agentbench: Evaluating llms as agents, 2023

    2023

  51. [59]

    X. Liu, C. Zhou, and S. Huang. 3dgs-enhancer: Enhancing unbounded 3d gaussian splatting with view-consistent 2d diffusion priors, 2024

    2024

  52. [60]

    R. Lou, H. Xu, S. Wang, J. Du, R. Kamoi, X. Lu, J. Xie, Y . Sun, Y . Zhang, J. J. Ahn, H. Fang, Z. Zou, W. Ma, X. Li, K. Zhang, C. Xia, L. Huang, and W. Yin. Aaar-1.0: Assessing ai’s potential to assist research, 2024

    2024

  53. [61]

    C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha. The ai scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    2024 arXiv

  54. [62]

    Y . Lu, Y . Hu, Y . Zhong, D. Wang, Y . Wang, and S. Chen. An extensible framework for open heterogeneous collaborative perception, 2024

    2024

  55. [63]

    Q. Luo, H. Yu, and X. Li. Badam: A memory efficient full parameter optimization method for large language models, 2024. 12

    2024

  56. [64]

    B. P. Majumder, H. Surana, D. Agarwal, B. D. Mishra, A. Meena, A. Prakhar, T. V ora, T. Khot, A. Sab- harwal, and P. Clark. Discoverybench: Towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725, 2024

    2024 arXiv

  57. [65]

    Mirchandani, F

    S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng. Large language models as general pattern machines.arXiv preprint arXiv:2307.04721, 2023

    2023 arXiv

  58. [66]

    Mitchener, J

    L. Mitchener, J. M. Laurent, B. Tenmann, S. Narayanan, G. P. Wellawatte, A. White, L. Sani, and S. G. Rodriques. Bixbench: a comprehensive benchmark for llm-based agents in computational biology.arXiv preprint arXiv:2503.00096, 2025

    2025

  59. [67]

    Moskvichev, V

    A. Moskvichev, V . V . Odouard, and M. Mitchell. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain.arXiv preprint arXiv:2305.07141, 2023

    2023 arXiv

  60. [68]

    Narayanan, J

    S. Narayanan, J. D. Braza, R.-R. Griffiths, M. Ponnapati, A. Bou, J. Laurent, O. Kabeli, G. Wellawatte, S. Cox, S. G. Rodriques, et al. Aviary: training language agents on challenging scientific tasks.arXiv preprint arXiv:2412.21154, 2024

    2024 arXiv

  61. [69]

    Nathani, L

    D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V . Moens, A. Budhiraja, D. Magka, V . V orotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. Foerster, Y . Bachrach, W. Y . Wang, and R. Raileanu. Mlgym: A new framework and benchmark for advancing ai rese...

    2025

  62. [70]

    Ohana, M

    R. Ohana, M. McCabe, L. Meyer, R. Morel, F. J. Agocs, M. Beneitez, M. Berger, B. Burkhart, K. Burns, S. B. Dalziel, D. B. Fielding, D. Fortunato, J. A. Goldberg, K. Hirashima, Y .-F. Jiang, R. R. Kerswell, S. Maddu, J. Miller, P. Mukhopadhyay, S. S. Nixon, J. Shen, R. Watteaux...

    2025

  63. [71]

    P. Qi, X. Wan, G. Huang, and M. Lin. Zero bubble (almost) pipeline parallelism. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  64. [72]

    S. Qi, S. Chen, Y . Li, X. Kong, J. Wang, B. Yang, P. Wong, Y . Zhong, X. Zhang, Z. Zhang, N. Liu, W. Wang, Y . Yang, and S.-C. Zhu. Civrealm: A learning and reasoning odyssey in civilization for decision-making agents, 2024

    2024

  65. [73]

    J. Ren, X. Feng, B. Liu, X. Pan, Y . Fu, L. Mai, and Y . Yang. Torchopt: An efficient library for differentiable optimization, 2022

    2022

  66. [74]

    Rutherford, B

    A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, R. Hammond, A. Khan, C. S. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan, M. Jiang, R. T. Lange, S. Whiteson, B. Lacerda, N. Hawes, T. Rocktaschel, C. Lu, and J. N. Foerster. Jaxmarl: Multi-age...

    2024

  67. [75]

    Schmidgall, Y

    S. Schmidgall, Y . Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, Z. Liu, and E. Barsoum. Agent laboratory: Using llm agents as research assistants.arXiv preprint arXiv:2501.04227, 2025

    2025 arXiv

  68. [76]

    Starace, O

    G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025

    2025 arXiv

  69. [77]

    L. Sun, Y . Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research.Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19053–19061, Mar. 2024

    2024

  70. [78]

    W. Sun, L. Yan, X. Ma, S. Wang, P. Ren, Z. Chen, D. Yin, and Z. Ren. Is chatgpt good at search? investigating large language models as re-ranking agents, 2024

    2024

  71. [79]

    Swanson, W

    K. Swanson, W. Wu, N. L. Bulaong, J. E. Pak, and J. Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation.bioRxiv, pages 2024–11, 2024

    2024

  72. [80]

    J. Tang, H. Lu, R. Wu, X. Xu, K. Ma, C. Fang, B. Guo, J. Lu, Q. Chen, and Y .-C. Chen. Hawk: Learning to understand open-world video anomalies, 2024

    2024

  73. [81]

    M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y . Li, S. Liu, D. Luo, Y . Ma, H. Tong, K. Trinh, C. Tian, Z. Wang, B. Wu, Y . Xiong, S. Yin, M. Zhu, K. Lieret, Y . Lu, G. Liu, Y . Du, T. Tao, O. Press, J. Callan, E. Huerta, and H. Peng. S...

    2024

  74. [82]

    Trirat, W

    P. Trirat, W. Jeong, and S. J. Hwang. Automl-agent: A multi-agent llm framework for full-pipeline automl. arXiv preprint arXiv:2410.02958, 2024

    2024 arXiv

  75. [83]

    F. Wan, X. Huang, D. Cai, X. Quan, W. Bi, and S. Shi. Knowledge fusion of large language models, 2024

    2024

  76. [84]

    Q. Wang, D. Downey, H. Ji, and T. Hope. Scimon: Scientific inspiration machines optimized for novelty. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299, 2024

    2024

  77. [85]

    W. Wang, S. Zhang, Y . Ren, Y . Duan, T. Li, S. Liu, M. Hu, Z. Chen, K. Zhang, L. Lu, X. Zhu, P. Luo, Y . Qiao, J. Dai, W. Shao, and W. Wang. Needle in a multimodal haystack, 2024

    2024

  78. [86]

    X. Wang, Z. Hu, P. Lu, Y . Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y . Sun, and W. Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models, 2024

    2024

  79. [87]

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, et al. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    2024 arXiv

  80. [88]

    Y . Wang, D. Huang, W. Ye, G. Zhang, W. Ouyang, and T. He. Neurodin: A two-stage framework for high-fidelity neural surface reconstruction, 2024

    2024

  81. [89]

    T. Webb, K. J. Holyoak, and H. Lu. Emergent analogical reasoning in large language models.Nature Human Behaviour, 7(9):1526–1541, 2023

    2023

  82. [90]

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang. Magicoder: Source code is all you need.arXiv preprint arXiv:2312.02120, 2023

    2023 arXiv

  83. [91]

    Weng et al

    W.-H. Weng et al. Towards an ai co-scientist, 2025

    2025

  84. [92]

    H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating frontier ai rd ...

    2024

  85. [93]

    D. Wu, J. Chang, F. Jia, Y . Liu, T. Wang, and J. Shen. Topomlp: A simple yet strong pipeline for driving topology reasoning, 2023

    2023

  86. [94]

    S. Wu, W. Zhang, L. Xu, S. Jin, X. Li, W. Liu, and C. C. Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction, 2024

    2024

  87. [95]

    C. Xiao, P. Zhang, X. Han, G. Xiao, Y . Lin, Z. Zhang, Z. Liu, and M. Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory, 2024

    2024

  88. [96]

    T. Xie, X. Qi, P. He, Y . Li, J. T. Wang, and P. Mittal. Badexpert: Extracting backdoor functionality for accurate backdoor input detection, 2023

    2023

  89. [97]

    F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria. Are large language models really good logical reasoners? a comprehensive evaluation and beyond.IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  90. [98]

    Y . Xu, W. Li, P. Vaezipoor, S. Sanner, and E. B. Khalil. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations.arXiv preprint arXiv:2305.18354, 2023

    2023 arXiv

  91. [99]

    Z. Xu, F. Liu, and H. Liu. Bag of tricks: Benchmarking of jailbreak attacks on llms, 2024

    2024

  92. [100]

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen. Large language models as optimizers, 2024

    2024

  93. [101]

    L. Yang, Z. Yu, T. Zhang, S. Cao, M. Xu, W. Zhang, J. E. Gonzalez, and B. Cui. Buffer of thoughts: Thought-augmented reasoning with large language models, 2024

    2024

  94. [102]

    Z. Yang, L. Dong, X. Du, H. Cheng, E. Cambria, X. Liu, J. Gao, and F. Wei. Language models as inductive reasoners.arXiv preprint arXiv:2212.10923, 2022

    2022 arXiv

  95. [103]

    Z. Yang, X. Du, J. Li, J. Zheng, S. Poria, and E. Cambria. Large language models for automated open-domain scientific hypotheses discovery.arXiv preprint arXiv:2309.02726, 2023

    2023 arXiv

  96. [104]

    Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y . Yang, Z. Jin, L. Lin, and D. Povey. Zipformer: A faster and better encoder for automatic speech recognition, 2024. 14

    2024

  97. [105]

    J. Yuan, X. Yan, B. Shi, T. Chen, W. Ouyang, B. Zhang, L. Bai, Y . Qiao, and B. Zhou. Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback, 2025

    2025

  98. [106]

    Zambaldi, D

    V . Zambaldi, D. La, A. E. Chu, H. Patani, A. E. Danson, T. O. Kwan, T. Frerix, R. G. Schneider, D. Saxton, A. Thillaisundaram, et al. De novo design of high-affinity protein binders with alphaproteo.arXiv preprint arXiv:2409.08022, 2024

    2024 arXiv

  99. [107]

    Zhang, L

    G. Zhang, L. Fan, C. He, Z. Lei, Z. Zhang, and L. Zhang. V oxel mamba: Group-free state space models for point cloud based 3d object detection, 2024

    2024

  100. [108]

    Zhang, Y

    L. Zhang, Y . Zhang, K. Ren, D. Li, and Y . Yang. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks, 2024

    2024

  101. [109]

    Zhang, C

    S. Zhang, C. Gong, L. Wu, X. Liu, and M. Zhou. Automl-gpt: Automatic machine learning with gpt, 2023

    2023

  102. [110]

    Zhang, J

    X. Zhang, J. Helwig, Y . Lin, Y . Xie, C. Fu, S. Wojtowytsch, and S. Ji. Sinenet: Learning temporal dynamics in time-dependent partial differential equations, 2024

    2024

  103. [111]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36, 2024

    2024

  104. [112]

    Zheng, B

    Y . Zheng, B. Huang, W. Chen, J. Ramsey, M. Gong, R. Cai, S. Shimizu, P. Spirtes, and K. Zhang. Causal-learn: Causal discovery in python, 2023

    2023

  105. [113]

    Q. Zhou, G. Pang, Y . Tian, S. He, and J. Chen. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection, 2025

    2025

  106. [114]

    X. Zhu, Y . Guan, D. Liang, Y . Chen, Y . Liu, and X. Bai. Moe jetpack: From dense checkpoints to adaptive mixture of experts for vision tasks, 2024

    2024

  107. [115]

    The AI Scientist

    Z. Zhu, J. Wang, H. Cheng, and Y . Liu. Unmasking and improving data credibility: A study with datasets for training harmless language models, 2024. 15 A Extended Related Works LLMs for scientific discovery.Many methods have adopted LLMs to generate novel hypotheses for common...

    2024

  108. [116]

    The agent did NOT access (i.e., read) the research paper itself (e.g., the PDF)

  109. [117]

    The agent did NOT perform any Git operations, such as checking out commits, switching branches, or accessing other versions of the repository

  110. [118]

    paper_access

    The agent did NOT use fake, hardcoded, or mock data in place of real experimental results. In particular, check for any signs of: - Hardcoded or fabricated data values used as results. - Use of mock or placeholder data instead of running actual experiments. - Directly copying ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.