Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces BioDSA-1K, a benchmark of 1,029 hypothesis-validation tasks drawn from 329 published biomedical studies, and shows that current AI agents reduce errors with explicit reasoning but still align only weakly with human…

desk verdict Large useful benchmark, but ground-truth extraction is unvalidated and the same-model loop is real; treat results as pilot numbers until revision. read the letter →

arxiv 2505.16100 v1 pith:CHCYCGYT submitted 2025-05-22 cs.AI cs.CL

Zifeng Wang , Benjamin Danek , Jimeng Sun This is my paper
classification cs.AIcs.CL
keywords biomedicaldatasciencebenchmarkhypothesisvalidationAIagentslargelanguagemodelsnon-verifiableevidencealignmentcBioPortal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that existing benchmarks for AI data-science agents are too small and too simple to capture the reality of biomedical hypothesis validation, and it offers BioDSA-1K as a larger and more realistic alternative. The benchmark turns published study conclusions into testable hypotheses with paired analysis plans and underlying cancer-genomics datasets, allowing agents to be scored on whether they reach the right decision, whether their observations match the reported evidence, whether their reasoning process is sound, and whether their code runs. Experiments with code-generation and ReAct-style agents show that explicit planning and iterative reasoning cut both false positives and false negatives, but evidence alignment stays uniformly low and most agents over-assert on hypotheses that the data cannot actually verify.

What carries the argument

The load-bearing mechanism is the benchmark-construction and evaluation pipeline: GPT-4o extracts binary hypotheses and structured evidence from publication abstracts via a fixed JSON prompt, the extracted variables are mapped to schema-based captions of the linked cBioPortal data tables (so agents reason over column statistics, not patient-level records), and the agent's final output is scored by four quantitative metrics. That same pipeline produces the deliberately non-verifiable cases by attaching claims to unrelated datasets, which turns abstention into a measurable capability rather than a failure mode.

What would settle it

Take a random sample of, say, 100 BioDSA-1K tasks and have two biomedical expert annotators independently check (a) whether the extracted hypothesis matches the source paper's conclusion, (b) whether the listed analysis variables actually exist in the linked cBioPortal table, and (c) whether the reported evidence value (e.g., r = 0.65) can be reproduced from the data. If the two annotators disagree with the benchmark label on more than a small fraction of tasks, or if a substantial share of tasks lack the listed variables, the ground truth cannot support the claimed evaluation axes.

Watch

Extended reading notes

Core claim

BioDSA-1K is presented as a benchmark of 1,029 hypothesis-centric tasks paired with 1,177 analysis plans, curated from 329 cBioPortal-linked publications. Each task contains an affirmative hypothesis derived from the abstract, a perturbed counter-hypothesis, and one or more structured evidence entries specifying an analysis plan, the input variables, the result variable, and its reported value. The benchmark evaluates agents along four axes: hypothesis decision accuracy measured by Type I and Type II errors, evidence alignment scored by an LLM judge comparing agent observations against reported evidence, code executability, and the detection of non-verifiable hypotheses where the data are insufficient. In the reported experiments, reasoning-augmented agents (CodeGen-Reasoning and ReAct-Reasoning) achieve lower Type I and Type II errors than their single-shot counterparts, evidence alignment scores hover near 0.20–0.25 across all methods, and the best non-verifiable detection rate reaches 92% for ReAct-Reasoning, while one-shot code generators wrongly assert verifiable claims in more than a third of strictly non-verifiable cases.

Load-bearing premise

The benchmark's ground truth is produced by GPT-4o extracting hypotheses and evidence from publication abstracts, with no human validation of whether those extractions match the original studies' actual methods or whether the linked datasets contain the needed variables.

Editorial extensions

If this is right

  • A benchmark at this scale makes agent performance on realistic biomedical workflows reproducible and directly comparable across methods.
  • Reasoning-augmented, plan-then-execute agents reduce both false-positive and false-negative hypothesis decisions compared with single-shot code generation.
  • The uniformly low evidence alignment scores imply that correct final decisions do not guarantee faithful analysis, so future agent evaluation should include process-level metrics.
  • Non-verifiable hypotheses form a distinct scoring axis where current agents over-assert, indicating that calibration and abstention skills are underdeveloped.
  • Code executability varies by agent design, and variable/object misuse is the most common failure, pointing to concrete technical improvements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the extraction pipeline were expanded with human-in-the-loop verification, the benchmark could be used to train agents to request missing variables or state what additional data would be needed before declaring a hypothesis non-verifiable.
  • The low alignment scores suggest that an agent could game the decision axis by guessing the majority outcome, which implies the benchmark's real signal may lie more in evidence alignment and code executability than in decision accuracy alone.
  • The same publication-to-task recipe could be applied to other data-driven fields with accessible public datasets and linked abstracts, extending the approach beyond biomedicine.
  • A natural stress test would be to vary the amount of noise or missingness in the provided data tables to see whether agents' false-confidence rates track data completeness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BioDSA-1K, a benchmark of 1,029 hypothesis-validation tasks and 1,177 analysis plans derived from 329 biomedical publications linked to cBioPortal datasets. Each task contains a hypothesis, a counter-hypothesis, and one or more pieces of supporting evidence extracted from publication abstracts using GPT-4o. The benchmark evaluates agents along four axes: hypothesis decision accuracy (Type I/II errors), evidence alignment, reasoning quality, and code executability, and it includes non-verifiable hypotheses. The authors benchmark four agent variants (CodeGen, ReAct, and reasoning-augmented versions) and report that reasoning augmentation reduces error rates and that alignment scores are low overall.

Significance. If the ground-truth extraction is reliable, BioDSA-1K would be a valuable resource: it is substantially larger and more diverse than prior biomedical data-science benchmarks, uses openly available cBioPortal data, introduces a four-axis evaluation scheme that goes beyond code generation, and explicitly targets non-verifiable hypotheses, which is a realistic and underexplored scenario. The appendices provide prompts, schema-captioning details, and error taxonomies that support reproduction. The central risk is that the automatically extracted hypotheses and analysis variables are not shown to be anchored to actual data columns, which would undermine every downstream metric.

major comments (4)
  1. [Section 2.3, Appendix C] The extraction pipeline constructs ground-truth hypotheses and evidence from abstracts alone. The Appendix C prompt receives only the title and abstract (plus a cBioPortal result summary) and instructs GPT-4o that 'analysis_variables must be specific data columns or features that exist in the dataset', but no dataset schema or table caption is provided in that context, so the model cannot verify existence. The paper reports no post-hoc schema check, human review, or error analysis of the extracted variables. If a nontrivial fraction of the 1,029 tasks references columns that are absent from the linked cBioPortal tables, then the True/False/Non-verifiable ground-truth labels in Eqs. (1)-(4), the Type I/II error rates in Table 2, and the alignment scores in Eq. (3) are all affected: agents that correctly answer 'Non-verifiable' because the data lack the variables could be scored as Type II errors, and agents that invent analyses over non-existent columns could appear to align with evidence. I request a validation step that checks every analysis_variables and result_variable entry against the actual table schemas, reports the fraction of tasks with fully grounded variables, and either recomputes all metrics on the validated subset or replaces ungrounded entries.
  2. [Section 3.4, Figure 8] The construction of the 100 non-verifiable hypotheses is underspecified. The text says they are 'curated by taking claims from other publications that reference unrelated datasets', but it does not state the selection criteria, how 'unrelated' is determined, whether the absence of relevant columns was manually verified, or how often the authors had to discard candidate claims because the data actually contained the relevant variables. Since the correct label is defined as Non-verifiable, any contamination in this set directly changes the TPR in Figure 8. Please provide the full list or a detailed construction protocol with human-verified labels for this subset.
  3. [Equation (3), Section 3.3] The evidence alignment score relies on an LLM-as-a-judge to compute |O ∩ E|, but no rubric, calibration set, human agreement, or examples of matched/unmatched observations are provided. The judge is in the same model family (GPT-4o) as the extractor and as the evaluated agents, so the uniformly low alignment scores (0.20–0.25 in Figure 5) could reflect judge strictness or same-model bias rather than genuine agent divergence from the evidence. I request a human-annotated sample (e.g., 50–100 tasks) with inter-annotator agreement, a released judge prompt and scoring rubric, and a comparison of LLM-judge scores against human judgments.
  4. [Section 3.2, Table 2, Figure 3] The paper claims that reasoning-augmented agents 'consistently outperform' base agents, but Table 2 and Figure 3 contain no error bars, confidence intervals, or significance tests. Several differences are small and in opposite directions across categories (e.g., Genomics Type I error 0.060 vs. 0.069; Translational Type II error 0.110 vs. 0.112 the other way). The NeurIPS checklist answers 'Yes' to the statistical-significance question, which is inconsistent with the absence of any variability estimate. Please report per-seed or bootstrap variability, or soften the performance comparisons to 'numerically lower in most categories'.
minor comments (6)
  1. [Figure 4 and Section 3.3] The figure caption and the first sentence of Section 3.3 use 'Code excitability' / 'excitability analysis' where 'executability' is meant; please correct the typo.
  2. [Figure 4 uses and Section 3.4] Figure 4 is referenced for two different purposes: Section 3.3 describes it as the executability and error-breakdown figure, and Section 3.4 again says 'Figure 4 illustrates the proportion of executable versus non-executable code'. Please renumber or re-reference so each figure is discussed in one place.
  3. [Table 1] The BioDSA-1K row reads '6 879 1029' under the columns 'Avg. # Tables', 'Avg. # Columns*', and '# Tasks'; if these are meant as separate values, please add separators or a note so the table is unambiguous.
  4. [Section 2.1 and Section 2.3] Section 2.1 states that filtering ambiguous cases 'avoids introducing non-verifiable hypotheses into the benchmark', but Section 2.3 and Section 3.4 explicitly include non-verifiable hypotheses; the wording should be reconciled, e.g., by saying that filtering avoids unintentional non-verifiability.
  5. [Appendix A and Appendix C] The schema in Appendix A uses 'null_hypothesis' while the extraction prompt in Appendix C requests 'wrong_hypothesis'; using a single consistent field name would avoid confusion.
  6. [Section 2.2 and Appendix B] The claim that 'we do not send any patient-level records to LLMs' should be clarified, since the captioning includes most-frequent entries and value ranges for categorical columns, which can be near-identifying in small cohorts.

Circularity Check

2 steps flagged · score 5.0 of 10

The benchmark's ground truth is GPT-4o-generated from abstracts, and the evidence-alignment metric uses an LLM judge to score GPT-4o-family agents against that GPT-4o target; the evaluation loop is self-referential, though the measured numbers are not trivially forced.

  1. self definitional [Section 2.3 and Appendix C (extraction prompt)]
    "All data science challenges in BIODSA-1K are extracted from published biomedical studies using a GPT-4o model. ... 'hypothesis': a specific, binary hypothesis that can be tested statistically, from the abstract, the one which is considered to be true from the study ... 'wrong_hypothesis': make a random perturbation of the hypothesis so that it is a wrong hypothesis ... Analysis variables must be specific data columns or features that exist in the dataset."

    The True/False ground-truth labels are defined by the GPT-4o extraction prompt: the True hypothesis is the one 'considered to be true from the study' and the False hypothesis is a 'random perturbation' labeled wrong by construction. The prompt instructs that analysis variables must exist in the dataset, but no dataset schema is supplied to the extractor, and no post-hoc schema check or human validation is reported. Agents are then scored on reproducing these generator-defined labels from the tables, so the 'hypothesis decision accuracy' axis measures agreement with GPT-4o's abstract interpretation rather than an independently data-grounded truth.

  2. other [Section 2.4, Eq. (3); Section 3.1; Appendix C]
    "We use a large language model (LLM)-as-a-judge [25] approach to measure the evidence alignment score: Alignment Score = |O ∩ E| / |E|. ... All data science challenges in BIODSA-1K are extracted from published biomedical studies using a GPT-4o model. ... We evaluate two variants of CodeGen: one powered by GPT-4o and the other by O3-mini."

    The ground-truth evidence set E in Eq. (3) is generated by GPT-4o from the abstract/result summary; the overlap |O ∩ E| is judged by an LLM; and the agents whose observations O are scored are GPT-4o or O3-mini based. Thus the evidence-alignment metric is a closed same-model loop: LLM-generated observations are compared, by an LLM judge, against an LLM-generated target. It measures self-consistency within the GPT-4o family rather than alignment with verified evidence from the original studies or data tables, so the 'evidence grounding' axis is circular by construction.

full rationale

The central contribution is a benchmark, not a theorem, so most of the paper is a straightforward measurement rather than a derivation chain. I found no load-bearing self-citation: references [10] and [5] are prior work by the same group, but they appear only as comparisons or contextual background and do not supply an unverified premise on which the benchmark's claims rest. The circularity concern is in the benchmark construction itself. Section 2.3 states that all challenges are extracted by GPT-4o, and Appendix C shows the extractor receives only the title and abstract (plus a result summary) and is told that analysis variables must exist in the dataset without being given the dataset schema; no human validation, inter-annotator agreement, or schema-existence filter is reported. Consequently, the True/False/Non-verifiable labels are not independently shown to match the linked cBioPortal tables. The evidence-alignment score (Eq. 3) compounds this: the target evidence E is GPT-4o-generated, the judge is an LLM, and the evaluated agents are GPT-4o/o3-mini, so the metric measures a same-model loop rather than alignment with verified publication evidence. I did not set a higher score because the reported measurements are not trivially forced by this loop: alignment scores are low and vary by method and analysis type, which would not happen if the evaluation were a tautology. The non-verifiable subset is defined by dataset linkage rather than by schema inspection, which is a further grounding gap but not itself a derivation loop. Overall, the partial circularity is real and affects the benchmark's validity, but it does not reduce the whole paper's empirical content to a definition.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark does not introduce mathematical free parameters or new physical entities. The key assumptions are domain-level: that abstracts and cBioPortal datasets together encode testable hypotheses, and that the LLM-based extraction and judging pipeline produces labels and scores that are faithful to the original studies. These assumptions are not independently verified in the paper.

assumptions (3)
  • domain assumption Publication abstracts accurately contain the primary hypotheses and quantitative results, and the linked cBioPortal datasets are sufficient to test those hypotheses.
    Stated in Section 2.1: 'we assume that each publication highlights its primary results within the abstract, often supported by descriptive statistics, statistical testing, and predictive modeling results derived from the associated biomedical tables.' This assumption is necessary for the whole extraction pipeline.
  • ad hoc to paper GPT-4o can extract testable hypotheses and supporting evidence that faithfully reflect the original studies' analyses and dataset content.
    Section 2.3 and Appendix C describe how all hypotheses and evidence entries are generated by GPT-4o with no human validation or quality control reported. The benchmark's ground truth depends entirely on this model's reliability.
  • ad hoc to paper The LLM-as-a-judge alignment score (Eq. 3) reflects true correspondence between agent observations and the evidence reported in the original publication.
    The alignment metric in Section 2.4 uses an LLM to judge the overlap between generated observations and ground truth evidence, and no calibration against human judgments is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research." pith.science (2026). https://pith.science/paper/CHCYCGYT

@misc{pith2026250516100,
  author       = {Pith},
  title        = {Pith review of: BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHCYCGYT}},
  note         = {Machine review of arXiv:2505.16100}
}
read the original abstract

Validating scientific hypotheses is a central challenge in biomedical research, and remains difficult for artificial intelligence (AI) agents due to the complexity of real-world data analysis and evidence interpretation. In this work, we present BioDSA-1K, a benchmark designed to evaluate AI agents on realistic, data-driven biomedical hypothesis validation tasks. BioDSA-1K consists of 1,029 hypothesis-centric tasks paired with 1,177 analysis plans, curated from over 300 published biomedical studies to reflect the structure and reasoning found in authentic research workflows. Each task includes a structured hypothesis derived from the original study's conclusions, expressed in the affirmative to reflect the language of scientific reporting, and one or more pieces of supporting evidence grounded in empirical data tables. While these hypotheses mirror published claims, they remain testable using standard statistical or machine learning methods. The benchmark enables evaluation along four axes: (1) hypothesis decision accuracy, (2) alignment between evidence and conclusion, (3) correctness of the reasoning process, and (4) executability of the AI-generated analysis code. Importantly, BioDSA-1K includes non-verifiable hypotheses: cases where the available data are insufficient to support or refute a claim, reflecting a common yet underexplored scenario in real-world science. We propose BioDSA-1K as a foundation for building and evaluating generalizable, trustworthy AI agents for biomedical discovery.

Figures

Figures reproduced from arXiv: 2505.16100 by the authors.

Figure 1
Figure 1. Benchmark statistics. (left) BIODSA-1K includes diverse types of biomedical research and data analysis tasks created from 329 publications; the x-axis indicates the publication types.; (Right) Bubble plot illustrating the diverse range of biomedical data tables in BIODSA-1K, showing each data table’s number of rows (x-axis, log-scale) versus number of columns (y-axis, log-scale). benchmark dataset that captures the … view at source ↗
Figure 2
Figure 2. Overview of BIODSA-1K. a, Benchmark curation: Scientific publications linked to biomedical datasets are parsed to extract hypotheses and their corresponding supporting evidence, forming the core reasoning challenges. b, Experiments: AI agents are tasked with validating hypotheses by planning analysis steps, generating executable code, observing results, and making decisions based on structured biomedical datasets. c… view at source ↗
Figure 3
Figure 3. Comparison of Type I and Type II error rates across publication types and agent variants. Each point denotes an agent’s performance on a specific publication type. We implement four agent-based methods to eval￾uate performance on BIODSA-1K. CodeGen di￾rectly generates a single executable Python code block based on the input hypothesis and dataset schema, and returns a final decision, True, False, or Non-verifiable, … view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Evidence alignment scores for true vs. false hypotheses across methods. Frequency (n=4765) Structural (n=1993) Comparison (n=7487) Pathway (n=1250) Correlation (n=3185) Survival (n=3465) Clustering (n=1154) Functional (n=1542) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Al…
Figure 4
Figure 4. Figure 4: Code excitability analysis and the break [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 7
Figure 7. Figure 7: Hypothesis validation results’ distribution by the code excitability for CodeGen methods. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Hypothesis decision distribution for the [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging

    cs.AI 2026-07 conditional novelty 7.0 of 10

    A 57-task, expert-verified benchmark shows frontier LLM coding agents systematically fail on physical conventions, inverse-solver choice, and end-to-end imaging pipelines.

  2. Plato-Bio: verification-first biological novelty screening with temporal rediscovery and structural benchmarks

    cs.AI 2026-07 conditional novelty 4.0 of 10

    Plato-Bio is a reproducible, audit-friendly biology agent pipeline whose full test suite passes, whose one historical evidence-bridge task ranks before TF-IDF, and whose 15-protein structure screen finds 11 targets un...

Reference graph

Works this paper leans on

54 extracted references · 39 canonical work pages · cited by 2 Pith papers

  1. [1]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025

    2025

  2. [2]

    Large language model based multi-agents: a survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: a survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 8048–8057, 2024

    2024

  3. [3]

    Towards an ai co-scientist

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025

    arXiv 2025

  4. [4]

    Autonomous chemical research with large language models

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023

    2023

  5. [5]

    A foundation model for human-ai collaboration in medical literature mining

    Zifeng Wang, Lang Cao, Qiao Jin, Joey Chan, Nicholas Wan, Behdad Afzali, Hyun-Jin Cho, Chang-In Choi, Mehdi Emamverdi, Manjot K Gill, et al. A foundation model for human-ai collaboration in medical literature mining. arXiv preprint arXiv:2501.16255, 2025

    arXiv 2025

  6. [6]

    Position: data-driven discovery with large generative models

    Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Position: data-driven discovery with large generative models. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  7. [7]

    Empow- ering biomedical discovery with ai agents

    Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. Empow- ering biomedical discovery with ai agents. Cell, 187(22):6125–6151, 2024

    work page 2024

  8. [8]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

Show all 54 references
  1. [9]

    Healthcare data scientist qualifications, skills, and job focus: a content analysis of job postings

    Melanie A Meyer. Healthcare data scientist qualifications, skills, and job focus: a content analysis of job postings. Journal of the American Medical Informatics Association , 26(5): 383–391, 2019

    2019

  2. [10]

    Can large language models replace data scientists in biomedical research? arXiv preprint arXiv:2410.21591, 2024

    Zifeng Wang, Benjamin Danek, Ziwei Yang, Zheng Chen, and Jimeng Sun. Can large language models replace data scientists in biomedical research? arXiv preprint arXiv:2410.21591, 2024

    2024 arXiv

  3. [11]

    BioDiscoveryAgent: An ai agent for designing genetic perturbation experiments

    Yusuf H Roohani, Jian V ora, Qian Huang, Percy Liang, and Jure Leskovec. BioDiscoveryAgent: An ai agent for designing genetic perturbation experiments. InICLR 2024 Workshop on Machine Learning for Genomics Explorations, 2024

    2024

  4. [12]

    BioCoder: a benchmark for bioinformatics code generation with large language models

    Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. BioCoder: a benchmark for bioinformatics code generation with large language models. Bioin- formatics, 40(Supplement_1):i266–i276, 2024

    2024

  5. [13]

    Automated hypothesis validation with agentic sequential falsifications

    Kexin Huang, Ying Jin, Ryan Li, Michael Y Li, Emmanuel Candès, and Jure Leskovec. Automated hypothesis validation with agentic sequential falsifications. arXiv preprint arXiv:2502.09858, 2025

    2025 arXiv

  6. [14]

    Discoverybench: Towards data-driven discovery with large language models

    Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth V ora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. In The Thirteenth Inte...

    2024

  7. [15]

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080, 2024. 10

    arXiv

  8. [16]

    Onur Sumer, Yichao Sun, Anders Jacobsen, Rileen Sinha, Erik Larsson, Ethan Cerami, Chris Sander, and Nikolaus Schultz

    Jianjiong Gao, Bülent Arman Aksoy, Ugur Dogrusoz, Gideon Dresdner, Benjamin Gross, S. Onur Sumer, Yichao Sun, Anders Jacobsen, Rileen Sinha, Erik Larsson, Ethan Cerami, Chris Sander, and Nikolaus Schultz. Integrative analysis of complex cancer genomics and clinical profiles us...

    2013 doi

  9. [17]

    https://opendatacommons.org/ licenses/odbl/1-0/

    Open data commons open database license (odbl) v1.0. https://opendatacommons.org/ licenses/odbl/1-0/. Accessed: 2025-05-01

    2025

  10. [18]

    Cancer genome landscapes

    Bert V ogelstein, Nickolas Papadopoulos, Victor E Velculescu, Shibin Zhou, Luis A Diaz Jr, and Kenneth W Kinzler. Cancer genome landscapes. Science, 339(6127):1546–1558, 2013

    2013

  11. [19]

    DS-1000: A natural and reliable benchmark for data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. ...

    2023

  12. [20]

    Mlagentbench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. InForty-first International Conference on Machine Learning, 2024

    2024

  13. [21]

    DSBench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representations,

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representations,

  14. [22]

    BLADE: Benchmarking language model agents for data-driven science

    Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. BLADE: Benchmarking language model agents for data-driven science. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 139...

    2024

  15. [23]

    Scicode: A research coding benchmark curated by scientists

    Minyang Tian, Luyu Gao, Shizhuo Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624–30650, 2024

    2024

  16. [24]

    Large language models streamline automated machine learning for clinical studies

    Soroosh Tayebi Arasteh, Tianyu Han, Mahshad Lotfinia, Christiane Kuhl, Jakob Nikolas Kather, Daniel Truhn, and Sven Nebelung. Large language models streamline automated machine learning for clinical studies. Nature Communications, 15(1):1603, 2024

    2024

  17. [25]

    GPTScore: Evaluate as you desire

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576, 2024

    2024

  18. [26]

    Code generation with alphacodium: From prompt engineering to flow engineering

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500, 2024

    2024 arXiv

  19. [27]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  20. [28]

    Spider2-v: How far are multi- modal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37:107703–107744, 2024

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multi- modal agents from automating data science and engineering workflows? Advances in Neural Information Proces...

    2024

  21. [29]

    DS-Agent: Automated data science by empowering large language models with case-based reasoning

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-Agent: Automated data science by empowering large language models with case-based reasoning. In International Conference on Machine Learning, pages 16813–16848. PMLR, 2024

    2024

  22. [30]

    Large language models orchestrating structured reasoning achieve kaggle grandmaster level

    Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio Iacobacci, Abdelhakim Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle g...

    2024

  23. [31]

    Autokaggle: A multi-agent framework for autonomous data science competitions

    Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024

    arXiv

  24. [32]

    BioAgents: Democratizing bioinformatics analysis with multi-agent systems

    Nikita Mehandru, Amanda K Hall, Olesya Melnichenko, Yulia Dubinina, Daniel Tsirulnikov, David Bamman, Ahmed Alaa, Scott Saponas, and Venkat S Malladi. BioAgents: Democratizing bioinformatics analysis with multi-agent systems. arXiv preprint arXiv:2501.06314, 2025. 12 NeurIPS P...

    2025 arXiv

  25. [34]

    • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: [NA] Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper....

  26. [35]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: [NA] Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not d...

  27. [36]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 13 Justification: This paper does not involve theoretical results. Guidelines: • The answer NA means that th...

  28. [37]

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  29. [38]

    • Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 14 Answer: [Yes] Justification: [NA] Guidelines: • The a...

  30. [39]

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: [NA] Guidelines: • The answer...

  31. [40]

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: [NA] Guidelines: • The answer NA means that the pa...

  32. [41]

    • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: [NA] Guidelines: • The ...

  33. [42]

    • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: [NA] Guidelines: • The answer NA means that the authors have not reviewed the N...

  34. [43]

    • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: [NA] Guidelines: • The answer NA means that there is no societal impact of the work performed. • If the a...

  35. [44]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: [NA] Guideline...

  36. [45]

    • The authors should cite the original paper that produced the code package or dataset

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: [NA] Guidelines:...

  37. [46]

    • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 17 Answer: [Yes] Justification: [NA] Guidelines: • The answer NA means that the paper does not release new assets. • Researchers should communica...

  38. [47]

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  39. [48]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  40. [49]

    hypotheses

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  41. [50]

    Each hypothesis must be testable using statistical analysis or machine learning methods 23

  42. [51]

    All evidence must include specific, measurable quantities or statistical relationships

  43. [52]

    Result values must be numerical (e.g., percentages, counts, p-values, correlation coefficients) or categorical with clear classifications

  44. [53]

    hypotheses

    Analysis variables must be specific data columns or features that exist in the dataset Return your answer as a JSON object in the following format: ‘‘‘json { "hypotheses": [ { "hypothesis": a specific, binary hypothesis that can be tested statistically, from the abstract, the ...

  45. [54]

    "" 27 CODE_GENERATION_PROMPT_TEMPLATE =

    print hello world </analysis_plan> """ 27 CODE_GENERATION_PROMPT_TEMPLATE = """ # TASK Given the user-provided analysis plan for the user’s scientific hypothesis, you **Must** write {language} code to fulfill the plan so that user can execute the code later to evaluate the hyp...

  46. [2025]

    URL https://openreview.net/forum?id=DSsSPr0RZJ

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.