Pith. sign in

REVIEW 3 major objections 4 minor 3 references

LLMs Outperform Experts on Challenging Biology Benchmarks

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Frontier LLMs match or exceed human biology experts on several challenging benchmarks, with o3 more than doubling the expert virologist baseline on VCT-Text.

desk verdict Valuable systematic benchmark suite, but the 'outperforms experts' headline outruns the mismatched human baselines. read the letter →

arxiv 2505.06108 v3 pith:CZRXPYKJ submitted 2025-05-09 cs.LG cs.AIq-bio.QM

Lennart Justen This is my paper · ORCID
classification cs.LGcs.AIq-bio.QM
keywords largelanguagemodelsbiologybenchmarksexpertbaselinesVirologyCapabilitiesTestGPQAbenchmarksaturationinferencescalingbiosecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This study evaluates 27 large language models released between 2022 and 2025 on eight biology benchmarks, running each model-benchmark pair ten times. It claims that frontier models now match or exceed documented expert-level performance on several of the hardest tests: o3 reaches 46.1% on VCT-Text against a 22.6% expert virologist baseline, and o3 and Claude 3.7 Sonnet reach 61% on CloningScenarios against a 60% expert baseline. The paper also finds that top-model performance grew roughly 2- to 4-fold on these challenging benchmarks over the study period, while PubMedQA, MMLU-Bio, and WMDP-Bio plateaued below 100%, suggesting saturation and noisy answer labels. A sympathetic reader would care because the result redraws the line where AI systems stand relative to specialized human expertise in biology, with immediate consequences for biosecurity risk assessment and for how future biology benchmarks should be built.

What carries the argument

The machinery is a standardized evaluation protocol: 27 models, each run ten times zero-shot on eight benchmarks, scored by exact answer-format matching, with performance summarized as mean accuracy and compared to random-guess and human-expert baselines. The central comparison device is a normalized score, $\frac{\text{model accuracy}-\text{expert accuracy}}{1-\text{expert accuracy}}$, which turns 'above expert' into a positive fraction of the expert-to-perfect gap. The two hardest benchmarks use formats that suppress guessing: VCT-Text is multiple-response, requiring selection of all true statements from 4-10 options, giving a random baseline near 2.5%, while CloningScenarios embeds multi-plasmid DNA sequences that the model must reason over without tools. This design lets the paper attribute gains to model capability rather than answer memorization, provided the external human baselines are comparable.

What would settle it

Retest the 101 VCT-Text questions with a panel of PhD virologists under the same no-LLM conditions used in the original baselining, scoring exact multiple-response matches; if their mean accuracy reaches 46.1% or higher, the paper's headline claim that o3 performs twice as well as expert virologists fails on that benchmark.

Watch

Extended reading notes

Core claim

The paper's central claim is that, as of early 2025, several frontier LLMs perform at or above the level of human experts on challenging biology question sets, not just on easier knowledge-recall tests. The strongest evidence is VCT-Text, where o3 scores 46.1% versus the 22.6% mean accuracy of PhD-level virologists on text-only questions, a more than doubling; on GPQA-Bio o3 reaches 81.2% against the 66.7% expert baseline; on WMDP-Bio Claude 3.7 Sonnet reaches 86.1% against the 60.5% expert baseline; and on CloningScenarios o3 and Claude 3.7 Sonnet score about 61% against the 60% expert baseline. The paper also claims that chain-of-thought prompting does not reliably improve zero-shot accuracy, while explicit reasoning-effort controls generally improve performance in line with inference scaling, with VCT-Text as the notable exception. These results are presented as evidence that the bottleneck in biology evaluation has shifted from model capability to benchmark design and human-baseline quality.

Load-bearing premise

The human-expert baselines were measured on different, usually smaller, subsets of each benchmark than the full question sets the models answered, so the model-versus-expert comparisons may compare different tasks.

Editorial extensions

If this is right

  • If the central claim is correct, static multiple-choice biology benchmarks such as PubMedQA, MMLU-Bio, and WMDP-Bio are saturating below 100% because of benchmark limitations, so new evaluations with higher ceilings and better question validation are needed.
  • If correct, zero-shot evaluation is a reliable baseline for biology capability tracking, since five-shot and chain-of-thought prompting rarely improve accuracy and CoT costs 75x more output tokens on average.
  • If correct, frontier models at 61% on CloningScenarios can solve complex multi-fragment cloning design problems without the DNA manipulation tools that the expert baseliners were allowed to use.
  • If correct, o3's 46.1% on VCT-Text means an LLM can offer expert-level troubleshooting advice for wet-lab virology experiments, a capability with direct dual-use implications.
  • If correct, reasoning-effort controls improve accuracy on single-answer biology benchmarks, while on exhaustive multiple-response formats extra reasoning can reduce accuracy by inducing extra wrong selections.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the expert baselines hold, expert-written multiple-choice questions no longer separate frontier models from top human specialists, so the measurable frontier moves to agentic tasks like literature retrieval, protocol execution, and experimental design.
  • Editorial inference: the paper's VCT reasoning anomaly suggests a testable fix: on exhaustive multiple-response formats, asking a model to first propose candidate answers and then prune them may recover accuracy lost when extra reasoning adds incorrect statements.
  • Editorial inference: because model scores on WMDP-Bio exceed the expert baseline, biosecurity evaluations that treat expert-level knowledge as the risk ceiling should be recalibrated upward.
  • Editorial inference: a held-out benchmark with time-stamped questions published after each model's training cutoff would distinguish genuine biological reasoning from memorization of benchmark content, which the present design cannot fully exclude.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports a systematic evaluation of 27 large language models released between November 2022 and April 2025 on eight biology-related benchmarks (PubMedQA, MMLU-Bio, GPQA-Bio, WMDP-Bio, LAB-Bench LitQA2, CloningScenarios, ProtocolQA, and VCT-Text), using ten independent zero-shot runs per model-benchmark combination and the Inspect AI framework. The headline empirical findings are large performance gains over time, most strikingly a 4-fold improvement on VCT-Text with o3 at 46.1% versus a 22.6% expert virologist baseline, and claims that several models match or exceed documented expert performance on GPQA-Bio, WMDP-Bio, and CloningScenarios. Secondary analyses compare zero-shot, five-shot, and chain-of-thought prompting on four models and vary reasoning effort for o3-mini and Claude 3.7 Sonnet. The paper concludes that zero-shot evaluation is generally reliable, inference scaling usually helps, several established benchmarks are saturating, and better human baselines are needed.

Significance. The study is a useful independent benchmarking contribution: it applies a consistent protocol across a broad model family, performs repeated runs, reports standard deviations, makes evaluation code publicly available, and connects performance to model release dates and training compute. If the raw scores are taken at face value, the model-to-model time series on GPQA-Bio, VCT-Text, and LAB-Bench is informative for tracking AI capabilities in biology. The central 'outperforms experts' claim, however, is not established at the same standard as the model-to-model comparisons, because it relies on external human baselines measured on different subsets, with different tool access and, in at least one case, a comparison within sampling noise. The Discussion's call for more rigorous human baselines is appropriate but is in tension with the strength of the abstract and title.

major comments (3)
  1. [2.2.2 and Figure 2] Section 2.2.2 states that 'baseline measurements were conducted on slightly different subsets than my evaluation,' and this directly affects the paper's central claim. The GPQA-Bio expert baseline of 66.7% comes from GPQA_extended biology questions, whereas models were evaluated on 78 GPQA_main biology questions; the WMDP-Bio expert baseline comes from two RAND researchers on roughly 20% of WMDP-Bio; and the MMLU-Bio expert baseline of 89.8% is an estimate from exam percentiles across all MMLU categories, not a direct measurement on MMLU-Bio. Comparisons in Figure 2 therefore mix benchmark subsets and estimation methods, so the statement in Section 3.1 that top models 'surpassed expert performance on GPQA-Bio, WMDP-Bio, and CloningScenarios' is not supported at the same level of evidence as the model-to-model results. I would ask the author to either add same-question human baseline measurements for the exact evaluated subsets or to restrict the abstract, title, and Section 3.1 to 'exceeds published expert baselines' with explicit caveats.
  2. [3.1 and Table A4.1] CloningScenarios consists of 33 questions, and the 'exceed' claim is numerically fragile. In Table A4.1, o3 attains 61.2% ± 4.5% and Claude 3.7 Sonnet (16K) attains 60.9% ± 4.4% against a 60% expert baseline; the difference is less than one question, and no confidence interval is given for the expert baseline, which itself came from a representative sample. Under an exact binomial test, 61% on 33 questions versus 60% is entirely consistent with noise, so Section 3.1's claim of 'slightly exceeding the 60% expert baseline' overstates the evidence. Please report an uncertainty estimate for this comparison and avoid claiming 'exceed' unless it survives a test or is explicitly framed as a point estimate.
  3. [2.4 and 2.2.2 (VCT-Text)] The 'twice as well as expert virologists' claim depends on scoring-rule comparability. Section 2.4 describes using Inspect's choice scorer with multiple_correct=True, which requires an exact all-or-nothing match, but Section 2.2.2 simply reports the VCT human baseline of 22.6% without stating whether the original human evaluation used the same exact-match scoring on the same 101 text-only questions. If the human evaluation allowed partial credit, or if the text-only baseline was computed over the experts' tailored subsets rather than the full VCT-Text set, the 46.1% versus 22.6% comparison would no longer be apples-to-apples. The manuscript should verify and document the original VCT scoring rule and subset, or downgrade the abstract's 'twice as well' claim.
minor comments (4)
  1. [Table A4.1] The table header contains '03-mini' where 'o3-mini' is intended.
  2. [Section 3.1] The sentence 'performance increased 2.6-fold the study period' appears to be missing 'over' before 'the study period.'
  3. [Figure 2 caption] The explanation of negative normalized values is hard to parse; a concrete example would improve clarity.
  4. [Section 2.4 and Data Availability] Raw logs being 'available upon reasonable request' is weaker than the reproducibility standard implied by the public code; consider a more specific data-sharing plan.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: results are direct measurements against external benchmarks and baselines, with acknowledged subset-mismatch limitations that concern validity rather than circularity.

full rationale

This is an empirical evaluation study, not a derivation. Model scores are produced by running 27 externally hosted models against eight publicly available benchmark datasets using the Inspect AI framework; no parameter is fitted from any subset of the data and then renamed as a prediction. The human expert baselines are imported from the original benchmark publications or subsequent RAND baselining efforts, i.e., from external sources that do not depend on the present paper's measurements. The paper explicitly concedes that some baseline measurements were made on slightly different subsets than its own evaluation and warns that comparisons should be interpreted with uncertainty; that is a validity and comparability limitation, not a circularity. There is no self-citation chain supporting the central claim: citations to LAB-Bench, GPQA, WMDP, VCT, and RAND are external benchmark sources, and the single-author paper does not rely on any prior work by itself as the justification for its conclusions. The reference to inference scaling laws is peripheral and merely contextualizes observed reasoning-effort trends. No equation or quantity in the paper is defined in terms of the outcome it is used to predict, and no fitted value is masquerading as a result. The central claims are direct measurements against external ground truth, so the appropriate circularity finding is none.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical evaluation, so it does not introduce fitted parameters or new theoretical entities. Its conclusions rest on external benchmark labels, external human baselines, and the automated evaluation framework, all treated as unexamined inputs.

assumptions (3)
  • domain assumption Human expert baselines from original benchmark papers are valid and directly comparable to model performance on the evaluated subsets.
    The central 'exceeds expert' claim depends on comparing model scores on full or filtered benchmark subsets with human scores obtained on different (often smaller) subsets. Section 2.2.2 acknowledges the baseline subsets differ.
  • domain assumption Benchmark ground-truth labels are correct.
    The paper interprets below-100% performance plateaus as evidence of errors in benchmark data (Section 4.3), which presumes the labels define the correct answers for measuring model accuracy.
  • domain assumption The Inspect AI framework's choice scorer correctly extracts and scores model answers.
    All accuracy numbers are produced by automated scoring of model outputs; models failing the output schema were excluded, which assumes the scorer correctly identifies valid answers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs Outperform Experts on Challenging Biology Benchmarks." pith.science (2026). https://pith.science/paper/CZRXPYKJ

@misc{pith2026250506108,
  author       = {Pith},
  title        = {Pith review of: LLMs Outperform Experts on Challenging Biology Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZRXPYKJ}},
  note         = {Machine review of arXiv:2505.06108}
}
read the original abstract

This study systematically evaluates 27 frontier Large Language Models on eight biology benchmarks spanning molecular biology, genetics, cloning, virology, and biosecurity. Models from major AI developers released between November 2022 and April 2025 were assessed through ten independent runs per benchmark. The findings reveal dramatic improvements in biological capabilities. Top model performance increased more than 4-fold on the challenging text-only subset of the Virology Capabilities Test over the study period, with OpenAI's o3 now performing twice as well as expert virologists. Several models now match or exceed expert-level performance on other challenging benchmarks, including the biology subsets of GPQA and WMDP and LAB-Bench CloningScenarios. Contrary to expectations, chain-of-thought did not substantially improve performance over zero-shot evaluation, while extended reasoning features in o3-mini and Claude 3.7 Sonnet typically improved performance as predicted by inference scaling. Benchmarks such as PubMedQA and the MMLU and WMDP biology subsets exhibited performance plateaus well below 100%, suggesting benchmark saturation and errors in the underlying benchmark data. The analysis highlights the need for more sophisticated evaluation methodologies as AI systems continue to advance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 3 linked inside Pith

  1. [1]

    GPT-4 Technical Report

    OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al. GPT-4 Technical Report. arXiv [csCL] 2023. http://arxiv.org/abs/2303.08774. [2] Zhang B, Takeuchi M, Kawahara R, Asthana S, Hossain MM, Ren G-J, et al. Enterprise benchmarks for large language model evaluation. arXiv [csCL] 2024. http://arxiv.org/abs/2410.12857. [3] Kim Y, Jeong H, Chen S, Li...

    arXiv 2023

  2. [3]

    unspecialized humans

    http://arxiv.org/abs/2411.15114. [37] Gema AP, Leang JOJ, Hong G, Devoto A, Mancino ACM, Saxena R, et al. Are We Done with MMLU? arXiv [csCL] 2024. http://arxiv.org/abs/2406.04127. [38] Glazer E, Erdil E, Besiroglu T, Chicharro D, Chen E, Gunning A, et al. FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI. arXiv [csAI] 2024. h...

    arXiv 2024

  3. [2024]

    [19] Li N, Pan A, Gopal A, Yue S, Berrios D, Gatti A, et al

    http://arxiv.org/abs/2407.10362. [19] Li N, Pan A, Gopal A, Yue S, Berrios D, Gatti A, et al. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. arXiv [csLG] 2024. http://arxiv.org/abs/2403.03218. [20] O’Donoghue O, Shtedritski A, Ginger J, Abboud R, Ghareeb AE, Booth J, et al. BioPlanner: Automatic evaluation of LLMs on protocol pl...

    arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.