SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding

Chuan Qin; Hengshu Zhu; Jinmiao Chen; Meng Xiao; Qingqing Long; Xiaohan Huang; Yuanchun Zhou

arxiv: 2601.12805 · v4 · pith:EIKF5R2Enew · submitted 2026-01-19 · 🧬 q-bio.GN · cs.AI· cs.CL

SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding

Xiaohan Huang , Meng Xiao , Chuan Qin , Qingqing Long , Jinmiao Chen , Yuanchun Zhou , Hengshu Zhu This is my paper

Pith reviewed 2026-05-25 07:02 UTC · model grok-4.3

classification 🧬 q-bio.GN cs.AIcs.CL

keywords LLM benchmarkinggene function inferencebiomedical AIhallucination evaluationcell atlas interpretationbiological reasoningliterature groundingSciHorizon-GENE

0 comments

The pith

LLMs display wide variation in gene reasoning and often fail to produce complete, literature-grounded functional interpretations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SciHorizon-GENE, a benchmark built from authoritative databases that holds curated knowledge on more than 190,000 human genes and over 540,000 questions spanning gene-to-function reasoning. It tests models on four targeted dimensions: sensitivity to research attention, hallucination rates, answer completeness, and dependence on literature. Systematic runs across many general-purpose and biomedical LLMs uncover large differences in performance and repeated shortfalls in faithful, complete outputs. These results matter because accurate gene-level reasoning is required for safe application of LLMs to cell atlas interpretation and related biological tasks. The benchmark supplies a concrete way to measure progress toward reliable knowledge-enhanced pipelines.

Core claim

SciHorizon-GENE integrates knowledge for over 190K human genes into more than 540K questions and evaluates LLMs on research attention sensitivity, hallucination tendency, answer completeness, and literature influence, exposing substantial heterogeneity in gene-level reasoning capabilities together with persistent shortfalls in faithful, complete, and literature-grounded functional interpretations.

What carries the argument

The SciHorizon-GENE benchmark, which organizes questions around four biologically critical perspectives to expose failure modes in gene-to-function reasoning.

If this is right

LLMs require explicit model selection and validation before deployment in knowledge-enhanced cell atlas interpretation.
Development efforts should target improvements in faithfulness, completeness, and literature grounding for gene-level outputs.
The benchmark supplies a reusable test bed for tracking progress in biological reasoning capabilities.
Heterogeneity across models indicates that current general-purpose and biomedical LLMs are not interchangeable for gene-function tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Extending the benchmark to multi-gene or pathway-level questions could reveal whether the observed gaps scale with task complexity.
The four evaluation perspectives might be adapted to measure similar reasoning limits in other scientific domains that rely on curated knowledge bases.
If the heterogeneity persists after retrieval augmentation, it would point to deeper architectural constraints rather than simple knowledge gaps.

Load-bearing premise

The questions drawn from authoritative databases accurately represent the range of gene-to-function scenarios and failure modes that would affect LLM use in biological interpretation pipelines.

What would settle it

A single new LLM that scores uniformly high on all four evaluation perspectives across the full set of 540K questions without extra training or retrieval would contradict the reported persistent challenges.

Figures

Figures reproduced from arXiv: 2601.12805 by Chuan Qin, Hengshu Zhu, Jinmiao Chen, Meng Xiao, Qingqing Long, Xiaohan Huang, Yuanchun Zhou.

**Figure 1.** Figure 1: Observations of LLM behavior on gene-related tasks, motivating the need for our gene-centric benchmark. (a) Model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: The benchmark integrates curated biological databases and verified literature sources to construct gene nodes. These [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: PubMed reference count distribution for human [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Model performance on three tasks for high- and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 6.** Figure 6: Completeness evaluation of LLMs. All questions [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

**Figure 7.** Figure 7: Comparison of Gene Ontology answering perfor [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

**Figure 8.** Figure 8: Comparison of functional summary answering per [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

**Figure 10.** Figure 10: Each example corresponds to a specific evaluation perspective. A and B indicate variants within the same genomic [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

**Figure 11.** Figure 11: Prompt templates for all question types, including the unified system prompt and task-specific instruction prompts. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗

read the original abstract

Large language models (LLMs) have shown growing promise in biomedical research, particularly for knowledge-driven interpretation tasks. However, their ability to reliably reason from gene-level knowledge to functional understanding, a core requirement for knowledge-enhanced cell atlas interpretation, remains largely underexplored. To address this gap, we introduce SciHorizon-GENE, a large-scale gene-centric benchmark constructed from authoritative biological databases. The benchmark integrates curated knowledge for over 190K human genes and comprises more than 540K questions covering diverse gene-to-function reasoning scenarios relevant to cell type annotation, functional interpretation, and mechanism-oriented analysis. Motivated by behavioral patterns observed in preliminary examinations, SciHorizon-GENE evaluates LLMs along four biologically critical perspectives: research attention sensitivity, hallucination tendency, answer completeness, and literature influence, explicitly targeting failure modes that limit the safe adoption of LLMs in biological interpretation pipelines. We systematically evaluate a wide range of state-of-the-art general-purpose and biomedical LLMs, revealing substantial heterogeneity in gene-level reasoning capabilities and persistent challenges in generating faithful, complete, and literature-grounded functional interpretations. Our benchmark establishes a systematic foundation for analyzing LLM behavior at the gene scale and offers insights for model selection and development, with direct relevance to knowledge-enhanced biological interpretation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

SciHorizon-GENE offers a large new benchmark for LLM gene reasoning but the abstract leaves the question curation and validation steps unshown.

read the letter

The paper's main contribution is SciHorizon-GENE itself: a benchmark built from external databases covering 190K genes and 540K questions, aimed at four angles—research attention sensitivity, hallucination, completeness, and literature influence. That scale and those targeted perspectives are new for gene-to-function LLM testing and directly address a stated gap in biomedical interpretation tasks like cell type annotation. Credit to the authors for grounding it in authoritative sources rather than made-up examples and for framing the evaluation around failure modes that matter for safe use in life sciences pipelines. The abstract also states they ran a wide set of general and biomedical LLMs and saw heterogeneity plus ongoing problems with faithful outputs. That setup could be useful for model selection work if the numbers hold. The soft spot is the lack of any reported detail on how the questions were generated, sampled, or checked for representativeness. No mention of expert review, coverage of mechanism-oriented cases, or controls for database artifacts. Without that, the central claims about LLM limitations rest on unverified assumptions about what the benchmark actually measures. The stress-test note on real-world workflow alignment is fair based on what's shown. This is the sort of paper that belongs in a reading group for people doing LLM evaluation in biology, mainly to see the methods and raw results sections. It deserves peer review because the scale and framing fill an underexplored spot, even though the current description is too high-level to judge soundness yet.

Referee Report

1 major / 1 minor

Summary. The manuscript introduces SciHorizon-GENE, a large-scale benchmark constructed from authoritative biological databases containing curated knowledge for over 190K human genes and more than 540K questions. It evaluates state-of-the-art LLMs on four perspectives—research attention sensitivity, hallucination tendency, answer completeness, and literature influence—revealing substantial heterogeneity in gene-level reasoning capabilities and persistent challenges in generating faithful, complete, and literature-grounded functional interpretations relevant to cell atlas interpretation.

Significance. If the benchmark questions validly capture real-world failure modes in gene-to-function reasoning, the findings would provide a valuable systematic foundation for analyzing LLM behavior at the gene scale and informing model selection and development in biomedical applications.

major comments (1)

[Abstract] Abstract: The central claims of substantial heterogeneity and persistent challenges in LLM gene-to-function reasoning rest on SciHorizon-GENE accurately representing diverse real-world scenarios. The description provides no details on question validation, bias controls, sampling strategy, coverage of mechanism-oriented analysis, or validation of the curation process against expert workflows in cell-type annotation tasks.

minor comments (1)

[Abstract] Abstract: Specific performance metrics, example questions, and quantitative results are absent, making it difficult to assess the scale of reported heterogeneity.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback emphasizing the need for greater methodological transparency to support the benchmark's claims. We address the single major comment below and will incorporate clarifications in the revised manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: The central claims of substantial heterogeneity and persistent challenges in LLM gene-to-function reasoning rest on SciHorizon-GENE accurately representing diverse real-world scenarios. The description provides no details on question validation, bias controls, sampling strategy, coverage of mechanism-oriented analysis, or validation of the curation process against expert workflows in cell-type annotation tasks.

Authors: The abstract is intentionally brief. The full manuscript (Section 3) describes construction from authoritative databases covering >190K genes and >540K questions, with explicit inclusion of mechanism-oriented scenarios via pathway, interaction, and regulatory data relevant to cell-type annotation. Sampling is exhaustive (all curated entries) rather than subsampled. We acknowledge the abstract and main text lack explicit subsections on question validation procedures, bias controls, and direct comparison to expert cell-type annotation workflows. We will add a dedicated 'Benchmark Validation and Bias Controls' subsection detailing curation validation steps, relevance checks for cell atlas tasks, and any bias mitigation (e.g., source diversity), plus a brief reference in the abstract. This addresses the concern without changing results. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark constructed from external databases; LLM evaluations independent of paper inputs

full rationale

The paper introduces SciHorizon-GENE as a benchmark built directly from authoritative external biological databases covering 190K genes and 540K questions. The central claims concern observed heterogeneity in LLM performance across four perspectives (research attention sensitivity, hallucination tendency, answer completeness, literature influence) when evaluated on this benchmark. No equations, parameter fits, self-citations, or ansatzes are invoked as load-bearing steps in the derivation chain. The benchmark construction and evaluation results do not reduce to the paper's own inputs by definition or construction, satisfying the criteria for a self-contained, non-circular analysis.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; main unverified premise is that database-curated questions capture real biological reasoning failure modes. No free parameters or invented entities are described.

axioms (1)

domain assumption Authoritative biological databases provide accurate and representative gene knowledge sufficient for constructing a benchmark that reflects real-world functional interpretation needs.
The benchmark integrates curated knowledge for over 190K human genes from these databases.

pith-pipeline@v0.9.0 · 5781 in / 1168 out tokens · 44278 ms · 2026-05-25T07:02:27.323232+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

BioHarness: Substrate-Aware Evidence Assembly for Biomedical Question Answering across Literature, Knowledge Bases, and Biological Atlases
q-bio.QM 2026-06 unverdicted novelty 6.0

BioHarness improves pooled biomedical QA score from 65.9 to 71.0 on 19,302 items by using staged, substrate-aware evidence assembly that escalates only when needed.