EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

Carl Yang; Dehai Min; Guanchen Wu; Lu Cheng; Max S. Y. Lau; Mingyang Wei; Qi He; Wei Jin; Yuzhang Xie; Zewen Liu

REVIEW 4 major objections 5 minor 1 cited by

EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EpiQAL shows current LLMs are weak at epidemiological reasoning, with multi-step inference the hardest step.

desk verdict A genuinely new benchmark with a plausible headline finding, but the B/C gold labels rest on LLM checkers and low human agreement, so treat the rankings as provisional until labels are independently verified. read the letter →

arxiv 2601.03471 v3 pith:X6XLPECZ submitted 2026-01-06 cs.CL cs.AI

Mingyang Wei , Dehai Min , Zewen Liu , Yuzhang Xie , Guanchen Wu , Ziyang Zhang , Carl Yang , Max S. Y. Lau

show 3 more authors

Qi He Lu Cheng Wei Jin

This is my paper

classification cs.CLcs.AI

keywords epidemiologicalquestionansweringlargelanguagemodelsmulti-stepinferencebenchmarkconstructionevidencegroundingconclusionreconstructionchain-of-thoughtpromptingset-basedevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that population-level epidemiological reasoning is a distinct capability that current large language models have not mastered, and that it can be measured separately from clinical knowledge or general reading comprehension. It introduces EpiQAL, a benchmark of about 1,430 questions in three subsets built from open-access neglected-tropical-disease articles: text-grounded factual recall, multi-step inference that combines several pieces of evidence, and conclusion reconstruction with the Discussion section withheld. Across fourteen models, no model comes close to ceiling performance; multi-step inference is the steepest drop, rankings change depending on the subset, and parameter count does not predict success. A sympathetic reader would care because the benchmark offers a way to tell apart three skills—retrieving facts, integrating evidence, and judging what evidence can support—that matter for using research literature in public health decisions.

What carries the argument

The central object is EpiQAL's three-subset structure. EpiQAL-A tests retrieval of facts stated verbatim in the source article; EpiQAL-B requires combining at least two pieces of evidence with epidemiological principles, with distractors built around causal reversal, variable confusion, and assumption violations; EpiQAL-C masks the Discussion section and asks which conclusions can be derived from the remaining text. The construction pipeline—expert taxonomy, multi-model verification with vote-ratio thresholds and human review of uncertain cases, difficulty scoring and stem refinement to remove lexical shortcuts, and set-based Exact Match/F1 evaluation—is what lets the authors attribute perfo

What would settle it

Have two practicing epidemiologists independently re-annotate all EpiQAL-B and EpiQAL-C items without seeing the generated labels. If the proportion of gold 'correct' answers they endorse falls well below the models' best scores, or if their disagreements concentrate on multi-step items, the benchmark's central diagnostic conclusion would not survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that EpiQAL is the first diagnostic benchmark that isolates text-grounded recall, multi-step inference, and conclusion reconstruction in epidemiological question answering over full research articles. The central empirical finding is that current LLMs perform well on explicit fact retrieval (most exceed 0.76 Exact Match on EpiQAL-A) but degrade sharply when answers require combining multiple findings (best Exact Match 0.868 with CoT on EpiQAL-B), and that a modest 7B-parameter model can outperform much larger models on conclusion reconstruction. The paper further claims that model rankings are subset-dependent and that scale alone does not predict success,

Load-bearing premise

The gold answers are correct: correct options for the hardest subsets were generated by one LLM, accepted mostly by three other LLM checkers with only 1.5–3.9% of options seeing human review, and a 60-question spot-check had only 37–40.8% exact annotator agreement on B/C; if those answers are wrong or ambiguous, every model ranking and the claim that multi-step inference is the bottleneck would collapse.

Editorial extensions

If this is right

If the central claim holds, multi-step epidemiological inference is an open capability gap for current large language models, not a solved problem.
Rankings that shift across subsets mean a single aggregate leaderboard can mask real strengths and weaknesses; evaluation should be reported per capability.
Chain-of-thought prompting helps multi-step inference but can hurt when evidence is incomplete, so prompting strategies need to be matched to task type.
The F1–Exact Match gap shows that many models over-select plausible distractors; for public-health uses where false positives carry real costs, precision-focused evaluation matters.
EpiQAL's three subsets provide reusable diagnostic signals for improving evidence grounding and inferential reasoning in biomedical question answering systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not pursue is to test whether fine-tuning on EpiQAL-B transfers to real-world outbreak-analysis tasks; the benchmark's structure makes that a direct next experiment.
The paper's own finding that knowledge-graph augmentation contributed little during construction hints that retrieval-based augmentation at test time may also be limited when source documents are information-dense.
The low inter-annotator agreement on the human-quality sample (37–40.8% exact agreement on B/C) implies the gold labels carry more uncertainty than the performance numbers convey; re-annotation by practicing epidemiologists would tell whether the bottleneck claim is robust.
The single-journal, English-only corpus means the benchmark's generality to other disease domains and languages is an open empirical question, though the taxonomy could plausibly be reused there.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

EpiQAL is worth reading. It is the first benchmark I know that separates epidemiological QA into factual recall, multi-step inference, and conclusion reconstruction under masked input, and it ships with a taxonomy, a set-based evaluation protocol, and a public dataset. That is real novelty. The construction pipeline is described in enough detail to reproduce, and the authors are unusually honest about the limitations.

The headline claim — models do worst on multi-step inference and rankings shift across subsets — is broadly consistent with Table 4, but I would not call it established. The load-bearing issue is the gold labels for EpiQAL-B/C. They are generated by one 30B model and accepted by LLM checkers unless the checkers flag them; only 1.5–3.9% of options get human review. The 60-question human evaluation scores answer correctness at ≥2.75/3, which is encouraging, but the annotators are CS PhD students, not epidemiologists, and exact agreement on B/C is 37–41%. That agreement level is exactly what you'd expect if some items have more than one defensible answer. If that's true, the EM rankings and the bottleneck claim could shift. The paper's own limitations concede residual label errors and non-epidemiologist annotators. This is the first thing I'd ask the authors to fix: an expert re-derivation of a random sample of labels, or at least a disagreement analysis.

The difficulty-control loop is a second, softer problem. DiffScore is defined using four models that are then evaluated in Table 4, so the 'models struggle' result is partially baked in for those four. The effect is probably modest — the threshold is low and most instances pass unrefined — but it should be acknowledged or re-run with a held-out difficulty pool.

The contamination issue is real but the authors already concede it, and the masked-input design plus synthesized distractors makes pure memorization less likely. Minor: abstract says fifteen models, body says fourteen. Single-journal English corpus is a scope limitation, not a flaw.

Bottom line: this deserves serious refereeing. It addresses a real gap, and the resource itself is useful even if some claims are provisional. I'd ask for label verification, confidence intervals on the model scores (they're single-run point estimates), and a contamination check before treating it as a definitive diagnostic. For a reading group, it's a good example of how to build a benchmark pipeline, warts and all.

Referee Report

4 major / 5 minor

Summary. The paper introduces EpiQAL, a document-grounded multiple-choice QA benchmark for epidemiological reasoning over open-access research articles. It consists of three subsets: EpiQAL-A for text-grounded factual recall, EpiQAL-B for multi-step inference, and EpiQAL-C for conclusion reconstruction with the Discussion section masked. Construction uses a taxonomy, a Qwen3-30B-A3B generator, three-LLM verification with thresholded human review, and difficulty control via a four-model DiffScore pool. The authors evaluate fourteen models and report that model performance is well below ceiling, that multi-step inference is the largest bottleneck, that rankings are subset-dependent, that scale does not predict success, and that Chain-of-Thought prompting helps mainly on EpiQAL-B. The paper also provides distractor-level deception analysis and releases code/data.

Significance. If the benchmark's labels are trustworthy, EpiQAL would fill a real gap: it targets population-level epidemiological inference rather than clinical QA, uses full documents rather than abstracts, and separates factual recall, multi-step synthesis, and masked-conclusion reconstruction. The paper is transparent about its construction pipeline, ships a released benchmark and code, grounds EpiQAL-A answers in verbatim spans, and provides a fine-grained distractor taxonomy. The distractor-deception analysis is a useful diagnostic contribution. However, the central value of the benchmark depends on the validity of the automatically generated gold labels for EpiQAL-B/C, and that validity is not yet established to the standard the headline claims require.

major comments (4)

[§3.4, Table 2, §4.2.2/Table 3] The gold labels for EpiQAL-B/C are generated by Qwen3-30B-A3B and accepted by three LLM checkers, with human review of only 1.5–3.9% of options. The post-hoc human evaluation (60 questions, 4.2% of the benchmark) rates quality dimensions but does not independently re-derive the correct-answer sets, and exact annotator agreement on B/C is only 37.0–40.8%. If accepted correct options are not actually derivable from the evidence, or rejected distractors are also derivable, then the EM rankings, the A-vs-B/C difficulty gap, and the multi-step-bottleneck claim are not trustworthy. The authors should provide expert epidemiological adjudication of a substantially larger sample of B/C items, ideally all of them, together with quantitative agreement between experts and the automated labels.
[§3.5, §4.1, Table 4] Difficulty control is circular for the paper's headline. DiffScore is averaged over four models—GPT-5-mini, DeepSeek-V3.2-Reasoner, Qwen3-32B, and Phi-4-mini-instruct—and the same four models appear in the main evaluation (Table 4). An item is 'hard' if those models fail it, and then the paper reports that those same models struggle on the benchmark. This partially builds in the conclusion that current LLMs perform poorly. The authors should report results on the original, unscreened instance set, or use a held-out model pool for difficulty control that is disjoint from the evaluation set, to show that the difficulty gap is not an artifact of the selection procedure.
[§4.3.2, Table 4, §4.5] All model scores are single-run point estimates. Several of the paper's claims are based on small exact-match differences, for example GPT-5-mini's CoT degradation on EpiQAL-C (0.599→0.555) and Mistral-7B outperforming Mistral-Large (0.789 vs. 0.644 on EpiQAL-B). With temperatures 0.3–1.0 and stochastic decoding, these differences may lie within run-to-run variation. The paper should report multiple runs with means and confidence intervals, or at least a statistical significance test for the specific comparisons used in the Discussion, before drawing conclusions about model rankings and CoT effects.
[§4.2.2, Table 3, Appendix E] The human quality evaluation is too weak to certify the correctness of the B/C answer keys. The annotators are CS PhD students rather than epidemiologists, and exact agreement on B/C is 37.0–40.8%. The paper attributes much of this to one stricter annotator, but such low agreement on a 3-point scale still indicates substantial label ambiguity. The authors should report per-item disagreement, separate agreement on the Answer Correctness dimension from other dimensions, and ideally compare human judgments with expert epidemiological labels to show that the answer keys are not ambiguous or non-unique.

minor comments (5)

[Abstract] The abstract says 'fifteen models' while the rest of the paper and Table 4 report fourteen models. Please align the count.
[Appendix B.2 / Tables 7–9] The taxonomy topics are described in long paragraphs; a short one-line usage example for each topic would make the constraint schema easier to apply and evaluate.
[§2 / References] Raza et al. appears twice as 2022a and 2022b with the same title and venue. Please merge or distinguish properly.
[Table 3] The low exact-agreement rates for EpiQAL-B and C should be discussed as a limitation in the main text, not only in the appendix, because they directly qualify the answer-correctness scores.
[Table 5] The dagger symbol on 'Background only' in Table 5 lacks a matching footnote in the main text; the reader has to infer the explanation from the table caption and later appendices.

Circularity Check

1 steps flagged · score 5.0 of 10

Difficulty screening and evaluation share the same model pool, so the headline performance gap is partly built in; the benchmark retains independent grounding.

self definitional [Section 3.5 (Difficulty Control); Section 4.1; Table 4]
"We estimate instance difficulty using a pool of models ranging from small to large. For each model, we compare the predicted answer set A with the reference set Oc using set-based F1 and Exact Match (Appendix B.1), then combine them into a difficulty score: DiffScore= 1−(α·F 1 + (1−α)·EM)... We average DiffScore across the model pool. Items below a threshold are treated as easy and passed to stem refinement. ... Difficulty control uses a four-model pool (GPT-5-mini, DeepSeek-V3.2-Reasoner, Qwen3-32B, Phi-4-mini-instruct) with α= 0.3 and threshold θd = 0.2 ."

The same four models whose F1/EM define DiffScore are the ones reported in Table 4 for EpiQAL-B/C (GPT-5-mini, DeepSeek-Reasoner, Qwen3-32B, Phi-4-mini). Since instances with DiffScore below θd are stem-refined until the pooled score clears the threshold, the final B/C sets are selected to be hard for exactly these models. Their low EM/F1 in Table 4 is therefore partly an output of the selection filter, not an independent measurement; the claim that multi-step inference is the primary bottleneck is partially built into the benchmark for the pool models. Independent support remains from non-pool models (e.g., Mistral-7B, Llama-3.2-3B, GPT-4o-mini) and source-document anchoring, so the circularity is partial rather than total.

full rationale

The derivation chain is largely self-contained: questions are anchored to source documents, labels are checked by a multi-LLM group with human review of flagged options, and headline results are reported for fourteen models, most outside the difficulty-judging pool. I found no load-bearing self-citation chain and no imported uniqueness/ansatz: cited prior work (Xie et al. 2024; Min et al. 2025; Wu et al. 2025b; etc.) supports auxiliary components such as entity linking and retrieval, but does not establish the benchmark's conclusions. The one genuine circularity is the overlap between the difficulty-control pool and the evaluated models. DiffScore is computed from F1/EM of GPT-5-mini, DeepSeek-Reasoner, Qwen3-32B, and Phi-4-mini; instances below θd are stem-refined until the pooled score clears the threshold. The same four models' B/C scores in Table 4 are therefore filtered by their own errors, so the claim that multi-step inference is the primary bottleneck is partly built into the benchmark for these models. This does not eliminate all independent content: non-pool models also show the pattern, and human evaluation of a 60-question sample provides some external check. The gold-label concern raised in the Limitations (residual errors, non-epidemiologist annotators) is a validity risk rather than a circularity, and the paper explicitly acknowledges it. Overall score 5 reflects one partial by-construction reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on LLM-generated labels, the representativeness of the PLOS NTD corpus, and the assumption that public articles have not been memorized. These are stated in the paper but remain unverified load-bearing assumptions.

free parameters (4)

alpha (DiffScore weighting) = 0.3
Weights Exact Match more heavily than F1 in DiffScore; hand-set and used to select difficult items.
difficulty threshold theta_d = 0.2
Chosen from a sharp transition in the DiffScore distribution (Appendix B.6); determines which items are refined and therefore the final difficulty of EpiQAL-B/C.
checker vote thresholds theta_h, theta_c = 6/9 accept, 5/9 reject
Hand-set in Appendix C.3; options between thresholds go to human review.
refinement iterations Tr and retrieval budget Kr = Tr=3, Kr=6
Chosen based on diminishing difficulty gains (Appendix F.1); affects final stems.

assumptions (4)

ad hoc to paper LLM-generated gold labels for EpiQAL-B/C are correct enough to serve as ground truth.
Multi-model verification plus human review of only 1.5-3.9% flagged options (Table 2) and 60 human-evaluated questions (Appendix E) underpin all model performance claims; residual errors are acknowledged in Limitations.
domain assumption The PLOS NTD corpus and taxonomy support claims about epidemiological reasoning broadly.
Corpus is one journal, English-only, and limited to neglected tropical diseases; acknowledged in Limitations.
domain assumption Models have not memorized the source articles during pretraining.
The authors state they do not perform contamination analysis; public PLOS articles may be in training data.
domain assumption Stem refinement removes lexical shortcuts without adding answer clues.
Appendix F.1 shows refinement increases difficulty, but Iteration 3 can over-expand and inadvertently supply domain knowledge, so this assumption is imperfect.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning." pith.science (2026). https://pith.science/paper/X6XLPECZ

@misc{pith2026260103471,
  author       = {Pith},
  title        = {Pith review of: EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X6XLPECZ}},
  note         = {Machine review of arXiv:2601.03471}
}

read the original abstract

Reliable epidemiological reasoning requires synthesizing study evidence to infer disease burden, transmission dynamics, and intervention effects at the population level. Existing medical question answering benchmarks primarily emphasize clinical knowledge or patient-level reasoning, yet few systematically evaluate evidence-grounded epidemiological inference. We present EpiQAL, the first diagnostic benchmark for epidemiological question answering across diverse diseases, comprising three subsets built from open-access literature. The three subsets progressively test factual recall, multi-step inference, and conclusion reconstruction under incomplete information, and are constructed through a quality-controlled pipeline combining taxonomy guidance, multi-model verification, and difficulty screening. Experiments on fifteen models spanning open-source and proprietary systems reveal that current LLMs show limited performance on epidemiological reasoning, with multi-step inference posing the greatest challenge. Model rankings shift across subsets, and scale alone does not predict success. Chain-of-Thought prompting benefits multi-step inference but yields mixed results elsewhere. EpiQAL provides fine-grained diagnostic signals for evidence-grounding, inferential reasoning, and conclusion reconstruction.

Figures

Figures reproduced from arXiv: 2601.03471 by the authors.

**Figure 2.** Version selection distribution across difficulty [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

**Figure 3.** DiffScore distribution across stem refinement iterations for EpiQAL-B (left) and EpiQAL-C (right), [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Class distribution for EpiQAL-A and EpiQAL-B. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

**Figure 5.** Figure 5: Topic distribution for EpiQAL-A and EpiQAL-B. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs
cs.AI 2026-05 unverdicted novelty 6.0 of 10

EHRBench uses an EHR-LLM-KB pipeline to automatically create 960,067 reliable QA items spanning diagnosis, treatment, and prognosis for large-scale LLM evaluation in clinical decision making.

Reference graph

Works this paper leans on

4 extracted references · 4 linked inside Pith · cited by 1 Pith paper

[2]

Preprint, arXiv:2104.14337

Dynabench: Rethinking benchmarking in nlp. Preprint, arXiv:2104.14337. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq- qa: A manually curated corpus for biomedical ques- tion answering.Scientific Data, 10. Seongyun Lee, Hyunjae Kim, and Jaewoo Kang. 2023. Liquid: a framework for list question answe...

arXiv 2023
[4]

Which two Acanthamoeba species were used in the study to evaluate their interaction with Leishmania pro- mastigotes?

Can question generation debias question an- swering models? a case study on question–context lexical overlap. InProceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 63–72, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, ...

arXiv 2025
[2011]

PLOS ONE, 6(7):e21704

The use of research evidence in public health decision making processes: Systematic review. PLOS ONE, 6(7):e21704. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain ques- tion answering. InProceedings of the Conference on Health, Inference, and Learning, volu...

arXiv 2022
[2021]

Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Ser- gio E Baranzini

Measuring massive multitask language under- standing.Preprint, arXiv:2009.03300. Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Ser- gio E Baranzini. 2017. Systematic integration of biomedical knowledge prioritizes drugs for repurpos- ing.eLife, 6:e26726. Gautier...

arXiv 2009

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Preprint, arXiv:2104.14337

Dynabench: Rethinking benchmarking in nlp. Preprint, arXiv:2104.14337. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq- qa: A manually curated corpus for biomedical ques- tion answering.Scientific Data, 10. Seongyun Lee, Hyunjae Kim, and Jaewoo Kang. 2023. Liquid: a framework for list question answe...

arXiv 2023

[2] [4]

Which two Acanthamoeba species were used in the study to evaluate their interaction with Leishmania pro- mastigotes?

Can question generation debias question an- swering models? a case study on question–context lexical overlap. InProceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 63–72, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, ...

arXiv 2025

[3] [2011]

PLOS ONE, 6(7):e21704

The use of research evidence in public health decision making processes: Systematic review. PLOS ONE, 6(7):e21704. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain ques- tion answering. InProceedings of the Conference on Health, Inference, and Learning, volu...

arXiv 2022

[4] [2021]

Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Ser- gio E Baranzini

Measuring massive multitask language under- standing.Preprint, arXiv:2009.03300. Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Ser- gio E Baranzini. 2017. Systematic integration of biomedical knowledge prioritizes drugs for repurpos- ing.eLife, 6:e26726. Gautier...

arXiv 2009