The reviewed record of science sign in
Pith

arxiv: 2607.06452 · v1 · pith:V2VHWEXO · submitted 2026-07-07 · cs.CL · cs.AI

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

Taeyun Roh , Eunha Lee , Wonjune Jang , Sohyun Chung , Junha Jung , Jaewoo Kang This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 05:07 UTCglm-5.2pith:V2VHWEXOrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords biomedical question answeringquestion-type-aware routinglarge language modelsmulti-agent systemsensemble methodschain-of-thought promptingBioASQ
0
0 comments X

The pith

Type-Aware Pipelines Beat One-Size-Fits-All for Biomedical QA

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that biomedical question answering systems should route questions to different inference pipelines based on the expected answer type (yes/no, factoid, or list) rather than using a single prompting strategy for all questions. The authors build and deploy a system for the BioASQ 14b challenge that uses snippet-shuffled majority voting for binary questions, chain-of-thought in-context learning with multi-model consensus for factoid questions, and a four-agent collaboration pipeline for list questions. The central claim is that matching the inference mechanism to the structural demands of the answer type yields more reliable results than a uniform approach.

Core claim

The paper demonstrates that a type-routed architecture—combining ensemble voting for yes/no, retrieval-based in-context learning with consensus filtering for factoid, and a sequential multi-agent verification pipeline for list questions—achieves competitive performance on BioASQ 14b, including first place on the factoid subtask of Batch 4 by MRR. The development experiments on BioASQ 13b show that the best strategy varies by question type: ensemble voting stabilized yes/no decisions, in-context learning improved factoid entity identification, and agent-based architectures outperformed single-pass inference on list questions.

What carries the argument

The central mechanism is a type router that dispatches each question to one of three specialized pipelines: (1) a yes/no branch using snippet-order shuffling across four LLMs with selective verification on disagreements, (2) a factoid branch using full-snippet input with BioBERT-retrieved chain-of-thought demonstrations and cross-model consensus filtering, and (3) a list branch using a four-agent chain (Evidence Analyst, Reasoning Agent, Verification Agent, Supervisor) that separates evidence extraction, candidate generation, auditing, and final aggregation.

If this is right

  • Type-aware routing could be extended to additional answer formats such as summary or explanatory questions, each with its own specialized pipeline.
  • The multi-agent verification architecture for list questions could be adapted to other domains requiring high-precision entity extraction from scattered evidence, such as legal or financial document analysis.
  • The selective verification trigger on yes/no questions—only invoking agents when models disagree—suggests a cost-efficient pattern for deploying expensive verification only on uncertain cases.
  • Retrieval-based in-context learning with verified chain-of-thought exemplars could be generalized as a technique for improving surface-form matching in any exact-answer evaluation setting.

Load-bearing premise

The headline results in the main results table report the maximum score per metric across five separately submitted system variants, not the performance of any single fixed configuration; this means no one system achieves all the reported numbers simultaneously, and the relative contribution of each design choice cannot be cleanly isolated from variant selection.

What would settle it

If a single uniform prompting strategy applied across all question types matched or exceeded the type-routed system on the same set of BioASQ 14b questions, the central design claim would be undermined.

Figures

Figures reproduced from arXiv: 2607.06452 by Eunha Lee, Jaewoo Kang, Junha Jung, Sohyun Chung, Taeyun Roh, Wonjune Jang.

Figure 1
Figure 1. Figure 1: Overall architecture for exact-answer generation pipeline in BioASQ 14b. The system is inference-time and prompt-driven. Multiple LLMs are used as complementary biomed￾ical reasoners, and their outputs are aggregated to reduce sensitivity to a single model’s prompt inter￾pretation. The main aggregation logic differs by question type: majority voting for yes/no, consensus filtering for factoid answers, and … view at source ↗
Figure 2
Figure 2. Figure 2: List-agent workflow for BioASQ list questions, combining Evidence Agent, Reasoning Agent, and Verification Agent. Verified outputs are reviewed by a supervisor to produce a final evidence-supported list answer. requested, or under-generate by missing valid entities that appear only in low-salience snippets. By separating evidence extraction, answer generation, and entity-level verification, the list subsys… view at source ↗
read the original abstract

Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents. This study presents a question-type-specific large language model (LLM) framework for BioASQ 14b Task B, designed to improve answer robustness and evidence grounding in biomedical question answering. Rather than applying a single prompting strategy to all questions, the framework selects different inference procedures for yes/no, factoid, and list questions according to their distinct reasoning and evaluation requirements. For yes/no questions, snippet shuffling and self-reflection are used to reduce sensitivity to evidence ordering and improve decision stability. For factoid questions, full-snippet input is combined with chain-of-thought-based in-context learning to support accurate biomedical entity identification. For list questions, a multi-agent architecture is employed, in which evidence extraction, candidate generation, answer verification, and final aggregation are handled collaboratively. Preliminary experiments on BioASQ 13b were used to identify effective inference strategies for each question type, and the resulting framework was subsequently evaluated in the official BioASQ 14b Task B challenge. In the official evaluation, our framework showed competitive performance across multiple batches and achieved first place in the factoid subtask of Batch 4. These results demonstrate the effectiveness of combining question-type-specific inference, ensemble prediction, and agent-based verification for reliable biomedical question answering.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. This paper presents the ku_dmis system for BioASQ 14b Task B, which routes questions by type (yes/no, factoid, list) to different LLM-based inference strategies. For yes/no questions, the system uses snippet shuffling with majority voting and selective verification. For factoid questions, it combines full-snippet input with retrieval-based chain-of-thought in-context learning and multi-model consensus filtering. For list questions, it employs a four-agent collaborative pipeline (evidence analyst, reasoning, verification, supervisor). Development experiments on BioASQ 13b informed strategy selection, and the final system was evaluated on the official BioASQ 14b test set across four batches, achieving competitive results including first place in factoid MRR for Batch 4.

Significance. The paper provides a clear, well-structured system description for a competitive shared-task submission. The answer-type-aware routing design is sensible, and the list-question multi-agent pipeline is described in sufficient detail (including prompt templates in Appendix A) to be reproducible in principle. The official BioASQ 14b evaluation provides an external benchmark with independently prepared gold answers. The variant analysis in Table 5 offers useful insight into which design choices matter for which question types. The paper is appropriate in scope for a CLEF working notes contribution.

major comments (3)
  1. §5.1, Table 4 and Abstract/Conclusion: Table 4 reports per-metric maximums across five submission variants (ku_dmis1–5), not the performance of any single fixed system. The authors acknowledge this in §6, but the abstract ('our framework showed competitive performance across multiple batches') and conclusion ('the system achieved average scores of 0.9110... 0.4626... 0.4439') present these numbers as if they describe one system. Table 5 confirms that no single variant achieves all reported values. This framing overstates the evidence for the central claim. The abstract and conclusion should either (a) report a single representative variant's scores, or (b) explicitly state in both places that the numbers are per-metric maxima across variants, not a single system's performance.
  2. §4.1, Table 1: The 'previous baseline' (the authors' own BioASQ 13b system [8]) achieves 1.0000 macro F1 on yes/no questions, strictly outperforming all proposed GPT-4o variants (0.9444) and GPT-5 variants (0.9244). The paper states it 'extended this baseline with self-reflection, repeated inference, and a verification agent to improve robustness,' yet the extended system performs worse. The paper offers no analysis of why the proposed yes/no strategies underperform the baseline on the development set. Since the yes/no subsystem is a load-bearing component of the type-aware framework claim, this gap should be addressed: either explain the discrepancy (e.g., test-set size, baseline overfitting to 13b) or qualify the yes/no effectiveness claim.
  3. §4.2, Table 2: For GPT-5.2, adding CoT-ICL2 reduces MRR from 0.5320 to 0.4230, a substantial degradation. The paper selects GPT-4o+ICL2 (0.5580 MRR) for the final factoid system, which is reasonable, but the text in §4.2 states that 'in-context learning and full-snippet input were effective in the BioASQ-13b development experiments' without acknowledging that CoT-ICL actively hurt the strongest single model (GPT-5.2). The claim that CoT-ICL is effective for factoid QA should be qualified to note that its benefit is model-dependent.
minor comments (6)
  1. Table 1: The yes/no test set appears to contain 17 questions (0.9412 accuracy = 16/17). This small sample size should be noted, as it affects the reliability of the development comparisons.
  2. §4.1: The final yes/no system uses GPT-5 Thinking, GPT-5 Pro, GPT-4o, and Gemini 3.0 Thinking, but Table 1 reports development results for GPT-4o, GPT-5.2, GPT-5.4, Claude Sonnet-4.6, and Opus-4.6. The relationship between the development models and the deployed models is unclear and should be clarified.
  3. §4.4: The submission variants are described only vaguely ('small changes in prompting, answer normalization, and aggregation thresholds'). A table listing the specific differences between ku_dmis1–5 would improve reproducibility.
  4. Figure 1: The 'Selective Verification' box in the yes/no branch and the 'Supervisor Agent' boxes are shown but their connection to the output is not clearly drawn. The figure could be improved.
  5. §5.1: The paper attributes list-question improvement across batches to 'the addition and refinement of the collaborative list-answer pipeline,' but does not clarify whether the pipeline itself changed between batches or whether the improvement reflects test-set difficulty differences. This should be disambiguated.
  6. References [28]–[38]: Several model references point to product announcement pages rather than technical reports. Where model cards or technical reports exist, they should be cited instead.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. All three major comments identify legitimate issues in how results and claims are framed. We agree with each point and will revise the manuscript accordingly: (1) the abstract and conclusion will explicitly state that reported numbers are per-metric maxima across submission variants, not a single system's performance; (2) we will add analysis of why the proposed yes/no strategies underperform the previous baseline on the development set, with appropriate qualification of the yes/no effectiveness claim; and (3) we will qualify the CoT-ICL effectiveness claim for factoid QA to note that its benefit is model-dependent, acknowledging the GPT-5.2 degradation. No standing objections remain.

read point-by-point responses

  1. Referee: §5.1, Table 4 and Abstract/Conclusion: Table 4 reports per-metric maximums across five submission variants (ku_dmis1–5), not the performance of any single fixed system. The abstract and conclusion present these numbers as if they describe one system, overstating the evidence for the central claim. The abstract and conclusion should either report a single representative variant's scores or explicitly state that the numbers are per-metric maxima across variants.

    Authors: The referee is correct. The abstract and conclusion currently present per-metric maxima across five submission variants as if they describe a single system, which is misleading. We acknowledge this framing issue. While §6 (Limitations) does note that Table 4 reports per-metric maxima rather than a single fixed system's scores, this qualification should appear in the abstract and conclusion as well, not only in the limitations section. We will revise both the abstract and conclusion to explicitly state that the reported numbers are per-metric maxima across submission variants, not the performance of any single fixed system. We will retain the per-metric maximum presentation in Table 4 itself (as the caption already states this), but will ensure the abstract and conclusion are transparent about this. We considered the alternative of reporting a single representative variant's scores, but since different variants were optimal for different question types and batches, no single variant is truly representative, and reporting one would understate the range of strategies explored. Explicitly labeling the numbers as per-metric maxima is the more honest approach. revision: yes

  2. Referee: §4.1, Table 1: The previous baseline (authors' own BioASQ 13b system [8]) achieves 1.0000 macro F1 on yes/no questions, strictly outperforming all proposed GPT-4o variants (0.9444) and GPT-5 variants (0.9244). The paper states it extended this baseline with self-reflection, repeated inference, and a verification agent to improve robustness, yet the extended system performs worse. No analysis is offered for why the proposed yes/no strategies underperform the baseline on the development set.

    Authors: The referee correctly identifies a gap in our analysis. The previous baseline achieving 1.0000 macro F1 on the BioASQ-13b yes/no development set while all proposed variants score lower is a striking result that we should have addressed. We will add analysis of this discrepancy in the revised manuscript. The most likely explanation is overfitting to the 13b test set: the previous baseline was specifically developed and tuned on 13b data, and the yes/no subset is small (17 questions, as indicated by the granularity of the scores), making perfect performance plausible but not necessarily generalizable. The proposed extensions (self-reflection, repeated inference, verification agent) were designed for robustness on unseen data rather than to maximize performance on the 13b development set specifically. We will add this explanation to §4.1 and qualify the yes/no effectiveness claim to note that the proposed strategies did not improve over the previous baseline on the development set, while noting that the development set size limits the reliability of comparisons. We will also adjust the language in §4.1 to avoid implying that the extensions were demonstrated to improve yes/no performance on the development set. revision: yes

  3. Referee: §4.2, Table 2: For GPT-5.2, adding CoT-ICL2 reduces MRR from 0.5320 to 0.4230, a substantial degradation. The text states that 'in-context learning and full-snippet input were effective in the BioASQ-13b development experiments' without acknowledging that CoT-ICL actively hurt the strongest single model (GPT-5.2). The claim that CoT-ICL is effective for factoid QA should be qualified to note that its benefit is model-dependent.

    Authors: The referee is correct. The current text in §4.2 makes an unqualified claim that CoT-ICL was effective, when Table 2 clearly shows that CoT-ICL2 reduced GPT-5.2's MRR from 0.5320 to 0.4230—a substantial degradation for the strongest single model. This is an important omission. We will revise §4.2 to explicitly acknowledge that the benefit of CoT-ICL is model-dependent: it improved GPT-4o's MRR from 0.4420 to 0.5580 but degraded GPT-5.2's MRR from 0.5320 to 0.4230. We will qualify the general claim about CoT-ICL effectiveness to note this model dependency and explain that the final factoid system's design—using GPT-4o with ICL2 as a primary component while incorporating multiple models with consensus filtering—was motivated in part by this observation. The revised text will present a more balanced picture of when CoT-ICL helps and when it hurts. revision: yes

Circularity Check

0 steps flagged

No circularity detected

full rationale

The paper presents an applied system for BioASQ 14b biomedical question answering. The derivation chain is straightforward: development experiments on BioASQ 13b (Tables 1-3) inform strategy selection, which is then evaluated on the external BioASQ 14b benchmark (Tables 4-5) with gold-standard answers from independent biomedical experts. There are no self-definitional loops, no fitted parameters renamed as predictions, and no self-citation chains that are load-bearing for the central claim. The paper does cite prior work by some of the same authors (reference [8], the 'previous baseline' from BioASQ 13b), but this citation provides a comparison point rather than serving as an unverified premise that the paper's results depend on. The strategies employed (snippet shuffling, CoT-ICL, multi-agent verification) are standard techniques evaluated against an external benchmark, not derived quantities that reduce to their own inputs. The per-metric maximum reporting in Table 4 is acknowledged as a limitation in Section 6, but this is a methodological concern about evidence strength, not circularity. The development-to-test pipeline (13b → 14b) uses distinct datasets, so the design choices informed by 13b experiments are not tautologically validated on the same data. No step in the paper's argument reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The system introduces no new theoretical entities, particles, or mathematical objects. It is an engineering pipeline composed of existing LLM components.

free parameters (5)
  • Consensus threshold for factoid (answers appearing in >=3 model outputs) = 3
    Stated in Section 4.2; chosen based on BioASQ 13b development experiments but no systematic threshold sweep is reported.
  • Number of ICL examples for factoid = 2
    Stated in Table 2 as ICL2; selected from development experiments.
  • Number of snippet shuffles for yes/no = 4
    Stated in Section 4.1; each model queried four times with different snippet orders.
  • Verification confidence threshold for list supervisor = 0.7
    Appears in Appendix A Supervisor Agent prompt as decision rule threshold.
  • Maximum list items from reasoning agent = 7-8
    Stated in Appendix A Reasoning Agent prompt.
axioms (3)
  • domain assumption Gold snippets provided by BioASQ organizers contain sufficient evidence to answer each question
    The system does not perform retrieval; it relies entirely on organizer-provided snippets (Section 2, Task setting).
  • domain assumption Multiple proprietary LLMs produce complementary errors that ensemble aggregation can reduce
    The ensemble design across GPT-5, Claude, and Gemini assumes model-specific biases are partially independent (Section 3).
  • domain assumption Question type is known at inference time
    The type router requires the BioASQ question type label, which is provided by the task setup (Section 3).

pith-pipeline@v1.1.0-glm · 16438 in / 3465 out tokens · 256481 ms · 2026-07-08T05:07:53.285078+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 57 canonical work pages · 11 internal anchors

  1. [1]

    Tsatsaronis, G

    G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos, Y. Almirantis, J. Pavlopoulos, N. Baskiotis, P. Gallinari, T. Artières, A. Ngonga, N. Heino, E. Gaussier, L. Barrio-Alvers, M. Schroeder, I. Androutsopoulos, G. Paliouras, An overview of the BIOASQ large-scale...

    work page doi:10.1186/s12859-015-0564-6 2015

  2. [2]

    A. Nentidis, et al., BioASQ at CLEF2026: The fourteenth edition of the large-scale biomedical semantic indexing and question answering challenge, in: Advances in Information Retrieval, volume 16486 ofLecture Notes in Computer Science, Springer, 2026, pp. 315–324. doi: 10.1007/ 978-3-032-21321-1_42

    work page 2026

  3. [3]

    Krithara, A

    A. Krithara, A. Nentidis, K. Bougiatiotis, G. Paliouras, BioASQ-QA: A manually curated corpus for biomedical question answering, Scientific Data 10 (2023) 170. doi: 10.1038/ s41597-023-02068-4

    work page 2023

  4. [4]

    Liévin, C

    V. Liévin, C. E. Hother, A. G. Motzfeldt, O. Winther, Can large language models reason about medical questions?, Patterns 5 (2024) 100943. doi:10.1016/j.patter.2024.100943

    work page doi:10.1016/j.patter.2024.100943 2024

  5. [5]

    Singhal, T

    K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al., Toward expert-level medical question answering with large language models, Nature medicine 31 (2025) 943–950

    work page 2025

  6. [6]

    Zhang, Z

    G. Zhang, Z. Xu, Q. Jin, F. Chen, Y. Fang, Y. Liu, J. F. Rousseau, Z. Xu, Z. Lu, C. Weng, et al., Leveraging long context in retrieval augmented language models for medical question answering, npj Digital Medicine 8 (2025) 239

    work page 2025

  7. [7]

    J. Jung, M. Jeong, S. Lim, S. Jung, J. Yun, T. Roh, M. Sung, J. Kang, Breaking failure cascades: Step- aware reinforcement learning for medical multimodal reasoning, arXiv preprint arXiv:2606.31825 (2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [8]

    H. Kim, H. Lee, Y. Cho, J. Park, J. Park, S. Park, Y. T. Chok, S. Baek, D. Lee, J. Kang, Prompting matters: Snippet-aware strategies for biomedical qa with llms in BioASQ 13b, in: CEUR Workshop Proceedings, volume 4038, 2025, pp. 373–382

    work page 2025

  9. [9]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: How language models use long contexts, 2023. URL: https://arxiv.org/abs/2307.03172. arXiv:2307.03172

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    C. Zong, J. Wan, S. Tang, L. Zhang, Evidencemap: Learning evidence analysis to unleash the power of small language models for biomedical question answering, Artificial Intelligence in Medicine 169 (2025) 103246. URL: http://dx.doi.org/10.1016/j.artmed.2025.103246. doi:10.1016/j.artmed. 2025.103246

    work page doi:10.1016/j.artmed.2025.103246 2025

  11. [11]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL: https://arxiv.org/abs/2005.11401.arXiv:2005.11401

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  12. [12]

    J. H. Merker, A. Bondarenko, M. Hagen, A. Viehweger, Mibi at bioasq 2024: Retrieval-augmented generation for answering biomedical questions, in: Conference and Labs of the Evaluation Forum,

    work page 2024

  13. [13]

    URL: https://api.semanticscholar.org/CorpusID:271822482

    work page

  14. [14]

    Overview of BioASQ 2025: The Thirteenth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

    A. Nentidis, G. Katsimpras, A. Krithara, M. Krallinger, M. Rodríguez-Ortega, E. Rodriguez-López, N. Loukachevitch, A. Sakhovskiy, E. Tutubalina, D. Dimitriadis, G. Tsoumakas, G. Giannakoulas, A. Bekiaridou, A. Samaras, G. M. D. Nunzio, N. Ferro, S. Marchesin, M. Martinelli, G. Silvello, G. Paliouras, Overview of bioasq 2025: The thirteenth bioasq challeng...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Z. Zhu, Y. Zhang, X. Zhuang, F. Zhang, Z. Wan, Y. Chen, Q. Long, Y. Zheng, X. Wu, Can we trust AI doctors? a survey of medical hallucination in large language and large vision-language models, in: W. Che, J. Nabende, E. Shutova, M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics: ACL 2025, Association for Computational Lingui...

    work page 2025

  16. [16]

    Asgari, N

    E. Asgari, N. Montaña-Brown, M. Dubois, S. Khalil, J. Balloch, J. A. Yeung, D. Pimenta, A framework to assess clinical safety and hallucination rates of llms for medical text summarisation, NPJ digital medicine 8 (2025) 274

    work page 2025

  17. [17]

    Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models

    M. Jeong, J. Sohn, M. Sung, J. Kang, Improving medical reasoning through retrieval and self- reflection with retrieval-augmented large language models, 2024. URL: https://arxiv.org/abs/2401. 15269.arXiv:2401.15269

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A

    S. Ateia, U. Kruschwitz, Bioragent: A retrieval-augmented generation system for showcasing generative query expansion and domain-specific search for scientific q&a, 2024. URL: https: //arxiv.org/abs/2412.12358.arXiv:2412.12358

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    D. N. Panou, A. C. Dimopoulos, M. Reczko, Farming open llms for biomedical question answering., in: CLEF (Working Notes), 2024, pp. 188–196

    work page 2024

  20. [20]

    T. Roh, W. Jang, J. Jung, J. Kang, Clag: Adaptive memory organization via agent-driven clustering for small language model agents, in: Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 16707–16726

    work page 2026

  21. [21]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, S. Yao, Reflexion: Lan- guage agents with verbal reinforcement learning, 2023. URL: https://arxiv.org/abs/2303.11366. arXiv:2303.11366

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [22]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, 2023. URL: https://arxiv.org/abs/2201.11903. arXiv:2201.11903

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, D. Zhou, Self-consistency improves chain of thought reasoning in language models, 2023. URL: https://arxiv.org/abs/2203. 11171.arXiv:2203.11171

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    X. Tang, A. Zou, Z. Zhang, Z. Li, Y. Zhao, X. Zhang, A. Cohan, M. Gerstein, Medagents: Large language models as collaborators for zero-shot medical reasoning, 2024. URL: https://arxiv.org/ abs/2311.10537.arXiv:2311.10537

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Angulo, V

    J. Angulo, V. Yeste, AQAMS and AQAMS2: Multi agent systems for biomedical question answering, in: CEUR Workshop Proceedings, volume 4038, 2025

    work page 2025

  26. [26]

    BioASQ, Task 14b: Test results of phase b, https://participants-area.bioasq.org/results/14b/phaseB/,

    work page

  27. [28]

    BioASQ, Task 14b guidelines, https://participants-area.bioasq.org/general_information/Task14b/,

    work page

  28. [29]

    Accessed: 2026-05-26

    work page 2026

  29. [30]

    Malakasiotis, I

    P. Malakasiotis, I. Pavlopoulos, I. Androutsopoulos, A. Nentidis, Evaluation Measures for Task B, Deliverable excerpt from D4.1 Version 1.1, BioASQ, 2020. URL: http://www.bioasq.org

    work page 2020

  30. [31]

    Accessed: 2026-05- 28

    OpenAI, Introducing gpt-5, https://openai.com/index/introducing-gpt-5/, 2025. Accessed: 2026-05- 28

    work page 2025

  31. [32]

    Accessed: 2026- 05-28

    OpenAI, Gpt-4o system card, https://openai.com/index/gpt-4o-system-card/, 2024. Accessed: 2026- 05-28

    work page 2024

  32. [33]

    Accessed: 2026-05-28

    Google, Gemini 3: Google’s latest multimodal model family, https://blog.google/ products-and-platforms/products/gemini/gemini-3/, 2025. Accessed: 2026-05-28

    work page 2025

  33. [34]

    J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, W. Chen, What makes good in-context examples for gpt-3?, 2021. URL: https://arxiv.org/abs/2101.06804.arXiv:2101.06804

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  34. [35]

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, Biobert: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics 36 (2020) 1234–1240

    work page 2020

  35. [36]

    Accessed: 2026-05-28

    OpenAI, Gpt-5.2, https://openai.com/gpt-5/, 2025. Accessed: 2026-05-28

    work page 2025

  36. [37]

    Accessed: 2026-05-28

    Anthropic, Claude opus 4.6, https://www.anthropic.com/claude, 2025. Accessed: 2026-05-28

    work page 2025

  37. [38]

    Accessed: 2026-05-28

    Anthropic, Claude sonnet 4.6, https://www.anthropic.com/claude, 2025. Accessed: 2026-05-28

    work page 2025

  38. [39]

    Accessed: 2026-05-28

    OpenAI, Gpt-5.4, https://openai.com/index/introducing-gpt-5-5/, 2026. Accessed: 2026-05-28

    work page 2026

  39. [40]

    Accessed: 2026-05-28

    Google, Gemini 3.1 pro preview, https://blog.google/innovation-and-ai/models-and-research/ gemini-models/gemini-3-1-pro/, 2026. Accessed: 2026-05-28

    work page 2026

  40. [41]

    Accessed: 2026-05-28

    Google, Gemini 3 flash preview, https://docs.cloud.google.com/vertex-ai/generative-ai/docs/ models/gemini/3-flash, 2025. Accessed: 2026-05-28. A. Prompt Examples This appendix provides representative prompt templates used in the list-question agent pipeline. The templates are included for transparency; implementation details such as model-specific wrapper...

    work page 2025

  41. [42]

    For each snippet, assess its relevance to the question (0.0 to 1.0)

    work page

  42. [43]

    Extract key entities, facts, and relationships from the relevant snippets

    work page

  43. [44]

    Provide a concise evidence summary that organizes the extracted information

    work page

  44. [45]

    Determine whether the evidence is sufficient to answer the question

    work page

  45. [46]

    relevant_snippets

    Extract ALL entities/items that belong to the category asked about. Scan every snippet exhaustively — do not miss items. Respond ONLY in valid JSON with this exact format: { "relevant_snippets": [{"idx": 1, "relevance_score": 0.95, "key_facts": ["fact1", "fact2"]}, . . . ], "extracted_entities": ["entity1", "entity2", . . . ], "evidence_summary": "A conci...

    work page

  46. [47]

    GROUNDING: Is every claim in the answer directly supported by at least one snippet?

    work page

  47. [48]

    ACCURACY: Is the answer factually correct according to the snippets?

    work page

  48. [49]

    COMPLETENESS: Does the answer address the question fully?

    work page

  49. [50]

    FORMAT: Is the answer in the correct format for a list question?

    work page

  50. [51]

    CONSISTENCY: Is the answer consistent with the evidence analysis report?

    work page

  51. [52]

    verdict":

    LIST-SPECIFIC: Check completeness of the returned entities, flag any false positives, and remove duplicate entities. CROSS-CHECK: Compare the answer’s entities with the Evidence Analyst’s extracted entities. Flag any entities in the answer that don’t appear in either the snippets or the evidence report. Respond ONLY in valid JSON: { "verdict": "supported"...

    work page

  52. [53]

    supported

    If Verification verdict is "supported" with high confidence (≥0.7): →Use the Reasoning Agent’s answer as-is

    work page

  53. [54]

    needs_revision

    If Verification verdict is "needs_revision" and a suggested_correction exists: →Evaluate whether the correction is better than the original. Use your judgment

    work page

  54. [55]

    unsupported

    If Verification verdict is "unsupported": →Check the suggested_correction. If valid, use it. →Otherwise, re-derive the answer from the snippets yourself

    work page

  55. [56]

    If there’s a mismatch, investigate and resolve

    CROSS-CHECK: Compare entities in the Reasoning answer against the Evidence Analyst’s extracted_entities. If there’s a mismatch, investigate and resolve

    work page

  56. [57]

    ALWAYS verify your final answer is grounded in the original snippets

    work page

  57. [58]

    final_answer

    When in doubt, prefer the answer with higher snippet support. FINAL ANSWER FORMAT: Return the final answer as a JSON array of entity strings. Respond ONLY in valid JSON: { "final_answer": ["entity1", "entity2", . . . ], "decision_reasoning": "Brief explanation of why you made this decision. . . ", "source": "reasoning" or "verification_correction" or "sup...

    work page