BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A

Samy Ateia; Udo Kruschwitz

REVIEW 3 major objections 5 minor 1 cited by

BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BioRAGent demonstrates a transparent, editable-query retrieval-augmented generation pipeline for biomedical questions, with competitive BioASQ 2024 results.

desk verdict A useful, transparent RAG demo whose effectiveness claims are borrowed from a previous challenge paper rather than demonstrated here. read the letter →

arxiv 2412.12358 v1 pith:NSGEHJL7 submitted 2024-12-16 cs.CL cs.AI

Samy Ateia , Udo Kruschwitz This is my paper · ORCID

classification cs.CLcs.AI

keywords retrieval-augmentedgenerationbiomedicalquestionansweringqueryexpansionfew-shotlearningprofessionalsearchBioASQtransparencylargelanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BioRAGent is a web-based retrieval-augmented generation system for biomedical question answering that tries to show how large language models can be integrated into professional search without sacrificing transparency. The paper's central claim is that few-shot LLM query expansion, snippet extraction, and snippet reranking can produce evidence-grounded answers from PubMed literature, and that the resulting system is competitive: the authors report multiple first and second places in the 12th BioASQ challenge. What makes the approach distinctive is that every step is visible to the user: the expanded query is displayed and editable, and answers carry inline citations linking to the underlying PubMed articles. If the claim holds, it would demonstrate a practical middle path between opaque dense vector retrieval and plain keyword search for scientific question answering.

What carries the argument

The central mechanism is the editable expanded query. A fast commercial LLM, prompted with three examples, rewrites a user's natural-language biomedical question into the query language of the underlying search engine, and that generated query is displayed to the user, who can inspect and modify it before the system retrieves documents. Supporting it are two additional few-shot LLM steps: snippet extraction from the top-50 BM25-ranked PubMed abstracts, and snippet reranking against the original question, with the few-shot examples for query expansion selected from a training set by highest F1 score. The two answer formats—a short BioASQ-style paragraph and a sentence-level citation-annotated paragraph—are generated from the reranked snippets. The design choice that carries the argument is that the semantic enrichment happens in visible, editable text (the query string and the snippets) rather than in an opaque vector space, which is what the paper means by transparency.

What would settle it

On a fixed sample of BioASQ questions, run the system with and without LLM query expansion and snippet reranking, keeping the same answer generator and document index; if the stripped version matches or beats the full pipeline on retrieval precision and answer accuracy, the central claim about the generative components is not supported.

Watch

Extended reading notes

Core claim

BioRAGent is a web interface for a retrieval pipeline in which an LLM performs three jobs: expanding the user's biomedical question into a search-engine query, extracting relevant snippets from retrieved abstracts, and reranking those snippets by relevance. The paper's central empirical claim is that this straightforward few-shot approach was competitive in the 12th BioASQ challenge, where the authors report multiple first and second places across tasks, with their strongest results in question answering rather than in document retrieval or snippet extraction. On those latter tasks, systems using dense and hybrid retrieval took the leading spots. The authors present the system as a demonstration that an LLM's semantic knowledge can be injected into professional search by generating the query string itself, making the semantics inspectable and controllable by the user instead of hidden inside embedding vectors.

Load-bearing premise

The demo's value depends on the assumption, not tested inside this paper, that the LLM-generated expanded queries and the LLM-based snippet reranking actually beat plain BM25 keyword retrieval in search and answer quality; the paper points to its earlier challenge participation for that evidence instead of reporting an ablation.

Editorial extensions

If this is right

A searcher can check and correct the LLM's expanded query before retrieval, turning query expansion from a black box into an auditable step.
Answers with per-sentence PubMed citations give every generation a traceable source, which is the transparency professional biomedical search needs.
The whole pipeline is few-shot and requires no fine-tuning, so moving it to a new collection or question domain mainly means swapping the index and the example prompts.
The challenge results suggest the LLM steps add the most value in generating and grounding answers rather than in the initial document-finding stage, since the system was most competitive in question-answering tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same visible-query mechanism could transfer to other specialized-search domains (legal, patent, regulatory) where users need to vet the terminology a machine adds before trusting the results.
The editable expanded query also makes the system a natural platform for studying human trust: one could vary whether users are allowed to edit the query and measure whether answer acceptance changes.
A direct ablation experiment, which the paper does not report, would isolate the gain from LLM query expansion and reranking versus plain BM25 retrieval; such an experiment is the natural next check of the load-bearing assumption.
Pairing the generated-query approach with dense or hybrid retrieval could keep the transparency of the visible query while closing the documented gap in document-retrieval performance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

BioRAGent is a working, open-source RAG demo for biomedical Q&A, and the transparency features—editable expanded queries, inline PubMed citations, visible snippets—are genuinely useful for a professional-search audience. That is the paper's real contribution: a clean, reproducible implementation of a simple LLM-in-the-loop pipeline (3-shot query expansion, BM25 retrieval over a PubMed snapshot, LLM snippet extraction/reranking, answer generation). The code and demo are public, which is real evidence the system exists and does what the architecture describes.

The soft spot is exactly where the stress-test lands: the paper makes performance claims ('competitive results', 'winning multiple first and second places', 'effectively applied') but contains no evaluation data, no ablations, no error analysis. The only support is a pointer to the authors' own BioASQ 2024 working-notes paper, and it is not shown that the deployed BioRAGent pipeline is the same as what produced those numbers. Worse, Section 3 concedes that in document retrieval and snippet extraction—the two components this demo showcases—other dense/hybrid systems took the leading spots. That makes it entirely possible the competitive QA results came from answer generation, not from the query-expansion/reranking mechanism the paper highlights. The few-shot examples are even described as not optimized except for query expansion, so the 'effectively applied' claim is under-powered.

For a demo-track paper, the lack of full evaluation is common, but the abstract and conclusion overstate. The paper should either include a small intrinsic comparison (e.g., expanded query vs. original query on a handful of BioASQ questions) or explicitly reframe itself as a system description without performance claims. As it stands, the system is worth knowing about, but the scientific claim is unverified.

I'd send this to peer review if the venue is a demo track—a working, open system with a clear architecture deserves referee time—but I'd tell the referees to demand the claims be matched to evidence. Not a paper to build a research program on, but a useful resource for retrieval practitioners and for anyone teaching RAG transparency.

Referee Report

3 major / 5 minor

Summary. The paper presents BioRAGent, an interactive web-based retrieval-augmented generation (RAG) system for biomedical question answering. The system uses Gemini 1.5 Flash for few-shot query expansion, snippet extraction, snippet reranking, and answer generation, with Elasticsearch/BM25 retrieval over a 2023 PubMed snapshot. The interface shows the expanded query in an editable box, returns a short paragraph answer and an answer with inline PubMed citations, and provides links to retrieved snippets. The authors state that the system builds on their participation in the BioASQ 2024 challenge, where they report competitive results and multiple first and second places, and they position the demo as showing how few-shot LLM capabilities can be integrated transparently into professional search.

Significance. If the performance claims are substantiated, the paper offers a useful, transparent, and reproducible RAG demonstration for biomedical search: the code is public, the demo is online, and the design makes query expansion visible and user-editable, which is a genuine strength for professional-search settings. The work also connects to two external evaluation venues (BioASQ 2024 and TREC 2024 BioGen), which gives it a credible grounding. However, the paper's central claim that the LLM-based query expansion and snippet reranking are effective is not supported by any in-paper evaluation; the only evidence is a citation to a working-notes paper, and the paper itself concedes that the retrieval and snippet-extraction components did not lead the BioASQ tasks. The significance therefore depends on whether the authors can either supply direct evidence or appropriately scope their claims.

major comments (3)

[Abstract and Section 3] The abstract and Section 3 claim that BioRAGent achieved "competitive results" and won "multiple first and second places" in the 12th BioASQ challenge, but this paper contains no evaluation data, metrics, baselines, or statistical analysis to support that claim. The only support is a reference to the authors' prior CLEF 2024 working-notes paper [1]; that reference is not reproduced here, and the paper does not establish that the pipeline evaluated in [1] matches the deployed BioRAGent system (Gemini 1.5 Flash, Elasticsearch, 2023 PubMed snapshot). To make this claim load-bearing, the authors should either report the relevant BioASQ results directly in the paper or clearly state which parts of [1] apply to the current system.
[Sections 2.1, 2.2, and 3] The central showcased mechanisms are generative query expansion and LLM-based snippet reranking, yet the paper provides no in-paper evidence that these components improve retrieval or answer quality over a plain BM25 baseline. A simple experiment comparing retrieval with and without the expanded query, or answer quality with and without reranking, would substantiate the claim that the LLM components are "effectively applied." As written, Section 3 concedes that in the document-retrieval and snippet-extraction tasks, systems using dense and hybrid retrieval took the leading spots, which makes it plausible that any competitive question-answering performance came from the answer-generation stage rather than from the showcased query-expansion/reranking components. The demo and code show the system works, but they do not show that the LLM-specific retrieval components are beneficial.
[Section 2.2] The snippet-reranking step is described only as prompting the LLM to "rerank the extracted snippets based on their relevance to the original question," but the paper does not specify how many snippets are retained after reranking, whether reranking changes the document set or only the order of snippets, and how the reranked snippets are fed into the answer generator. This level of detail matters for reproducibility, especially because the paper claims the approach is simple and transparent. Please clarify the exact input/output of each stage and state the values of the free parameters (number of few-shot examples, top_k retrieved, few-shot selection metric) that are used in the deployed system.

minor comments (5)

[Section 1] "we presentBioRAGent" is missing a space after "present".
[Section 3] "in the 12th. BioASQ challenge" uses an awkward period after the ordinal; it should read "12th BioASQ challenge."
[Reference [3]] Reference [3] lists the same DOI twice (https://doi.org/10.1145/3490238); the duplicate should be removed.
[Section 2.1] The paper states that few-shot examples for query expansion are "sampled by highest f1 score," but it does not define what f1 score is being computed against, or what gold data is used for this sampling. Please clarify this selection criterion.
[Section 2.3] The two answer formats are described, but the paper does not explain how the model avoids hallucination in the non-citation format where it "doesn't force the model to use the retrieved information." A brief sentence on how the system maintains transparency in that mode would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: BioRAGent's claims rest on an external challenge evaluation rather than on the paper's own definitions or fitted outcomes.

full rationale

The paper is a system/demo description with no formal derivation chain whose conclusions could reduce to its own inputs. Query expansion, snippet extraction, snippet reranking, and answer generation are described procedurally in Sections 2.1-2.3, and no component is defined in terms of the performance claims. The only evaluation evidence is a self-citation: 'The core RAG components of BioRAGent were evaluated through our participation in the BioASQ 2024 challenge [1].' Although reference [1] is by the same authors, it reports results from the externally organized BioASQ/CLEF benchmark, and the paper links to the twelfth challenge winners page; this is independent, externally falsifiable evidence rather than a circular reduction. Section 3 also concedes that 'in the document retrieval and snippets extraction tasks other systems that also used dense and hybrid retrieval techniques took the leading spots,' which weakens the empirical support for the showcased retrieval mechanisms but is a correctness/evidence concern, not a circularity. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely relabeled.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The system introduces no new theoretical constructs or entities. It relies on existing algorithms and models. The free parameters are standard hyperparameters of the pipeline, and the axioms are common assumptions in retrieval and LLM applications.

free parameters (3)

number_of_few_shot_examples = 3
The number of examples used in few-shot prompting for query expansion is chosen by hand (Section 2.1). No ablation is provided to show this is optimal.
top_k_retrieved_documents = 50
The system retrieves the top 50 articles from Elasticsearch (Section 2.2). This is a fixed design choice with no sensitivity analysis.
few_shot_selection_metric = highest f1 score
Few-shot examples for query expansion are sampled by highest f1 score from the BioASQ training set (Section 2.2). This is a tuning choice that may bias the system toward the training distribution.

assumptions (3)

domain assumption BM25 relevance scoring in Elasticsearch is an adequate lexical retrieval baseline for biomedical abstracts.
The system relies on Elasticsearch's default BM25 ranking to retrieve candidate documents (Section 2.2). This is a standard assumption in IR but not evaluated here.
domain assumption The PubMed 2023 snapshot used for the index is representative of the biomedical literature required to answer the questions.
The system indexes a 2023 snapshot of PubMed articles (Section 2.1), and downstream answers depend on this coverage.
domain assumption Large language models such as Gemini 1.5 Flash can generate useful query expansions and relevant snippets from the retrieved documents.
The entire pipeline depends on the LLM's ability to perform query expansion, snippet extraction, and reranking correctly (Sections 2.1, 2.2). This is not proven within the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A." pith.science (2026). https://pith.science/paper/NSGEHJL7

@misc{pith2026241212358,
  author       = {Pith},
  title        = {Pith review of: BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSGEHJL7}},
  note         = {Machine review of arXiv:2412.12358}
}

read the original abstract

We present BioRAGent, an interactive web-based retrieval-augmented generation (RAG) system for biomedical question answering. The system uses large language models (LLMs) for query expansion, snippet extraction, and answer generation while maintaining transparency through citation links to the source documents and displaying generated queries for further editing. Building on our successful participation in the BioASQ 2024 challenge, we demonstrate how few-shot learning with LLMs can be effectively applied for a professional search setting. The system supports both direct short paragraph style responses and responses with inline citations. Our demo is available online, and the source code is publicly accessible through GitHub.

Figures

Figures reproduced from arXiv: 2412.12358 by the authors.

**Figure 1.** Screenshot of part of the BioRAGent interface, showcasing query expansion While dense vector search is an intriguing technique due to the demonstrated state-of-the-art (SOTA) performance in multiple benchmarks, the resulting retrieval rankings based on embedding vectors are not transparent or easily controllable by a search expert. Our approach, on the other hand, makes the encoded semantic knowledge of the LLM us… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
cs.CL 2026-07 conditional novelty 3.0 of 10

A question-type-specific LLM ensemble and multi-agent pipeline achieved competitive results on BioASQ 14b Task B, including first place in the factoid subtask of Batch 4.

Reference graph

Works this paper leans on

11 extracted references · 6 canonical work pages · cited by 1 Pith paper

[1]

In: Faggioli, G., Ferro, N., Galusc \' a kov \' a , P., de Herrera, A.G.S

Ateia, S., Kruschwitz, U.: Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks . In: Faggioli, G., Ferro, N., Galusc \' a kov \' a , P., de Herrera, A.G.S. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024...

work page 2024
[2]

ACM Comput

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of Hallucination in Natural Language Generation . ACM Comput. Surv. 55(12) (2023). doi:10.1145/3571730

doi:10.1145/3571730 2023
[3]

ACM Comput

Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., Yu, S.: Biomedical Question Answering: A Survey of Approaches and Challenges . ACM Comput. Surv. 55(2) (Jan 2022). doi:10.1145/3490238, https://doi.org/10.1145/3490238

doi:10.1145/3490238 2022
[4]

u ttler, H., Lewis, M., Yih, W.t., Rockt \

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.t., Rockt \"a schel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459--9474 (2020)

work page 2020
[5]

Intelligent Systems with Applications 15, 200091 (2022)

MacFarlane, A., Russell-Rose, T., Shokraneh, F.: Search strategy formulation for systematic reviews: Issues, challenges and opportunities. Intelligent Systems with Applications 15, 200091 (2022). doi:https://doi.org/10.1016/j.iswa.2022.200091

work page arXiv 2022
[6]

In: Goeuriot, L., Mulhem, P., Quénot, G., Schwab, D., Soulier, L., Maria Di Nunzio, G., Galuščáková, P., García Seco de Herrera, A., Faggioli, G., Ferro, N

Nentidis, A., Katsimpras, G., Krithara, A., Lima-López, S., Farré-Maduell, E., Krallinger, M., Loukachevitch, N., Davydova, V., Tutubalina, E., Paliouras, G.: Overview of BioASQ 2024: The twelfth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering . In: Goeuriot, L., Mulhem, P., Quénot, G., Schwab, D., Soulier, L., Maria Di...

work page 2024
[7]

arXiv preprint arXiv:2403.05530 (2024)

Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al.: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . arXiv preprint arXiv:2403.05530 (2024)

arXiv 2024
[8]

In: Findings of the Association for Computational Linguistics: EMNLP 2021

Shuster, K., Poff, S., Chen, M., Kiela, D., Weston, J.: Retrieval Augmentation Reduces Hallucination in Conversation . In: Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 3784--3803 (2021)

work page 2021

Show all 11 references

[9]

Wang, S., Scells, H., Koopman, B., Zuccon, G.: Can ChatGPT write a good boolean query for systematic review literature search? In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1426--1436 (2023)

2023
[10]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

In: Faggioli, G., Ferro, N., Galusc \' a kov \' a , P., de Herrera, A.G.S

Ateia, S., Kruschwitz, U.: Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks . In: Faggioli, G., Ferro, N., Galusc \' a kov \' a , P., de Herrera, A.G.S. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024...

work page 2024

[2] [2]

ACM Comput

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of Hallucination in Natural Language Generation . ACM Comput. Surv. 55(12) (2023). doi:10.1145/3571730

doi:10.1145/3571730 2023

[3] [3]

ACM Comput

Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., Yu, S.: Biomedical Question Answering: A Survey of Approaches and Challenges . ACM Comput. Surv. 55(2) (Jan 2022). doi:10.1145/3490238, https://doi.org/10.1145/3490238

doi:10.1145/3490238 2022

[4] [4]

u ttler, H., Lewis, M., Yih, W.t., Rockt \

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.t., Rockt \"a schel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459--9474 (2020)

work page 2020

[5] [5]

Intelligent Systems with Applications 15, 200091 (2022)

MacFarlane, A., Russell-Rose, T., Shokraneh, F.: Search strategy formulation for systematic reviews: Issues, challenges and opportunities. Intelligent Systems with Applications 15, 200091 (2022). doi:https://doi.org/10.1016/j.iswa.2022.200091

work page arXiv 2022

[6] [6]

In: Goeuriot, L., Mulhem, P., Quénot, G., Schwab, D., Soulier, L., Maria Di Nunzio, G., Galuščáková, P., García Seco de Herrera, A., Faggioli, G., Ferro, N

Nentidis, A., Katsimpras, G., Krithara, A., Lima-López, S., Farré-Maduell, E., Krallinger, M., Loukachevitch, N., Davydova, V., Tutubalina, E., Paliouras, G.: Overview of BioASQ 2024: The twelfth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering . In: Goeuriot, L., Mulhem, P., Quénot, G., Schwab, D., Soulier, L., Maria Di...

work page 2024

[7] [7]

arXiv preprint arXiv:2403.05530 (2024)

Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al.: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . arXiv preprint arXiv:2403.05530 (2024)

arXiv 2024

[8] [8]

In: Findings of the Association for Computational Linguistics: EMNLP 2021

Shuster, K., Poff, S., Chen, M., Kiela, D., Weston, J.: Retrieval Augmentation Reduces Hallucination in Conversation . In: Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 3784--3803 (2021)

work page 2021