{"id":"76b6c51e-fd46-4c2f-b2cb-3f66fcb5f1cf","arxiv_id":"2412.12358","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BioRAGent applies few-shot LLM query expansion, snippet extraction, and citation-grounded answer generation to biomedical question answering over PubMed in a public demo.","lead":"BioRAGent is a web-based biomedical question answering system that uses large language models to expand search queries and generate cited answers from PubMed. The demo shows how generative query expansion can make professional literature search more transparent and controllable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM query expansion and reranking are the showcased mechanisms, but the paper gives no in-paper evidence that they improve retrieval or answers; the only cited support is a prior challenge paper whose evaluation is not reproduced, and Section 3 concedes retrieval/snippet tasks were not leading.","rationale":"The paper is best read as a demo-system paper: its public code and online demo are real evidence that the architecture runs as described and that expanded queries and citations are displayed. The load-bearing issue is not whether the demo exists but whether the central claim of showing LLM capabilities 'effectively applied' to professional search is supported. The reader's weakest assumption names exactly this: the paper does not test whether the LLM query expansion and reranking improve retrieval or answer quality. I agree with that assessment and found no separate flaw that would change the verdict. Two details strengthen the concern: Section 3 explicitly concedes that document-retrieval and snippet-extraction tasks, the components tied to query expansion and reranking, were not the leading spots in BioASQ; and the cited evaluation [1] is a different working-notes paper that may not correspond to the deployed Gemini-1.5-Flash pipeline. This does not make the paper's claims false, but it makes them unverified. The proposed concrete test is a straightforward ablation using the public code; it would settle whether the showcased component is load-bearing. Therefore the appropriate verdict remains CONDITIONAL, as the reader already concluded.","tokens_in":3581,"tokens_out":5299,"duration_ms":49380,"concrete_test":"Run a head-to-head evaluation on a held-out sample of 100 questions from the BioASQ 2024 Phase B dataset used in [1]. Using the public BioRAGent code, compare: (a) the full pipeline, (b) the same pipeline with query expansion disabled so the original question is used as the Elasticsearch query, and (c) the same pipeline with both query expansion and snippet reranking disabled, leaving plain BM25 top-k retrieval. Report recall@50 for retrieved documents/snippets and answer relevance using official BioASQ metrics or expert labels. If configuration (a) is not significantly better than (b) and (c), the central demonstration of generative query expansion and reranking for professional search is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4 claim that BioRAGent demonstrates how SOTA LLM capabilities can be integrated into professional search while maintaining transparency. For that demonstration to support the paper's stronger wording that few-shot LLM approaches are 'effectively applied', the LLM-generated expanded queries and LLM snippet reranking introduced in Sections 2.1 and 2.2 must actually help. The only evidence offered is Section 3's reference to the BioASQ 2024 participation [1] and the challenge-winners link; no metrics, baselines, ablations, or error analysis appear in this paper. That reference is not self-contained: it is a CLEF 2024 working-notes paper that may not evaluate the deployed BioRAGent/Gemini-1.5-Flash pipeline, so the connection between the cited numbers and this demo is assumed rather than shown. Section 3 also states that in the document-retrieval and snippet-extraction tasks, which are exactly the components the demo showcases, other systems using dense and hybrid retrieval took the leading spots. This makes it plausible that any competitive QA performance came from answer generation rather than from the query-expansion/reranking mechanism. The public code and online demo substantiate that the system exists and is transparent, but they do not substantiate that the showcased LLM retrieval components outperform a plain BM25 baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents BioRAGent, an interactive web-based retrieval-augmented generation (RAG) system for biomedical question answering. The system uses Gemini 1.5 Flash for few-shot query expansion, snippet extraction, snippet reranking, and answer generation, with Elasticsearch/BM25 retrieval over a 2023 PubMed snapshot. The interface shows the expanded query in an editable box, returns a short paragraph answer and an answer with inline PubMed citations, and provides links to retrieved snippets. The authors state that the system builds on their participation in the BioASQ 2024 challenge, where they report competitive results and multiple first and second places, and they position the demo as showing how few-shot LLM capabilities can be integrated transparently into professional search.","tokens_in":3843,"tokens_out":2675,"duration_ms":24861,"significance":"If the performance claims are substantiated, the paper offers a useful, transparent, and reproducible RAG demonstration for biomedical search: the code is public, the demo is online, and the design makes query expansion visible and user-editable, which is a genuine strength for professional-search settings. The work also connects to two external evaluation venues (BioASQ 2024 and TREC 2024 BioGen), which gives it a credible grounding. However, the paper's central claim that the LLM-based query expansion and snippet reranking are effective is not supported by any in-paper evaluation; the only evidence is a citation to a working-notes paper, and the paper itself concedes that the retrieval and snippet-extraction components did not lead the BioASQ tasks. The significance therefore depends on whether the authors can either supply direct evidence or appropriately scope their claims.","major_comments":[{"comment":"The abstract and Section 3 claim that BioRAGent achieved \"competitive results\" and won \"multiple first and second places\" in the 12th BioASQ challenge, but this paper contains no evaluation data, metrics, baselines, or statistical analysis to support that claim. The only support is a reference to the authors' prior CLEF 2024 working-notes paper [1]; that reference is not reproduced here, and the paper does not establish that the pipeline evaluated in [1] matches the deployed BioRAGent system (Gemini 1.5 Flash, Elasticsearch, 2023 PubMed snapshot). To make this claim load-bearing, the authors should either report the relevant BioASQ results directly in the paper or clearly state which parts of [1] apply to the current system.","section":"Abstract and Section 3"},{"comment":"The central showcased mechanisms are generative query expansion and LLM-based snippet reranking, yet the paper provides no in-paper evidence that these components improve retrieval or answer quality over a plain BM25 baseline. A simple experiment comparing retrieval with and without the expanded query, or answer quality with and without reranking, would substantiate the claim that the LLM components are \"effectively applied.\" As written, Section 3 concedes that in the document-retrieval and snippet-extraction tasks, systems using dense and hybrid retrieval took the leading spots, which makes it plausible that any competitive question-answering performance came from the answer-generation stage rather than from the showcased query-expansion/reranking components. The demo and code show the system works, but they do not show that the LLM-specific retrieval components are beneficial.","section":"Sections 2.1, 2.2, and 3"},{"comment":"The snippet-reranking step is described only as prompting the LLM to \"rerank the extracted snippets based on their relevance to the original question,\" but the paper does not specify how many snippets are retained after reranking, whether reranking changes the document set or only the order of snippets, and how the reranked snippets are fed into the answer generator. This level of detail matters for reproducibility, especially because the paper claims the approach is simple and transparent. Please clarify the exact input/output of each stage and state the values of the free parameters (number of few-shot examples, top_k retrieved, few-shot selection metric) that are used in the deployed system.","section":"Section 2.2"}],"minor_comments":[{"comment":"\"we presentBioRAGent\" is missing a space after \"present\".","section":"Section 1"},{"comment":"\"in the 12th. BioASQ challenge\" uses an awkward period after the ordinal; it should read \"12th BioASQ challenge.\"","section":"Section 3"},{"comment":"Reference [3] lists the same DOI twice (https://doi.org/10.1145/3490238); the duplicate should be removed.","section":"Reference [3]"},{"comment":"The paper states that few-shot examples for query expansion are \"sampled by highest f1 score,\" but it does not define what f1 score is being computed against, or what gold data is used for this sampling. Please clarify this selection criterion.","section":"Section 2.1"},{"comment":"The two answer formats are described, but the paper does not explain how the model avoids hallucination in the non-citation format where it \"doesn't force the model to use the retrieved information.\" A brief sentence on how the system maintains transparency in that mode would be helpful.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a demo-track submission, so the bar for in-paper evaluation might be lower than for a full research paper. However, the authors make explicit empirical claims ('competitive results', 'winning multiple first and second places') in the abstract and Section 3, and the central design choices (query expansion, snippet reranking) are presented as the system's contribution. Given that the only support is a self-cited working-notes paper and that Section 3 concedes the retrieval/snippet tasks were not leading, I consider the missing evidence load-bearing rather than stylistic. The authors could address this by adding a small evaluation section (e.g., BM25 baseline vs. expanded query, or reranking on/off) or by significantly tempering the claims to a system-description paper. I would not recommend rejection because the demo and code are real and the transparency features are valuable; I would accept a revision that either provides the evidence or properly scopes the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BioRAGent is a working, open-source RAG demo for biomedical Q&A, and the transparency features—editable expanded queries, inline PubMed citations, visible snippets—are genuinely useful for a professional-search audience. That is the paper's real contribution: a clean, reproducible implementation of a simple LLM-in-the-loop pipeline (3-shot query expansion, BM25 retrieval over a PubMed snapshot, LLM snippet extraction/reranking, answer generation). The code and demo are public, which is real evidence the system exists and does what the architecture describes.\n\nThe soft spot is exactly where the stress-test lands: the paper makes performance claims ('competitive results', 'winning multiple first and second places', 'effectively applied') but contains no evaluation data, no ablations, no error analysis. The only support is a pointer to the authors' own BioASQ 2024 working-notes paper, and it is not shown that the deployed BioRAGent pipeline is the same as what produced those numbers. Worse, Section 3 concedes that in document retrieval and snippet extraction—the two components this demo showcases—other dense/hybrid systems took the leading spots. That makes it entirely possible the competitive QA results came from answer generation, not from the query-expansion/reranking mechanism the paper highlights. The few-shot examples are even described as not optimized except for query expansion, so the 'effectively applied' claim is under-powered.\n\nFor a demo-track paper, the lack of full evaluation is common, but the abstract and conclusion overstate. The paper should either include a small intrinsic comparison (e.g., expanded query vs. original query on a handful of BioASQ questions) or explicitly reframe itself as a system description without performance claims. As it stands, the system is worth knowing about, but the scientific claim is unverified.\n\nI'd send this to peer review if the venue is a demo track—a working, open system with a clear architecture deserves referee time—but I'd tell the referees to demand the claims be matched to evidence. Not a paper to build a research program on, but a useful resource for retrieval practitioners and for anyone teaching RAG transparency.","headline":"A useful, transparent RAG demo whose effectiveness claims are borrowed from a previous challenge paper rather than demonstrated here.","tokens_in":4388,"tokens_out":2321,"would_cite":false,"duration_ms":21067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BioRAGent demonstrates a transparent, editable-query retrieval-augmented generation pipeline for biomedical questions, with competitive BioASQ 2024 results.","keywords":["retrieval-augmented generation","biomedical question answering","query expansion","few-shot learning","professional search","BioASQ","transparency","large language models"],"falsifier":"On a fixed sample of BioASQ questions, run the system with and without LLM query expansion and snippet reranking, keeping the same answer generator and document index; if the stripped version matches or beats the full pipeline on retrieval precision and answer accuracy, the central claim about the generative components is not supported.","tokens_in":3366,"feed_emoji":"🧬","tokens_out":7239,"duration_ms":64079,"temperature":0.7,"pith_summary":"BioRAGent is a web-based retrieval-augmented generation system for biomedical question answering that tries to show how large language models can be integrated into professional search without sacrificing transparency. The paper's central claim is that few-shot LLM query expansion, snippet extraction, and snippet reranking can produce evidence-grounded answers from PubMed literature, and that the resulting system is competitive: the authors report multiple first and second places in the 12th BioASQ challenge. What makes the approach distinctive is that every step is visible to the user: the expanded query is displayed and editable, and answers carry inline citations linking to the underlying PubMed articles. If the claim holds, it would demonstrate a practical middle path between opaque dense vector retrieval and plain keyword search for scientific question answering.","feed_headline":"BioRAGent shows LLM-built search queries you can see and edit","feed_subtitle":"A transparent retrieval-augmented generation system grounds biomedical answers in PubMed with visible query strings and inline citations.","key_machinery":"The central mechanism is the editable expanded query. A fast commercial LLM, prompted with three examples, rewrites a user's natural-language biomedical question into the query language of the underlying search engine, and that generated query is displayed to the user, who can inspect and modify it before the system retrieves documents. Supporting it are two additional few-shot LLM steps: snippet extraction from the top-50 BM25-ranked PubMed abstracts, and snippet reranking against the original question, with the few-shot examples for query expansion selected from a training set by highest F1 score. The two answer formats—a short BioASQ-style paragraph and a sentence-level citation-annotated paragraph—are generated from the reranked snippets. The design choice that carries the argument is that the semantic enrichment happens in visible, editable text (the query string and the snippets) rather than in an opaque vector space, which is what the paper means by transparency.","core_discovery":"BioRAGent is a web interface for a retrieval pipeline in which an LLM performs three jobs: expanding the user's biomedical question into a search-engine query, extracting relevant snippets from retrieved abstracts, and reranking those snippets by relevance. The paper's central empirical claim is that this straightforward few-shot approach was competitive in the 12th BioASQ challenge, where the authors report multiple first and second places across tasks, with their strongest results in question answering rather than in document retrieval or snippet extraction. On those latter tasks, systems using dense and hybrid retrieval took the leading spots. The authors present the system as a demonstration that an LLM's semantic knowledge can be injected into professional search by generating the query string itself, making the semantics inspectable and controllable by the user instead of hidden inside embedding vectors.","pith_inferences":["The same visible-query mechanism could transfer to other specialized-search domains (legal, patent, regulatory) where users need to vet the terminology a machine adds before trusting the results.","The editable expanded query also makes the system a natural platform for studying human trust: one could vary whether users are allowed to edit the query and measure whether answer acceptance changes.","A direct ablation experiment, which the paper does not report, would isolate the gain from LLM query expansion and reranking versus plain BM25 retrieval; such an experiment is the natural next check of the load-bearing assumption.","Pairing the generated-query approach with dense or hybrid retrieval could keep the transparency of the visible query while closing the documented gap in document-retrieval performance."],"forward_implications":["A searcher can check and correct the LLM's expanded query before retrieval, turning query expansion from a black box into an auditable step.","Answers with per-sentence PubMed citations give every generation a traceable source, which is the transparency professional biomedical search needs.","The whole pipeline is few-shot and requires no fine-tuning, so moving it to a new collection or question domain mainly means swapping the index and the example prompts.","The challenge results suggest the LLM steps add the most value in generating and grounding answers rather than in the initial document-finding stage, since the system was most competitive in question-answering tasks."],"supporting_citations":[{"why":"The authors' earlier BioASQ 2024 participation paper, cited as the source of the system's competitive results and few-shot methodology.","marker":"[1]"},{"why":"The foundational retrieval-augmented generation paper that supplies the RAG paradigm grounding answers in retrieved documents.","marker":"[4]"},{"why":"The BioASQ 2024 overview that defines the challenge tasks and evaluation the paper uses as evidence.","marker":"[6]"},{"why":"Documents the LLM used as the system's backend generation and processing model.","marker":"[7]"},{"why":"Provides the evidence that retrieval augmentation reduces hallucination, motivating the citation-grounded answer format.","marker":"[8]"},{"why":"Prior work on LLM-generated boolean search queries, the line the system's query expansion extends.","marker":"[9]"}],"fun_headline_variants":["BioRAGent: edit the AI's search query","BioRAGent: LLM queries you can see and tweak","BioRAGent: transparent RAG with editable queries","BioRAGent: competitive BioASQ with editable search queries","BioRAGent: open the RAG black box"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demo's value depends on the assumption, not tested inside this paper, that the LLM-generated expanded queries and the LLM-based snippet reranking actually beat plain BM25 keyword retrieval in search and answer quality; the paper points to its earlier challenge participation for that evidence instead of reporting an ablation.","fun_headline_variants_meta":{"raw":{"variants":["BioRAGent: edit the AI's search query","BioRAGent: LLM queries you can see and tweak","BioRAGent: transparent RAG with editable queries","BioRAGent: competitive BioASQ with editable search queries","BioRAGent: open the RAG black box"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001425,"raw_usage":{"total_tokens":5688,"prompt_tokens":819,"completion_tokens":4869,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":4783}},"tokens_in":435,"tokens_out":4869,"duration_ms":32533,"temperature":1.0,"reasoning_tokens":4783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:08:13.013474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed sample of BioASQ questions, run the system with and without LLM query expansion and snippet reranking, keeping the same answer generator and document index; if the stripped version matches or beats the full pipeline on retrieval precision and answer accuracy, the central claim about the generative components is not supported.","supporting_citations":[{"cited_title":"In: Faggioli, G., Ferro, N., Galusc \\' a kov \\' a , P., de Herrera, A.G.S","cited_arxiv_id":null,"evidence_quote":"The authors' earlier BioASQ 2024 participation paper, cited as the source of the system's competitive results and few-shot methodology."},{"cited_title":"u ttler, H., Lewis, M., Yih, W.t., Rockt \\","cited_arxiv_id":null,"evidence_quote":"The foundational retrieval-augmented generation paper that supplies the RAG paradigm grounding answers in retrieved documents."},{"cited_title":"In: Goeuriot, L., Mulhem, P., Quénot, G., Schwab, D., Soulier, L., Maria Di Nunzio, G., Galuščáková, P., García Seco de Herrera, A., Faggioli, G., Ferro, N","cited_arxiv_id":null,"evidence_quote":"The BioASQ 2024 overview that defines the challenge tasks and evaluation the paper uses as evidence."},{"cited_title":"In: Findings of the Association for Computational Linguistics: EMNLP 2021","cited_arxiv_id":null,"evidence_quote":"Provides the evidence that retrieval augmentation reduces hallucination, motivating the citation-grounded answer format."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on LLM-generated boolean search queries, the line the system's query expansion extends."}],"review_version":1}