{"id":"b71d11c8-bf71-4691-8698-5cec8086fb11","arxiv_id":"2607.15283","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A BioASQ 13B system that adapts retrieval, reranking, and evidence clustering by question type achieved top Phase A retrieval scores, yet its own tables show non-adaptive baselines frequently performing better.","lead":"This paper describes a biomedical question-answering system that changes its search and evidence strategies based on question type, evaluated on the BioASQ 13B challenge. It reports top rankings in document retrieval, but its own comparisons show the adaptive pipeline often scores below simpler non-adaptive baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mainpipeline underperforms fixed top-10/top-20 baselines on the primary evidence-relevance metric, contradicting the abstract's central claim.","rationale":"We agree with the reader's REJECT verdict, but the most load-bearing problem is not the confounding between baselines—it is the direct contradiction between the abstract's claim of improvement and the paper's own quantitative results. In Phase A document retrieval (Table 2), the adaptive Mainpipeline has MAP 0.4796, 0.5423, 0.4513, 0.4051 for the four batches; each is strictly lower than the corresponding Baseline_Top10 value (0.5627, 0.6768, 0.5757, 0.4847) and Baseline_Top20 (same values). So the adaptive component consistently degrades the primary evidence-relevance metric. For List questions (Table 6), Mainpipeline F-measures 0.1698, 0.2829, 0.3195, 0.2703 are below both baselines in all four batches. For Yes/No (Table 4), Mainpipeline is never better than Baseline_Top10; it ties in Batch 1 and is worse in Batches 2–4. Only in Factoid MRR does Mainpipeline sometimes top one baseline, but it does not beat both in any batch (e.g., Batch 2: 0.4012 vs 0.4506 and 0.5185). The paper's own Section 4.4 states that 'Baseline_Top10 and Baseline_Top20 approaches were generally the strongest variants within our suite of methods, especially for List questions.' This admission, together with the tables, falsifies the central claim that adaptive retrieval strategies 'improve evidence relevance and answer quality across multiple question types.' The first-place BioASQ ranks in Phase A are obtained by the best of the five submission variants, not by the adaptive Mainpipeline; indeed, the baselines outperformed it on document MAP. Thus the claimed causal benefit of adaptation is unsupported and contradicted. The reader's concern about confounded comparisons is valid—the baselines differ in document count, LLM filtering, KG use, and selection—but that concern is secondary; even if the comparison were clean, the data show the opposite of the claimed improvement. A controlled ablation that isolates only the adaptation decision is the appropriate test, as proposed. We therefore keep the reader's REJECT verdict unchanged.","tokens_in":11526,"tokens_out":7129,"duration_ms":61620,"concrete_test":"Perform an ablation on BioASQ 13B batches: for each question, take the same reranked document pool, apply identical LLM filtering and (for List) KG augmentation, and then vary only the selection rule—fixed top-10 vs question-type-aware cluster selection. If the adaptive selection does not significantly improve Phase A MAP and snippet F-measure over the fixed top-10 under a paired Wilcoxon test (p<0.05) or at least shows a positive median difference across all four batches, the central claim is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim vs results: The abstract claims adaptive retrieval improves evidence relevance and answer quality. Table 2 shows Phase A document MAP: Mainpipeline 0.4796/0.5423/0.4513/0.4051 vs Baseline_Top10 0.5627/0.6768/0.5757/0.4847. In all four batches the adaptive pipeline is worse. Table 6 List F-measure: Mainpipeline 0.1698/0.2829/0.3195/0.2703 vs Baseline_Top10 0.3038/0.3758/0.4238/0.3307, again worse everywhere. Table 4 Yes/No Macro-F1: Mainpipeline is never better than both baselines. The paper's own text (Section 4.4) concedes: 'Baseline_Top10 and Baseline_Top20 approaches were generally the strongest variants within our suite of methods, especially for List questions.' Thus the headline claim is contradicted by the reported numbers. The confounded comparison matters, but the direction of the effect is already negative, so the adaptive component provides no demonstrable benefit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an adaptive retrieval framework for biomedical QA in the BioASQ 13B challenge. The system uses LLM-based keyword extraction, multi-strategy document retrieval, two-stage reranking, LLM filtering, knowledge graph augmentation, and question-type-specific evidence aggregation. The authors claim that adaptive retrieval strategies improve evidence relevance and answer quality across question types. They evaluate on the four official BioASQ 13B batches, reporting first-place competition ranks in Phase A document and snippet retrieval and various Phase A+ scores for yes/no, factoid, and list questions.","tokens_in":11820,"tokens_out":4729,"duration_ms":45868,"significance":"If the causal claim were supported, the paper would offer a useful design principle: tailoring retrieval and aggregation to question type. The evaluation on a public benchmark with official scores is a strength, and the system is comprehensively described. However, the reported results contradict the abstract's central claim: the adaptive Mainpipeline underperforms the fixed Top10/Top20 baselines on the primary Phase A MAP and List F-measure in every batch, and Section 4.4 concedes that the baselines were generally the strongest variants. The comparison is confounded by differences in document count, LLM filtering, clustering, and KG use, so no clean evidence isolates the benefit of adaptivity. The competition ranking 'first place' refers to the best among all submitted runs, not to the superiority of the adaptive framework over its own baselines. The central claim is therefore not demonstrated.","major_comments":[{"comment":"The abstract claims adaptive retrieval improves evidence relevance and answer quality, but Table 2 shows Mainpipeline Phase A MAP is below Baseline_Top10 in every batch (e.g., Batch 1: 0.4796 vs 0.5627; Batch 3: 0.4513 vs 0.5757). Table 6 shows List F-measure is lower for Mainpipeline than Baseline_Top10 in every batch (e.g., Batch 1: 0.1698 vs 0.3038; Batch 4: 0.2703 vs 0.3307). Section 4.4 explicitly states 'Baseline_Top10 and Baseline_Top20 approaches were generally the strongest variants within our suite of methods.' Therefore the paper's own results directly contradict the central claim, which must be revised or supported by a proper ablation.","section":"Section 4.4, Tables 2 and 6"},{"comment":"The comparison between Mainpipeline and Baseline_Top10/Top20 does not isolate question-type adaptation. The baselines differ in the number of documents supplied to the LLM (fixed 10 or 20 vs the adaptive selection), whether LLM filtering is applied, whether document clustering is used, and whether KG augmentation is included. Any performance difference could be attributed to these factors rather than adaptivity. An ablation that holds the document subset fixed and varies only the question-type-dependent selection/aggregation strategy is required to support the claim.","section":"Section 4.2"},{"comment":"The KG augmentation, introduced specifically for List questions, does not improve over Mainpipeline in List F-measure in Batches 1, 2, and 4 (e.g., Batch 1: 0.1645 vs 0.1698; Batch 4: 0.2533 vs 0.2703). The only improvement is in Batch 3 (0.3642 vs 0.3195). This further weakens the claim that the added components contribute to answer quality.","section":"Section 3.5, Table 6"},{"comment":"The 'Our Rank' column reports the best rank among all submitted runs (including Naive and Baseline variants) in the competition, not the rank of Mainpipeline relative to the paper's own baselines. For example, Table 4 Batch 1 shows Naive achieving a perfect 1.0000 Macro-F1 while Mainpipeline scores 0.9244. The statement that 'our system delivered the strongest performance among all participating teams' conflates the best of five variants with the adaptive framework itself, providing no evidence for the adaptive component.","section":"Tables 2–6, 'Our Rank' column"}],"minor_comments":[{"comment":"Section 3.3 describes nine retrieval strategies, while Section 4.1 states that six are used in the reported experiments. Clarify which strategies are used and whether the omitted ones affect the results.","section":"Section 3.3 vs Section 4.1"},{"comment":"In the List question evaluation paragraph, 'truth positives' should be 'true positives'. Equation (7) is typeset incorrectly and the definition of ROUGE-SU4 is incomplete. Also, the paper does not report standard deviations or significance tests, which is a limitation given the small number of batches.","section":"Section 4.3"},{"comment":"Minor typographical issue: 'Y es/No' contains an errant space. Also, the notation for the equations in Section 4.3 (e.g., Equation (2)) uses inconsistent spacing and summation symbols.","section":"Section 3.6"}],"recommendation":"reject","confidential_remarks":"This is a system-description paper for a BioASQ competition. The main obstacle is not the absence of novelty but the mismatch between the abstract's causal claim and the paper's own experimental results. The adaptive pipeline is consistently worse than the non-adaptive baselines on key metrics, and the comparison cannot be used to attribute differences to adaptivity. Rejection is recommended because this load-bearing issue cannot be fixed with a modest revision; the paper would need either a substantial re-framing as a system report or a new ablation study that isolates the effect of question-type adaptation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I just read the Adaptive Retrieval paper. If you only remember one thing: it's a useful description of a competitive BioASQ system, but its headline claim—that adaptive retrieval improves evidence relevance and answer quality—is not supported by the numbers in the paper. In fact, the adaptive Mainpipeline usually scores worse than the fixed Top10/Top20 baselines.\n\nWhat's genuinely new and good: the system's Phase A performance is impressive—rank 1 in document retrieval and snippet extraction across all four batches. The pipeline is described thoroughly enough to be replicated in broad strokes, and the speed optimization details are practical. The external BioASQ evaluation gives the system-level results credibility.\n\nThe central problem is that the adaptive component shows no demonstrable benefit. Table 2: Mainpipeline MAP is lower than Baseline_Top10 in all four batches. Table 6: List F-measure is lower in three of four. The paper itself concedes in Section 4.4 that the fixed baselines were generally the strongest variants. So the abstract's claim is simply wrong. The comparison is also confounded—Mainpipeline differs in document count, LLM filtering, KG use, and cluster selection—but even setting that aside, the direction is negative. No significance tests or error bars either. Reproducibility is limited by proprietary components (iSearch, Insilicom KG) and unreleased prompts.\n\nWho's this for? Teams building biomedical QA systems; it's a useful engineering report. As a scientific validation of adaptivity, it fails. But it deserves a serious referee because the system results are real and the paper contains enough detail to engage with. My recommendation: send it to peer review, but the authors need to either substantially reanalyze their data to find any net-positive effect of adaptivity, or reframe the paper as a system description with the causal claims removed. As is, I'd reject, not because the system is bad, but because the claims don't match the evidence.","headline":"A strong BioASQ system report whose central causal claim about adaptive retrieval is undercut by its own tables.","tokens_in":12314,"tokens_out":2158,"would_cite":false,"duration_ms":22729,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Biomedical question answering improves when retrieval adapts to the question type, using distinct evidence strategies for yes/no, factoid, list, and summary questions.","keywords":["biomedical question answering","adaptive retrieval","question-type-aware","BioASQ","retrieval-augmented generation","knowledge graph augmentation","document reranking","evidence clustering"],"falsifier":"Take the exact same adaptive pipeline and run it with adaptivity disabled—same number of documents, same LLM filtering, same KG augmentation, but one uniform strategy for all question types. If document/snippet scores stay the same, the adaptive routing by itself adds nothing.","tokens_in":11392,"feed_emoji":"🧬","tokens_out":4898,"duration_ms":47018,"temperature":0.7,"pith_summary":"The paper tries to establish that a single, one-size-fits-all retrieval pipeline is not the best way to support biomedical question answering. Instead, it argues, the system should first detect what kind of question is being asked and then tailor how it searches for, filters, clusters, and summarizes evidence. To test this, the authors built a multi-stage system that extracts keywords from both the question and an initial answer, retrieves documents through several complementary search strategies, reranks and filters them, augments the pool with knowledge-graph connections for list questions, and adapts the answering step to the question type. On the BioASQ 13B benchmark, they report first-place results in every batch for both document retrieval and snippet extraction, and competitive results in answer generation. The reason a reader should care is that the claim is actionable: if true, question-type-aware retrieval is a simple design principle that could be lifted into other retrieval-augmented systems.","feed_headline":"Adaptive retrieval tops all four BioASQ retrieval batches","feed_subtitle":"Matching search to yes/no, factoid, list, and summary questions beat fixed strategies in document and snippet ranking.","key_machinery":"The load-bearing mechanism is question-type-conditioned evidence selection. A routing step uses the official question-type label (yes/no, factoid, list, summary) to choose different document filters, clusterings, and answer-generation procedures. For example, list questions retain both low and high LLM relevance scores to preserve recall, while other types keep only top-scoring documents; list questions also add knowledge-graph-derived documents. This routing is what the paper credits for its balanced performance across heterogeneous biomedical questions.","core_discovery":"On its own terms, the paper's central discovery is that aligning evidence acquisition with the information need of each question type improves both retrieval and answer quality. The system routes questions into four tracks: yes/no questions get precise evidence and two-sided argument assessment; factoid questions get entity-focused evidence and clustering of candidate answers; list questions get entity-oriented retrieval, knowledge-graph augmentation, and multi-step synonym-aware entity merging; summary questions get broad evidence collection and hierarchical synthesis. The paper reports that this adaptive pipeline, together with multi-strategy keyword retrieval and staged reranking/filterin","pith_inferences":["A natural next experiment, which the paper does not run, is an ablation that disables only the adaptivity while holding document count, LLM filtering, and KG augmentation constant; without that ablation, the reported gains cannot be isolated to the question-type routing.","The paper's own tables show its fixed top-10 and top-20 baselines often beating the full adaptive pipeline on answer F1/MRR; this suggests adaptivity is not uniformly better, and the true value may be in retrieval ranking rather than answer synthesis.","The same 'routing by information need' idea could be tested outside biomedicine, e.g., legal or scientific QA where questions divide into lookup, comparison, and summary types."],"forward_implications":["If the claim holds, any biomedical QA system with typed questions can improve by branching its retrieval and evidence aggregation logic at the question-type level.","Document and snippet ranking can benefit from an answer-driven loop: keywords extracted from an LLM's draft answer expand the search queries and widen recall.","List questions are the hardest case for a unified pipeline; explicit entity-oriented retrieval plus knowledge-graph augmentation is a workable remedy that boosts recall.","Two-stage reranking (a fast reranker on a broad pool, then a stronger reranker on the top subset) lets a system scale to large biomedical corpora within time limits."],"fun_headline_variants":["Adaptive retrieval outperforms fixed pipelines on BioASQ","Matching search to question type lifts biomedical QA","Question-aware retrieval beats one-size-fits-all in bioQA","Tailored retrieval strategies win on BioASQ benchmark","Adaptive retrieval beats uniform pipeline across question types"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper assumes that any performance difference between its full adaptive pipeline and the simpler top-K baselines is caused by question-type adaptation, even though those baselines also differ in document count, LLM filtering, and knowledge-graph use.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive retrieval outperforms fixed pipelines on BioASQ","Matching search to question type lifts biomedical QA","Question-aware retrieval beats one-size-fits-all in bioQA","Tailored retrieval strategies win on BioASQ benchmark","Adaptive retrieval beats uniform pipeline across question types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1264,"prompt_tokens":695,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":506}},"tokens_in":439,"tokens_out":569,"duration_ms":6634,"temperature":1.0,"reasoning_tokens":506,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:24:30.856455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the exact same adaptive pipeline and run it with adaptivity disabled—same number of documents, same LLM filtering, same KG augmentation, but one uniform strategy for all question types. If document/snippet scores stay the same, the adaptive routing by itself adds nothing.","supporting_citations":[],"review_version":1}