{"id":"4a658307-87ee-4cd1-bb55-eec81b5b05ce","arxiv_id":"2504.21252","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A plug-and-play RAG module that runs multi-agent discussions and post-retrieval verification improves medical QA accuracy over MedRAG on four benchmarks by up to 16.67%.","lead":"Discuss-RAG asks a team of simulated medical experts to talk through a question before retrieving documents, then has another agent check the retrieved snippets before answering. On four medical QA benchmarks it beats the standard MedRAG pipeline by 4 to 17 percentage points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer leakage through the distilled summary D is unchecked; if D encodes the answer, the retrieval gains are the LLM rediscovering its own supplied content, so the central mechanism is not yet established.","rationale":"I agree with the reader's weakest-assumption analysis: the most load-bearing unverified step is the no-answer-leakage guarantee for the distilled summary D. The paper's mechanism is specifically that D improves retrieval relevance, not that D provides the answer. If D implicitly contains the answer or the decisive reasoning path, then the retriever is being pointed to confirming evidence, and the reported accuracy gains are inflated relative to the claimed 'better retrieval' contribution. This is not an internal inconsistency; it is a missing control that leaves the central claim underdetermined. The proposed concrete test directly settles the issue by comparing retrieval under D_orig versus D_masked. If the gains vanish under D_masked, the paper's core contribution is mischaracterized and would need to be reframed as answer-aware query expansion. If the gains persist, the concern is empirically refuted. Given the absence of this control, the current CONDITIONAL verdict is appropriate; I would not strengthen to REJECT because the empirical pattern across four datasets is plausible and the method could still be useful, but I would not accept the central mechanistic claim without the test. No error bars are reported, but the leakage control is more specific and more decisive for the paper's stated contribution, so I focus on it here.","tokens_in":7206,"tokens_out":4319,"duration_ms":47609,"concrete_test":"For each question in all four benchmarks, generate two versions of the distilled summary: D_orig from the full query, and D_masked from the same query with the correct answer removed (mask the answer option text for MMLU-Med, MedQA-US, and BioASQ; withhold the reference abstract conclusion for PubMedQA). Run retrieval with Q+D_orig and Q+D_masked, keeping the same downstream generator, and recompute Table 1. If the accuracy gain over MedRAG largely disappears under D_masked, answer leakage is the mechanism; if the gain persists, the improved retrieval is driven by generic background knowledge and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Discuss-RAG improves MedRAG accuracy by enriching retrieval with agent-led discussion. The load-bearing assumption is that the distilled summary D, generated by expert agents, improves retrieval without leaking the answer. Section 3 states that agents are 'explicitly instructed not to answer the original query or infer a final conclusion,' but no compliance check is reported. Because D is concatenated with Q and fed to the retriever, any answer-related content in D steers retrieval toward documents that confirm the correct answer, making the downstream LLM's task easier. The paper provides no control for this: no measurement of answer-term overlap between D and the gold answer, no ablation replacing D with a generic medical summary, and no comparison to a summary generated from a query with the answer options removed. The qualitative example in Fig. 4(B) is concerning: D for the MedQA-US question highlights 'oxygen extraction by tissues' and 'lowest oxygen content,' which closely tracks the reasoning needed to select 'coronary sinus.' If such leakage occurs, the reported gains (up to +16.67% on BioASQ, +12.20% on PubMedQA) may reflect the LLM's parametric knowledge being fed back through retrieval rather than improved retrieval relevance. The Limitation section (Sec. 6) acknowledges computational overhead and limited agent interaction, but does not address this verification gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Discuss-RAG, a plug-and-play module for medical retrieval-augmented generation (RAG) that first orchestrates a team of medical expert agents to discuss the query and produce a distilled summary D, which is then concatenated with the original query for retrieval; a separate decision-maker agent post-processes the retrieved snippets before answer generation. The authors evaluate on four medical QA benchmarks (MMLU-Med, MedQA-US, BioASQ, PubMedQA) against the MedRAG baseline and report consistent accuracy improvements, with the largest gains on BioASQ (+16.67%) and PubMedQA (+12.20%). The paper also includes a motivating pilot study and a qualitative example. The code is publicly available.","tokens_in":7509,"tokens_out":4706,"duration_ms":41796,"significance":"The proposed idea of using agent-led discussion to enrich retrieval queries in medical RAG is intuitive and practically relevant, and the modular design means it could be dropped into existing training-free pipelines. The paper ships code and reports positive results on four public benchmarks. However, the current evaluation is too thin to establish the central claim: the comparison is limited to a single baseline, there are no error bars or significance tests, and the two proposed components are not ablated. Most importantly, the paper does not rule out the possibility that the distilled summary D leaks answer-relevant content back into the retrieval step, in which case the gains would be driven by the LLM rediscovering its own parametric knowledge rather than by improved retrieval relevance. If these concerns are addressed, the contribution would be a useful empirical result; at present the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The instruction that agents 'not answer the original query' is asserted but not verified. The qualitative example in Figure 4(B) shows the distilled summary D stating that 'oxygen extraction by tissues plays a significant role in determining the area with the lowest oxygen content' and highlighting 'lowest oxygen content at both time points,' which closely track the reasoning steps needed to select 'coronary sinus.' Because D is concatenated with Q and fed to the retriever, answer-related content in D can steer retrieval toward documents that confirm the correct answer. The paper provides no control for this leakage: no measurement of answer-term overlap between D and the gold answer, no ablation replacing D with a generic medical summary, and no comparison to a summary generated from a query with answer options removed. Without such a control, the reported gains cannot be attributed to improved retrieval relevance rather than to the LLM's own output being recycled through the pipeline.","section":"Section 3 and Figure 4(B)"},{"comment":"The evaluation reports a single run for each benchmark and no variance or significance measures. Since GPT-3.5 (gpt-3.5-turbo-0125) is stochastic and the method involves multiple LLM calls, the reported improvements (e.g., +16.67% on BioASQ, +12.20% on PubMedQA) may not be stable. The authors should report results over multiple runs (at least three to five) with mean and standard deviation, and test whether the differences against MedRAG are statistically significant (e.g., with a paired test or bootstrap). This is particularly important because gains vary considerably across datasets.","section":"Section 4, Table 1"},{"comment":"The paper does not include ablations of its two proposed modules: the multi-turn discussion/summarization and the post-retrieval verification. Without ablations, one cannot determine which component drives the improvement, or whether the verifier alone (without the discussion) would yield similar gains. Additionally, the paper compares only to MedRAG; a stronger evaluation would include at least one more recent RAG baseline (e.g., self-RAG or chain-of-verification RAG) to place the improvement in context.","section":"Section 4, Table 1"},{"comment":"Several implementation details needed for reproducibility are missing: the number of expert agents n, the maximum discussion rounds m, the number of retrieved chunks k, and the LLM temperature are not reported. The description of the post-retrieval verification is also underspecified: when the decision-maker U returns a negative judgment, the paper says an 'alternative retrieval strategy (e.g., a CoT-based prompt)' is triggered, but it does not describe what this fallback is, how often it fires, or whether it changes the retrieved set. These details are necessary for others to reproduce or build on the method.","section":"Sections 3 and 4"}],"minor_comments":[{"comment":"The text says 'As shown in Fig. 2(A), when the corpus is fixed (i.e., textbooks (Jin et al., 2021)), varying the number of retrieved documents k results in fluctuating accuracy across six medical subjects.' The text could be clearer about whether the data are aggregated over the six subjects or shown separately, and whether the setup matches the later experiments.","section":"Section 2"},{"comment":"The sentence 'MedRAG (Xiong et al., 2024).' is incomplete; it should be integrated into a full sentence describing the baseline and its role.","section":"Section 4"},{"comment":"The contribution list says 'comprehensive experiments comparing Discuss-RAG with standard RAG systems,' but the evaluation compares only to MedRAG. Please revise the wording to match the actual scope of the experiments.","section":"Section 1"},{"comment":"The typo 'Disscuss-RAG' appears in the text near Figure 4; correct it to 'Discuss-RAG.'","section":"Figure 4 and surrounding text"},{"comment":"The notation T^j := f_C(I^j_1, ..., I^j_n ; T^{j-1}, Q) is confusing because the superscripts on I and T are inconsistent (the paper writes I^j_i but also I_j^i in the text). Please define the indices clearly and explain what f_C takes as input.","section":"Equation (1)"},{"comment":"The recruiter agent R is introduced but its role is not described in detail; the text should clarify how R assembles the team, whether it is a separate LLM call, and how the team composition is chosen for each query.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a compact empirical study with a plausible idea and code release, but for a journal venue the evaluation needs to be substantially strengthened: the leakage control, ablations, and statistical rigor are currently insufficient to support the strong claims. The topic fits the journal's scope; the issue is the level of evidence rather than the novelty. I would encourage the authors to expand the experiments and address the leakage concern, as the expected contribution is then a solid engineering result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know that Discuss-RAG is a genuinely new combination: a multi-agent discussion and summarization step before retrieval, plus a post-retrieval verifier. The reported gains on four medical QA benchmarks are large, and if they hold, this is a useful training-free plug-in for applied systems. But the evidence is thin, and I don't think the central mechanism is yet established.\n\nWhat's new: prior work does post-hoc verification (chain-of-verification) or multi-agent decision-making (MDAgents), but nobody has used agent-led discussion to construct a retrieval query. The idea is sensible—enrich the query with domain background before hitting the retriever—and the qualitative examples show the retrieved snippets arguably become more relevant. The method is modular, and the code is public, which helps.\n\nThe soft spots are real. The evaluation compares only against MedRAG, with no repeated runs. GPT-3.5 is stochastic; a 16.67-point gain on BioASQ could be partly run-to-run volatility. There's no ablation of either module, so you can't tell how much the discussion module contributes versus the verifier. And the leakage question is legitimately unresolved. The agents are instructed not to answer the query, but the paper never checks whether the distilled summary D contains answer-relevant content. The example in Fig. 4 is suggestive but not damning: D mentions 'lowest oxygen content,' which is the query phrase, not the answer 'coronary sinus.' Still, a simple control—like comparing against a summary that just paraphrases the question without domain expertise—would settle it. The limitation section acknowledges compute and limited interaction, but oddly not this verification gap.\n\nWho is this for? Anyone working on RAG query expansion, agentic retrieval, or medical QA will want to know about this design. It's a useful empirical kickoff, not a settled result. I'd send it to peer review; a good referee can push for the missing ablations and leakage controls, and the idea deserves a fair shot.","headline":"A plausible plug-and-play RAG enhancement with a novel discuss-then-retrieve design, but the evaluation is too thin to establish that the gains come from better retrieval rather than the LLM feeding its own knowledge back through the summary.","tokens_in":8023,"tokens_out":3143,"would_cite":false,"duration_ms":32959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding an agent-led discussion stage before retrieval and a verification stage after it improves medical QA accuracy on four benchmarks, by up to 16.67 percentage points.","keywords":["medical question answering","retrieval-augmented generation","multi-agent discussion","LLM agents","post-retrieval verification","MedRAG","BioASQ","PubMedQA"],"falsifier":"Take a sample of BioASQ and PubMedQA questions, run only the discussion stage with the answers withheld, and check whether a human reader can answer the original question from the distilled summary $D$ alone; if $D$ answers a substantial fraction of questions, the no-answer instruction is not honored and the retrieval comparison is confounded. Alternatively, replace $D$ with a topic-preserving paraphrase stripped of answer-specific terms and measure whether the accuracy gains disappear.","tokens_in":7049,"feed_emoji":"🩺","tokens_out":14014,"duration_ms":119972,"temperature":0.7,"pith_summary":"Discuss-RAG is a plug-and-play module that sits between a medical question and the usual retrieval step: before retrieving, a recruiter agent convenes a team of specialized medical agents who brainstorm the query over several turns, and a summarizer agent condenses their insights into a running summary that is appended to the original query. After retrieval, a decision-maker agent judges whether the returned snippets are relevant and coherent, and if not, triggers a chain-of-thought fallback retrieval. The paper reports that this two-stage design consistently outperforms the MedRAG baseline on MMLU-Med, MedQA-US, BioASQ, and PubMedQA, with the largest gains on BioASQ (58.61% to 75.28%) and PubMedQA (35.60% to 47.80%). Because both stages are training-free, the claim, if correct, means existing medical RAG pipelines can be made more accurate by adding discussion and verification rather than retraining the model.","feed_headline":"Talking before retrieving lifts medical QA accuracy up to 16.67 points","feed_subtitle":"A plug-in discussion-and-verification module beats standard RAG on four medical QA benchmarks.","key_machinery":"The load-bearing mechanism is the two-stage agent pipeline. Stage one is multi-turn discussion and summarization: at each turn $j$, expert agents $H_i$ each produce insight $I_i^j$, and a summarizer agent $C$ updates the running summary as $T^j := f_C(I_1^j,\\ldots,I_n^j; T^{j-1}, Q)$; a verifier $V$ then distills the final summary into a verification-passed summary $D$ that is appended to $Q$ for retrieval. Stage two is post-retrieval verification: a decision-maker agent $U$ inspects the top-$k$ snippets $S_i$ and either accepts them for answer generation or triggers an alternative retrieval strategy. The paper's key design choice is that the expert agents are instructed not to answer the query, so the discussion stage is meant to build context for retrieval rather than produce the final answer.","core_discovery":"The core discovery is that the quality of retrieved snippets, not the generator alone, is the bottleneck in medical RAG, and that agent-led context construction changes what the retriever returns. In Discuss-RAG, the distilled verification-passed summary $D$, produced by a team of expert agents and a summarizer, is concatenated with the original query $Q$ and fed to the retriever; this yields snippets that are more topically relevant and better ordered, as illustrated in the MedQA-US oxygen-saturation example. The decision-maker agent then gates the snippets before answer generation, filtering out irrelevant or misleading evidence and using a chain-of-thought prompt as a fallback when the retrieved set looks poor. On the four benchmarks, the paper reports consistent accuracy improvements over MedRAG, including large gains on BioASQ and PubMedQA, and attributes them to better snippet relevance and coherence rather than to modifications of the answer generator.","pith_inferences":["An editorial extension: the same discuss-before-retrieve, verify-after-retrieve pattern could transfer to other high-stakes QA domains, such as legal or scientific literature, where surface-similarity retrieval is also a weak signal.","A testable extension: replace the multi-agent discussion with a single agent writing a background summary under the same no-answer instruction; if accuracy stays at the Discuss-RAG level, the team discussion is unnecessary, and if it drops, the multi-perspective interaction is the active ingredient.","A cost-saving extension: because the authors note the framework adds computational overhead, distilled summaries could be cached or reused for similar queries rather than regenerated from scratch each time."],"forward_implications":["If the reported gains are real, any training-free medical RAG pipeline can likely be improved by inserting a discussion-based query enrichment step and a snippet-verification step, independent of the retriever and generator used.","The larger gains on BioASQ and PubMedQA, where baseline accuracy is lower, suggest the module is most valuable when retrieval noise is high or corpus coverage is weak.","The decision-maker's fallback mechanism implies that retrieval failures can be detected before answer generation, so a system could selectively apply more expensive retrieval strategies only when needed.","Because the module changes only the retriever's input and the selection of snippets, it should combine additively with better retrievers, better LLMs, or other RAG enhancements."],"supporting_citations":[{"why":"It defines the MedRAG baseline whose accuracy numbers and retrieval setup are the comparison standard.","marker":"(Xiong et al., 2024)"},{"why":"It supplies the MedQA-US benchmark and the medical textbook corpus used for retrieval in all experiments.","marker":"(Jin et al., 2021)"},{"why":"It provides the MedCPT retriever that fetches snippets in both the baseline and the proposed pipeline.","marker":"(Jin et al., 2023)"},{"why":"It specifies the GPT-3.5-turbo model used as the LLM for agents, retrieval evaluation, and answer generation.","marker":"(OpenAI, 2024)"},{"why":"It motivates the chain-of-thought fallback retrieval prompt used when the decision-maker agent rejects the retrieved snippets.","marker":"(Wei et al., 2022)"},{"why":"It defines the MMLU-Med benchmark on which Discuss-RAG reports a 5.70 percentage point gain.","marker":"(Hendrycks et al., 2020)"},{"why":"It defines the BioASQ benchmark, the dataset with the largest reported gain of 16.67 percentage points.","marker":"(Tsatsaronis et al., 2015)"},{"why":"It defines the PubMedQA benchmark, the second-largest reported gain of 12.20 percentage points.","marker":"(Jin et al., 2019)"}],"fun_headline_variants":["Agent chats before retrieval boost medical QA by 16.67%","Discuss-RAG: brainstorming agents improve retrieval, accuracy up 16.67%","Pre-retrieval discussion lifts RAG medical QA accuracy 16.67%","Expert agent talks enhance retrieval, boosting medical QA 16.67%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the expert agents, even when told not to answer the query, produce discussion summaries that improve retrieval without already containing the correct answer; if the summary leaks the answer, the reported retrieval gains are partly the model retrieving what it already wrote.","fun_headline_variants_meta":{"raw":{"variants":["Agent chats before retrieval boost medical QA by 16.67%","Discuss-RAG: brainstorming agents improve retrieval, accuracy up 16.67%","Pre-retrieval discussion lifts RAG medical QA accuracy 16.67%","Expert agent talks enhance retrieval, boosting medical QA 16.67%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1513,"prompt_tokens":951,"completion_tokens":562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":567,"tokens_out":562,"duration_ms":6141,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:08:35.511775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of BioASQ and PubMedQA questions, run only the discussion stage with the answers withheld, and check whether a human reader can answer the original question from the distilled summary $D$ alone; if $D$ answers a substantial fraction of questions, the no-answer instruction is not honored and the retrieval comparison is confounded. Alternatively, replace $D$ with a topic-preserving paraphrase stripped of answer-specific terms and measure whether the accuracy gains disappear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the MedRAG baseline whose accuracy numbers and retrieval setup are the comparison standard."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It specifies the GPT-3.5-turbo model used as the LLM for agents, retrieval evaluation, and answer generation."}],"review_version":1}